Simple Slider Demo

These instructions cover how to give a simple demo of slider, showing creating an application instance starting and stopping an instance publishing information in the registry flexing an instance recovery from component failure recovery from AM failure

It does not cover Configuring the application instance via app_conf.json Functional use of the deployed application other than viewing its web pages * Security


Before starting the demo you need

  1. An Apache Hadoop Hadoop 2.6+ cluster, one node or greater
  2. A local installation of the slider client (unzipped into a directory is sufficient)
  3. A slider-client.xml file containing the binding information for the Hadoop cluster

Hadoop cluster setup

The cluster must be set up with no checks on memory use, and no minimum size on container allocations. This ensures that all requested VMs will be created, irrespective of how much memory there is available on the VM.





Local environment and path setup


 export HADOOP_CONF_DIR=~/clusters/demo/hadoop-conf

 export SLIDER_CONF_DIR=~/clusters/demo/slider;

 export PATH=~/slider/slider-assembly/target/slider-0.80.0-incubating-all/slider-0.80.0-incubating/bin


 set -gx HADOOP_CONF_DIR ~/clusters/demo/hadoop-conf

 set -gx  SLIDER_CONF_DIR ~/clusters/demo/slider;

 set -gx SLIDER_VERSION slider-0.80.0-incubating
 set -gx PROJECTS /Users/stevel/Projects/Hortonworks/Projects

 set -gx  PATH $PATH $PROJECTS/slider/slider-assembly/target/$SLIDER_VERSION-all/$SLIDER_VERSION/bin

 set -gx SLIDER_CONF_DIR $PROJECTS/clusterconfigs/clusters/structor-1/slider
 set -gx HADOOP_CONF_DIR $PROJECTS/clusterconfigs/clusters/structor-1/hadoop-conf


slider version
slider list

Creating the HDFS paths

HDFS needs to have the user's path created, with the username of the user giving the demo, here demo. This username MUST be that of the local user.

hdfs dfs -chmod 777 /
hdfs dfs -mkdir -p /user/demo
hdfs dfs -chown demo /user/demo
hdfs dfs -chmod 777 /user

The YARN user account needs to be present too (in a reformatted VM this may be lost)

hdfs dfs -mkdir -p /user/yarn
hdfs dfs -chown yarn /user/yarn

Building and installing the application

This demo uses HBase, so must have the slider hbase application package installed in the cluster.

  1. Download the hbase bin package, such as hbase-0.99.1-bin.tar.gz
  2. rename it hbase-0.99.1.tar.gz, so that the name of the file matches the path to hbase inside it
  3. Untar it somewhere too (e.g. /Users/demo/hbase-0.99.1)
  4. Build the HBase application package

    mvn clean package -Phbase-app-package -Dpkg.version=0.99.1 \  \
  5. Upload it

    slider install-package --name HBASE --package target/

Preparing the resource definitions

Take the template app's resources-default.json and appConf-default.json FROM the target/ subdirectory. They must be the ones with package versions fully expanded.

In appConfig.json

  1. cut the heap size down to "256m" in all components
  2. make sure any needed path to the JDK base directory is valid.

In resources.json

  1. set "yarn.memory": "128" everywhere,
  2. Turn set the number of instances the THRIFT and REST components to 0.


slider resolve --path / --list

slider create hbase1 --resources resources.json --template appConfig.json

slider status hbase1 slider stop hbase1 slider start hbase1

slider flex hbase1 --component HBASE_REGIONSERVER 2

slider destroy hbase1


kill -9 HBase master kill -9 AM