Apache Slider AppPackage

Slider AppPackages are a declarative definition of an application for application management. AppPackage is not a packaging scheme for application binaries and artifacts. Tarball, zip files, rpms etc. are available for that purpose. Instead AppPackage includes the application binaries along with other artifacts necessary for application management.

An application instance consists of several active component such as one or more masters and several slaves. There may be a number of accompanying processes in addition to the basic master/slave processes - lets refer to all processes as app component instances. When run in the context of Yarn, the application specific processes are activated within individual Yarn Container. If you pry into an Yarn container (created through Slider) it will be apparent as to what is the role of Slider-Agent and the actual application components. The following image provides an high-level view. Within a container there are at least two processes - and instance of a slider agent and an instance of an application component. The application can itself spawn more procsses if needed.


Figure 1 - High-level view of a container

For example:

yarn      8849  -- python ./infra/agent/slider-agent/agent/main.py --label container_1397675825552_0011_01_000003___HBASE_REGIONSERVER --host AM_HOST --port 47830
yarn      9085  -- bash /hadoop/yarn/local/usercache/yarn/appcache/application_1397675825552_0011/ ... internal_start regionserver
yarn      9114 -- /usr/jdk64/jdk1.7.0_45/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=...

The above list shows three processes, the Slider-Agent process, the bash script to start HBase Region Server and the HBase Region server itself. Three of these together constitute the container.

Using an AppPackage

The following command creates an HBase application using the AppPackage for HBase.

  ./slider create cl1 --template /work/appConf.json --resources /work/resources.json

Lets analyze various parameters from the perspective of app creation:

  • --template: app configuration
  • --resources: yarn resource requests
  • … other parameters are described in accompanying docs.


The structure of an AppPackage is described at AppPackage.

In the enlistment, there are several example AppPackages:

  • app-packages/memcached
  • app-packages/memcached-win
  • app-packages/hbase
  • ...

The above folders, with minor edits, can be packaged as zip files to get the corresponding AppPackages. The application tarball file, containing the binaries/artifacts of the application itself is a component within the AppPackage. Refer to the README file in the individual folders on how to package an application.

For example, for Accumulo, package creation is automated in maven

mvn clean package -DskipTests -Paccumulo-app-package

Additional configuration parameters are described in app-packages/accumulo/README.txt.

Looking at the content through unzip -l "$@" .zip

Archive:  hbase-v098.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
     3158  06-19-2014 22:08   appConfig.json
        0  06-19-2014 22:15   configuration/
     5077  06-19-2014 22:15   configuration/global.xml
     5248  06-19-2014 22:15   configuration/hbase-log4j.xml
     2250  06-19-2014 22:15   configuration/hbase-policy.xml
    14705  06-19-2014 22:15   configuration/hbase-site.xml
     2142  06-19-2014 22:17   jmx_metrics.json
     3991  06-19-2014 22:45   metainfo.xml
        0  06-19-2014 22:33   package/
        0  06-19-2014 22:31   package/files/
 83278591  06-19-2014 22:31   package/files/hbase-0.98.3-hadoop2-bin.tar.gz
        0  06-19-2014 22:29   package/scripts/
      787  06-19-2014 22:27   package/scripts/__init__.py
     1378  06-19-2014 22:27   package/scripts/functions.py
     3738  06-19-2014 22:27   package/scripts/hbase.py
     1205  06-19-2014 22:27   package/scripts/hbase_client.py
     1640  06-19-2014 22:27   package/scripts/hbase_master.py
     1764  06-19-2014 22:27   package/scripts/hbase_regionserver.py
     1482  06-19-2014 22:27   package/scripts/hbase_service.py
     5089  06-19-2014 22:27   package/scripts/params.py
      973  06-19-2014 22:27   package/scripts/status_params.py
        0  06-19-2014 22:34   package/templates/
     2799  06-19-2014 22:34   package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
     2799  06-19-2014 22:34   package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
     3878  06-19-2014 22:34   package/templates/hbase-env.sh.j2
      909  06-19-2014 22:34   package/templates/hbase_client_jaas.conf.j2
      989  06-19-2014 22:34   package/templates/hbase_master_jaas.conf.j2
     1001  06-19-2014 22:34   package/templates/hbase_regionserver_jaas.conf.j2
      837  06-19-2014 22:34   package/templates/regionservers.j2
     3486  06-19-2014 22:39   pom.xml
     1921  06-19-2014 21:53   README.txt
      358  06-19-2014 22:36   resources.json
        0  06-19-2014 22:40   src/
        0  06-19-2014 23:02   src/assembly/
     2400  06-19-2014 22:45   src/assembly/hbase.xml
---------                     -------
 83354595                     35 files

Sample resources-default.json and appConfig-default.json files are also included in the enlistment. These are samples and are typically tested on one node test installations. These files are not used during the create commmand, rather the files provided as input parameter are the ones that are used. So you can leave these files as is in the package.

--template appConfig.json

An appConfig.json contains the application configuration. See Specifications InstanceConfiguration for details on how to create a template config file. The enlistment includes sample config files for HBase, Accumulo, and Storm.

--resources resources.json

Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application. Specification of Resources describes how to write a resource config json file. The enlistment includes sample config files for HBase, Accumulo, and Storm.

Scripting for AppPackage

Refer to App Command Scripts for details on how to write scripts for a AppPackage. These scripts are in the package/script folder within the AppPackage. Use the checked in samples for HBase/Storm/Accumulo as reference for script development.