Apache Slider: Dynamic YARN Applications

Apache Slider is a YARN application to deploy existing distributed applications on YARN, monitor them and make them larger or smaller as desired -even while the application is running.

Applications can be stopped then started; the distribution of the deployed application across the YARN cluster is persisted —enabling a best-effort placement close to the previous locations. Applications which remember the previous placement of data (such as HBase) can exhibit fast start-up times from this feature.

YARN itself monitors the health of "YARN containers" hosting parts of the deployed application -it notifies the Slider manager application of container failure. Slider then asks YARN for a new container, into which Slider deploys a replacement for the failed component. As a result, Slider can keep the size of managed applications consistent with the specified configuration, even in the face of failures of servers in the cluster -as well as parts of the application itself

Some of the features are:

  • Allows users to create on-demand applications in a YARN cluster

  • Allow different users/applications to run different versions of the application.

  • Allow users to configure different application instances differently

  • Stop / Suspend / Resume application instances as needed

  • Expand / shrink application instances as needed

The Slider tool is a Java command line application.

The tool persists the information as JSON documents in HDFS.

Once the cluster has been started, the cluster can be made to grow or shrink using the Slider commands. The cluster can also be stopped and later restarted.

Slider implements all its functionality through YARN APIs and the existing application shell scripts. The goal of the application was to have minimal code changes and as of this writing, it has required few changes.

Using

Developing

Design

Disclaimer

Apache Slider (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the name of Apache TLP sponsor. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.