Apache Hadoop with YARN: Configuring Pepperdata Activation (RPM/DEB)

Supported versions: See the Apache Hadoop entries for Pepperdata 8.1.x in the table of Supported Platforms by Pepperdata Version

To activate Pepperdata—inject the necessary instrumentation—across a manually configured cluster, add the activation script to each Hadoop service’s environment shell executable file.

Procedure

1. Beginning with any host, add the snippet below to the appropriate environment shell executable file(s), based on which services are configured to run on the host.

   • YARN ResourceManager/NodeManager: yarn-env.sh, typically located in /etc/hadoop/conf

   • Apache Spark: spark-env.sh (all hosts, including edge/client), typically located in /etc/spark/conf. If you are running multiple versions of Spark, add the snippet to the configuration file for each Spark version.
     
     

   Important: Add the snippet to the end of the script(s).
   
   This ensures that the activation script’s variable appends (YARN_NODEMANAGER_OPTS, YARN_RESOURCEMANAGER_OPTS, HBASE_REGIONSERVER_OPTS, and SPARK_SUBMIT_OPTS) are not overwritten by other assignments in the script(s).

   PEPPERDATA_ACTIVATE_SCRIPT_PATH="/opt/pepperdata/supervisor/lib/pepperdata-activate.sh"
   if [ -e $PEPPERDATA_ACTIVATE_SCRIPT_PATH ]; then
    . $PEPPERDATA_ACTIVATE_SCRIPT_PATH
   fi
   

2. Repeat step 1 on every host in the cluster.

3. Restart the following application daemons:

   • YARN ResourceManager
   • YARN NodeManagers (all in the cluster)