Verify and Validate Program Matching Rules

To ensure that Pepperdata can use your program matching rules for custom program monitoring, you should use the custom program matching rules linter that’s included in the Pepperdata package. The linter outputs warnings for any problems it finds and prints the combined configuration—the combination of the default rules file for preconfigured program monitoring and your user, custom file for overrides and programs that you want to add to Pepperdata monitoring.

Prerequisites

Configure the rules file name and location, and add your program matching rules, according to the procedure in Add Custom-Monitored Programs.

Procedure

Run the program matching rules linter, program_monitor_configuration_linter.

/opt/pepperdata/supervisor/program_monitor_configuration_linter
If the linter reports any problems, revise your file and repeat the test.

Repeat this process until the linter returns the following success message for every yaml file it reads:

Validation success - Valid definitions.

Example: Valid Rules File

Consider a simple rules file that has one rule that overrides the default monitoring for “impala”:

programs:
  impala:
    active: no

Running the program matching rules linter produces output with “Parsing success” and “Validation success” messages, and a listing of all the rules—default program monitoring, custom program monitoring, and extended JMX metrics. In this example, the “impala” program monitoring is disabled, and the extended JMX metrics is an empty set, jmx: {}.

$ /opt/pepperdata/supervisor/program_monitor_configuration_linter
17/08/24 13:40:36 INFO config.f: Using Hadoop configuration
17/08/24 13:40:36 INFO config.j: Pepperdata configuration file URL: file:/etc/pepperdata/pepperdata-site.xml (Future similar messages will be omitted up to 10 times)
17/08/24 13:40:36 INFO common.MonitoredProgramConfigParser: Reading default program definitions to monitor from '/opt/pepperdata/supervisor/lib/pepagent-program-monitor-config-default.yaml'
17/08/24 13:40:36 INFO common.MonitoredProgramConfigParser: File:'/opt/pepperdata/supervisor/lib/pepagent-program-monitor-config-default.yaml'. Parsing success - Successfully parsed the definitions file. Validation success - Valid definitions.
17/08/24 13:40:36 INFO common.MonitoredProgramConfigParser: Reading override program definitions to monitor from '/etc/pepperdata/jmx-metrics-defs.yml'
17/08/24 13:40:36 INFO common.MonitoredProgramConfigParser: File:'/etc/pepperdata/jmx-metrics-defs.yml'. Parsing success - Successfully parsed the definitions file. Validation success - Valid definitions.
Combined program definitions ---
programs:
  impala:
    active: false
    rules:
    - command-match:
        regex: "^.*impalad "
      pid-locations:
      - "/var/run/impala/impalad-impala.pid"
    - command-match:
        regex: "^.*impalad "
  mapreducehistory:
    active: true
    rules:
    - command-match:
        substring: "-Dproc_historyserver"
      pid-locations:
      - "/var/run/hadoop-mapreduce/mapred-mapred-historyserver.pid"
      - "/var/run/hadoop-mapreduce/mapred/mapred-mapred-historyserver.pid"
    - command-match:
        substring: "-Dproc_historyserver"
  sparkhistory:
    active: true
    rules:
    - command-match:
        substring: "org.apache.spark.deploy.history.HistoryServer"
      pid-locations:
      - "/var/run/spark/spark-history-server.pid"
      - "/var/run/spark/spark-spark-org.apache.spark.deploy.history.HistoryServer-1.pid"
jmx: {}

Example: Invalid Rules File

Now consider same rules file we used above, but that has a typo in the active key value, “nooo”:

programs:
  impala:
    active: nooo

Running the program matching rules linter produces output with “Parsing Failure” and “Validation Failure” messages, and a combined rules listing that shows only the default rules (because the custom rules file is invalid).

$ /opt/pepperdata/supervisor/program_monitor_configuration_linter
17/08/24 13:47:40 INFO config.f: Using Hadoop configuration
17/08/24 13:47:40 INFO config.j: Pepperdata configuration file URL: file:/etc/pepperdata/pepperdata-site.xml (Future similar messages will be omitted up to 10 times)
17/08/24 13:47:40 INFO common.MonitoredProgramConfigParser: Reading default program definitions to monitor from '/opt/pepperdata/supervisor/lib/pepagent-program-monitor-config-default.yaml'
17/08/24 13:47:40 INFO common.MonitoredProgramConfigParser: File:'/opt/pepperdata/supervisor/lib/pepagent-program-monitor-config-default.yaml'. Parsing success - Successfully parsed the definitions file. Validation success - Valid definitions.
17/08/24 13:47:40 INFO common.MonitoredProgramConfigParser: Reading override program definitions to monitor from '/etc/pepperdata/jmx-metrics-defs.yml'
17/08/24 13:47:40 WARN common.MonitoredProgramConfigParser: Unable to parse /etc/pepperdata/jmx-metrics-defs.yml. Can not construct instance of boolean from String value ("nooo"): only "true" or "false" recognized
 at [Source: java.io.StringReader@f7da237; line: 3, column: 13] (through reference chain: com.pepperdata.supervisor.agent.common.ProgramDefinitions["programs"]->java.util.LinkedHashMap["impala"]->com.pepperdata.supervisor.agent.common.Program["active"])
17/08/24 13:47:40 WARN common.MonitoredProgramConfigParser: Validation Failed for /etc/pepperdata/jmx-metrics-defs.yml
17/08/24 13:47:40 WARN common.MonitoredProgramConfigParser: File:'/etc/pepperdata/jmx-metrics-defs.yml'. Parsing Failure - Encountered exception while parsing the definition file. Validation Failure - Unable to parse and build program definitions from the config file.
Combined program definitions ---
programs:
  impala:
    active: true
    rules:
    - command-match:
        regex: "^.*impalad "
      pid-locations:
      - "/var/run/impala/impalad-impala.pid"
    - command-match:
        regex: "^.*impalad "
  sparkhistory:
    active: true
    rules:
    - command-match:
        substring: "org.apache.spark.deploy.history.HistoryServer"
      pid-locations:
      - "/var/run/spark/spark-history-server.pid"
      - "/var/run/spark/spark-spark-org.apache.spark.deploy.history.HistoryServer-1.pid"
  mapreducehistory:
    active: true
    rules:
    - command-match:
        substring: "-Dproc_historyserver"
      pid-locations:
      - "/var/run/hadoop-mapreduce/mapred-mapred-historyserver.pid"
      - "/var/run/hadoop-mapreduce/mapred/mapred-mapred-historyserver.pid"
    - command-match:
        substring: "-Dproc_historyserver"
jmx: {}