Syntax: JMX Metrics Definitions

To write JMX metrics definitions, you need to know which JMX beans you want to collect, understand how the beans are structured, and know the required YAML elements. We recommend that you study the examples and use them as models for your JMX metrics definitions.

On This Page

About JMX Beans
About JMX Metrics Definitions
Example: Annotated JMX Metric Definition
YAML Sections: JMX Metrics Definitions
Example: HBase Regionserver
Example: Kafka RMI Interface

About JMX Beans

To configure extended JMX bean definitions, it’s useful to understand the structure of a JMX bean as it appears in HTTP. A JMX bean always has a name attribute, which contains the domain and a set of properties (for example, type and name). The other attributes can have string, numeric, or composite values, and can essentially function as tags. Composite values can be nested, and there is often a concept of an array of key-value pairs.

Consider the following example bean:

{ "name" : "java.lang:type=MemoryPool,name=PS Survivor Space",
  "modelerType" : "sun.management.MemoryPoolImpl",
  "CollectionUsageThresholdCount" : 50,
  "tag.hostName" : "someHost.com",
  "Usage" : {
    "committed" : 7340032,
    "init" : 10485760,
    "max" : 7340032,
    "used" : 4016568
  }
}

In this example, the name attribute’s domain is “java.lang”. The CollectionUsageThresholdCount attribute is a numeric value; the Usage attribute is a composite value; and the tag.hostName attribute functions as a tag.

The following example bean has nested composite values and arrays of key-value pairs:

{ "name" : "java.lang:type=example,name=Another Example",
  "LastGcInfo" : {
    "memoryUsageAfterGc" : [ {
      "key" : "Par Survivor Space",
      "value" : {
        "committed" : 20971520,
        "init" : 20971520,
        "max" : 20971520,
        "used" : 47424
      }
    }, {
      "key" : "Compressed Class Space",
      "value" : {
        "committed" : 3399680,
        "init" : 0,
        "max" : 1073741824,
        "used" : 3306592
        }
      }
  ]}
}

About JMX Metrics Definitions

You can define multiple metrics for any given bean. Depending on the specified patterns, a given metric might translate into multiple metrics. You can control whether a metric is shown in your Pepperdata dashboard.

Generally, you define an extended JMX metric by its path of elements that precede it in its tree structure. For example, the committed attribute under the Compressed Class Space key could be addressed by LastGcInfo.memoryUsageAfterGc.Compressed Class Space.committed.

To specify which JMX beans’ metrics that you want Pepperdata to collect, you add JMX metrics definitions to your rules file. The PepAgent evaluates matching rules against all JMX, and if a bean matches a JMX metrics definition, PepAgent collects the bean’s data. The matching evaluation looks at the bean name, the matched groups in the metric name, and other attributes within the bean that are used to generate the tags and final metric name.

Example: Annotated JMX Metric Definition

This example shows a typical JMX metric definition. The jmx list is a top-level dictionary—he same level as the programs dictionary for custom program matching rules. The names of the JMX services, such as “datanode”, are arbitrary labels for the service.

jmx:
  datanode: # label for the service
    uri: "http://localhost:50075/jmx" # JMX url
    rules:
      - name: "Hadoop:service=DataNode,name=DataNode" # bean name
        metrics:
          - metric: "(.*)" # select all (numeric) metrics in this bean
            name: "hadoop.datanode.$1" # use $1 from metric pattern match
            display:
              type: gauge
            tags:
              context: "$tag.Context" # set type tag to value of
                                      # tag.Context attribute
  hbase_master:
    uri: "http://localhost:60010/jmx" # JMX url
    rules:
      - name: "Hadoop:service=HBase,name=Master,sub=IPC" # bean name
        metrics:
          - metric: "(.*)" # select all (numeric) metrics in this bean
            name: "hadoop.master.$1" # use $1 from metric pattern match
            display:
              type: gauge
            tags:
              context: "$tag.Context" # set type tag to value of
                                      # tag.Context attribute

The table describes options for generating names and tags. Use brackets around tag names if they are followed by non-tag text.

`$1`, `$2`, `${1}_$2`, etc.	Replaced with matched groups in the “metric” pattern
`$_type`, `$_name`, `$_service`	Replaced with corresponding property in the bean name
`$name`, `${tag.hostName}`, `$any-attribute-name`	Replaced with the attribute value of this name (if found)

YAML Sections: JMX Metrics Definitions

The tables describe the YAML sections for JMX metrics definitions. Unless labeled optional, keys are required.

Table: YAML Sections for JMX Metrics Definitions

Key	Value
uri	URI of JMX service. Can be an HTTP service (for example, `http://localhost:50075/jmx`) or an RMI service (for example, `service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi`)
rules	List of beans with metrics to collect
name	Name of bean (a regex pattern in Java regex syntax )
metrics	List of metrics to collect within a matched bean
metric	Pattern to match on a metric. The groups that are matched may be referred to when generating tags or names.
name (optional)	Name of the metric to generate. It can refer to matches from the metric pattern, properties, and other attributes in the bean (see below). The name is always lowercase and has spaces replaced with underscores. If not specified, a name is created based on the domain, the “type” and “service” properties, and the path to the attribute name.
display (optional)	Information about how to display the metric; see Table: YAML Sections for ‘display’ Dictionary, below.
tags	Map of tags to generate. The tag values can refer to matches from the metric pattern, properties, and other attributes in the bean (see below). The canonical hostname is always added as the “host” tag, and any “host” tag specified in the configuration is ignored.

Table: YAML Sections for display Dictionary

Key	Value
description	(Default is the metric name) User-visible description of the metric.
max	(Default is calculated) Maximum y-value of the metric.
scaling	(Default is 1) Scaling factor of the metric (a number).
type	(Default=GAUGE) Metric type: COUNTER or GAUGE.
units	User-visible units of the metric; for example, “GB”, “MBps”.

Example: HBase Regionserver

HBase has very complex names for its regionserver metrics. The bean looks similar to the following:

{
  "name" : "Hadoop:service=HBase,name=RegionServer,sub=Regions",
  "modelerType" : "RegionServer,sub=Regions",
  "tag.Context" : "regionserver",
  "tag.Hostname" : "amarillo-n1.pepperdata.com",
  "Namespace_default_table_TestTable_region_1027467e1682b63a8d091d486a076402_metric_storeFileSize" : 1731623391,
    "Namespace_default_table_TestTable_region_1027467e1682b63a8d091d486a076402_metric_memStoreSize" : 424,
}

The following JMX metrics definition enables Pepperdata to collect tagged metrics from the example bean:

jmx:
  hbase_regionserver:
    uri: "http://localhost:60030/jmx"
    rules:
      - name: "Hadoop:service=HBase,name=RegionServer,sub=Regions"
        metrics:
          - metric:"Namespace_([^_]*)_table_([^_]*)_region_([^_]*)_metric_(.*storeFileSize)"
            name: "hadoop.regionserver.$4"
            tags:
              namespace: "$1"
              table: "$2"
              region: "$3"
          - metric: "Namespace_([^_]*)_table_([^_]*)_region_([^_]*)_metric_(.*memStoreSize)"
            name: "hadoop.regionserver.$4"
            tags:
              namespace: "$1"
              table: "$2"
              region: "$3"

Example: Kafka RMI Interface

The following metrics definitions provide Kafka metrics that are exposed via RMI (Remote Method Invocation):

jmx:
  kafka:
    uri: "service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi"
    rules:
      - name: "kafka.server:name=.*,type=.*"
        metrics:
          - metric: "Value"
            name: "kafka.server.${_name}"
            tags:
              type: "$_type"
      - name: "kafka.server:clientId=.*,name=.*,type=.*"
        metrics:
          - metric: "Value"
            name: "kafka.server.${_name}"
            tags:
              client: "$_clientId"
              type: "$_type"
      - name: "kafka.server:delayedOperation=.*,name=.*,type=.*"
        metrics:
          - metric: "Value"
            name: "kafka.server.${_name}"
            tags:
              operation: "$_delayedOperation"
              type: "$_type"
      - name: "kafka.network:name=.*,request=.*,type=.*"
        metrics:
          - metric: "Value"
            name: "kafka.network.${_name}"
            tags:
              request: "$_request"
              type: "$_type"
      - name: "kafka.network:name=.*,networkProcessor=.*,type=.*"
        metrics:
          - metric: "Value"
            name: "kafka.network.${_name}"
            tags:
              processor: "$_networkProcessor"
              type: "$_type"