Integrate JMX Metrics from Java Virtual Machines

The following applications are supported by default:

• ActiveMQ
• Cassandra
• Elasticsearch
• HBase
• Kafka
• Tomcat
• Zookeeper

The agent can also be easily configured to extract custom JMX metrics coming from your own Java processes. Metrics extracted are shown in the pre-defined Application views or under the Metrics > JVM and JMX menus.

The default JMX metrics configuration is found in the /opt/draios/etc/dragent.default.yaml file. When customizing existing entries, copy the complete application’s bean listing from that defaults yaml file into the user settings file /opt/draios/etc/dragent.yaml. The Sysdig agent will merge configurations of both files.

Here is what your dragent.yaml file might look like for a customized entry for the Spark application:

customerid: 07c948-your-key-here-006f3b
tags: local:nyc,service:db3
jmx:
  per_process_beans:
    spark:
      pattern: "spark"
      beans:
        - query: "metrics:name=Spark shell.BlockManager.disk.diskSpaceUsed_MB"
          attributes:
            - name: VALUE
              alias: spark.metric

Include the jmx: and per_process_beans: section headers at the beginning of your application/bean list. For more information on adding parameters to a container agent’s configuration file, see Configure Sysdig Agent.

Bean Configuration

Basic JVM metrics are pre-defined inside the default_beans: section. This section is defined in the agent’s default settings file and contains beans and attributes that are going to be polled for every Java process, like memory and garbage collector usage:

jmx:
  default_beans:
    - query: "java.lang:type=Memory"
      attributes:
        - HeapMemoryUsage
        - NonHeapMemoryUsage
    - query: "java.lang:type=GarbageCollector,*"
      attributes:
        - name: "CollectionCount"
          type: "counter"
        - name: "CollectionTime"
          type: "counter"

Metrics specific for each application are specified in sections named after the applications. For example, this is the Tomcat section:

per_process_beans:
    tomcat:
      pattern: "catalina"
      beans:
        - query: "Catalina:type=Cache,*"
          attributes:
            - accessCount
            - cacheSize
            - hitsCount
            - . . .

The key name, tomcat in this case, will be displayed as a process name in the Sysdig Monitor user interface instead of just java. The pattern: parameter specifies a string that is used to match a java process name and arguments with this set of JMX metrics. If the process main class full name contains the given text, the process is tagged and the metrics specified in the section will be fetched.

The beans: section contains the list of beans to be queried, based on JMX patterns. JMX patterns are explained in details in the Oracle documentation, but in practice, the format of the query line is pretty simple: you can specify the full name of the bean like java.lang:type=Memory , or you can fetch multiple beans in a single line using the wildcard * as in: java.lang:type=GarbageCollector,* (note that this is just a wildcard, not a regex).

To get the list of all the beans and attributes that your application exports, you can use JVisualVM, Jmxterm, JConsole or other similar tools. Here is a screenshot from JConsole showing where to find the namespace, bean and attribute (metric) information (JConsole is available when you install the Java Development Kit):

For each query, you have to specify the attributes that you want to retrieve, and for each of them a new metric will be created. We support the following JMX attributes types (For these attributes, all the subattributes will be retrieved):

• Numeric

• CompositeDataSupport

Attributes may be absolute values or rates. For absolute values, we need to calculate a per second rate before sending them. In this case, you can specify type: counter , the default is rate which can be omitted, so usually you can simply write the attribute name.

Limits

The total number of JMX metrics polled per host is limited to 500. The maximum number of beans queried per process is limited to 300. If more metrics are needed please contact your sales representative with your use case.

In agents 0.46 and earlier, the limit was 100 beans for each process.

Aliases

JMX beans and attributes can have very long names. To avoid interface cluttering we added support for aliasing, you can specify an alias in the attribute configuration. For example:

  cassandra:
    pattern: "cassandra"
    beans:
      - query: "org.apache.cassandra.db:type=StorageProxy
        attributes:
          - name: RecentWriteLatencyMicros
            alias: cassandra.write.latency
          - name: RecentReadLatencyMicros
            alias: cassandra.read.latency

In this way the alias will be used in Sysdig Monitor instead of the raw bean name. Aliases can be dynamic as well, getting data from the bean name - useful where you use pattern bean queries. For example, here we are using the attribute name to create different metrics:

      - query: "java.lang:type=GarbageCollector,*"
        attributes:
          - name: CollectionCount
            type: counter
            alias: jvm.gc.{name}.count
          - name: CollectionTime
            type: counter
            alias: jvm.gc.{name}.time

This query will match multiple beans (All Garbage collectors) and the metric name will reflect the name of the Garbage Collector. For example: jvm.gc.ConcurrentMarkSweep.count . General syntax is: {<bean_property_key>} , to get all beans properties you can use a JMX explorer like JVisualVM or Jmxterm.

Collect JMX Labels

You can configure dragent.yaml to collect labels from Java processes.

jmx:
  per_process_beans:
    equinoxTest:
      pattern: org.eclipse.equinox
      beans:
      - query: java.lang:type=Memory
        attributes:
        - name: "*"
          alias: test_attributes_{sysdigAttribute}
          subattributes:
          - name: "*"
            labels:
            - key: type
              value: "{type}"
            - key: attribute
              value: "{sysdigAttribute}"
            - key: subattribute
              value: "{sysdigSubattribute}"
            - key: foo
              value: bar

where org.eclipse.equinox is a string in the main class of a Java process.

For each attribute of the java.lang:type=Memory bean, there is an associated alias in the format test_attributes_{sysdigAttribute}, where {sysdigAttribute} represents the attribute name. Additionally, several labels are attached to each subattribute when the attribute is a CompositeData. The agent substitutes the placeholders such as {type}, {sysdigAttribute}, and {sysdigSubattribute} with the actual values.

Troubleshooting: Why Can’t I See Java (JMX) Metrics?

The Sysdig agent normally auto-discovers Java processes running on your host and enables the JMX extensions for polling them.

JMX Remote

If your Java application is not discovered automatically by the agent, try adding the following parameter on your application’s command line:

 -Dcom.sun.management.jmxremote

For more information, see Oracle’s web page on monitoring using JMXvtechnology.

Java Versions

Java-Based Applications and JMX Authentication

For Java-based applications (Cassandra, Elasticsearch, Kafka, Tomcat, Zookeeper and etc.), the Sysdig agent requires the Java runtime environment (JRE) to be installed to poll for metrics (beans).

If the Docker-container-based Sysdig agent is installed, the JRE is installed alongside the agent binaries and no further dependencies exist. However, if you are installing the service-based agent (non-container) and you do not see the JVM/JMX metrics reporting, your host may not have the JRE installed or it may not be installed in the expected location: usr/bin/java

To confirm if the Sysdig agent is able to find the JRE, restart the agent with service dragent restart and check the agent’s /opt/draios/logs/draios.log file for the two Java detection and location log entries recorded during agent startup.

Example if Java is missing or not found:

2017-09-08 23:19:27.944, Information, java detected: false
2017-09-08 23:19:27.944, Information, java_binary:

Example if Java is found:

2017-09-08 23:19:27.944, Information, java detected: true
2017-09-08 23:19:27.944, Information, java_binary: /usr/bin/java

If Java is not installed, the resolution is to install the Java Runtime Environment. If your host has Java installed but not in the expected location ( /usr/bin/java ) you can install a symlink from /usr/bin/java to the actual binary OR set the java_home: variable in the Sysdig agent’s configuration file: /opt/draios/etc/dragent.yaml

java_home: /usr/my_java_location/

Add Arbitrary Java Command Names

You can specify the command names to launch Java processes. This helps agent to detect Java processes for JMX metric collection.

For example, if you want the agent to detect a process by the name of jsivm , while still detecting the other commands, you should add the following to dragent.yaml:

jmx:
  java_commands:
    - java
    - jsvc
    - jsivm

The values specified in dragent.yaml will override the default values, therefore, you need to include the defaults if you wish to continue detecting them.

Disabling JMX Polling

If you do not need it or otherwise want to disable JMX metrics reporting, you can add the following two lines to the agent’s user settings configuration file /opt/draios/etc/dragent.yaml:

jmx:
  enabled: false

After editing the file, restart the native Linux agent via service dragent restart or restart the container agent to make the change take effect.

If using our containerized agent, instead of editing the dragent.yaml file, you can add this extra parameter in the docker run command when starting the agent:

-e ADDITIONAL_CONF="jmx:\n  enabled: false\n"