This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

    Integrate JMX Metrics from Java Virtual Machines

    The Sysdig agent retrieves data from your Java virtual machines using the JMX protocol. The agent is configured to automatically discover active Java virtual machines and poll them for basic JVM metrics like Heap Memory and Garbage collector as well as application-specific metrics. Now, the following applications are supported by default:

    • ActiveMQ
    • Cassandra
    • Elasticsearch
    • HBase
    • Kafka
    • Tomcat
    • Zookeeper

    The agent can also be easily configured to extract custom JMX metrics coming from your own Java processes. Metrics extracted are shown in the pre-defined Application views or under the Metrics > JVM and JMX menus.

    The module java.management must be loaded for the Sysdig agent to collect both JVM and JMX metrics.

    The default JMX metrics configuration is found in the /opt/draios/etc/dragent.default.yaml file. When customizing existing entries, copy the complete application’s bean listing from that defaults yaml file into the user settings file /opt/draios/etc/dragent.yaml. The Sysdig agent will merge configurations of both files.

    Java versions 7 - 10 are currently supported by the Sysdig agents.

    For Java 11-14 you must be running minimum agent version 10.1.0 and must run the app with the JMX Remote option.

    Here is what your dragent.yaml file might look like for a customized entry for the Spark application:

    customerid: 07c948-your-key-here-006f3b
    tags: local:nyc,service:db3
    jmx:
      per_process_beans:
        spark:
          pattern: "spark"
          beans:
            - query: "metrics:name=Spark shell.BlockManager.disk.diskSpaceUsed_MB"
              attributes:
                - name: VALUE
                  alias: spark.metric
    

    Include the jmx: and per_process_beans: section headers at the beginning of your application/bean list. For more information on adding parameters to a container agent’s configuration file, see Understanding the Agent Config Files.

    Bean Configuration

    Basic JVM metrics are pre-defined inside the default_beans: section. This section is defined in the agent’s default settings file and contains beans and attributes that are going to be polled for every Java process, like memory and garbage collector usage:

    jmx:
      default_beans:
        - query: "java.lang:type=Memory"
          attributes:
            - HeapMemoryUsage
            - NonHeapMemoryUsage
        - query: "java.lang:type=GarbageCollector,*"
          attributes:
            - name: "CollectionCount"
              type: "counter"
            - name: "CollectionTime"
              type: "counter"
    

    Metrics specific for each application are specified in sections named after the applications. For example, this is the Tomcat section:

    per_process_beans:
        tomcat:
          pattern: "catalina"
          beans:
            - query: "Catalina:type=Cache,*"
              attributes:
                - accessCount
                - cacheSize
                - hitsCount
                - . . .
    

    The key name, tomcat in this case, will be displayed as a process name in the Sysdig Monitor user interface instead of just java. The pattern: parameter specifies a string that is used to match a java process name and arguments with this set of JMX metrics. If the process main class full name contains the given text, the process is tagged and the metrics specified in the section will be fetched.

    The class names are matched against the process argument list. If you implement JMX metrics in a custom manner that does not expose the class names on the command line, you will need to find a pattern which conveniently matches your java invocation command line.

    The beans: section contains the list of beans to be queried, based on JMX patterns. JMX patterns are explained in details in the Oracle documentation, but in practice, the format of the query line is pretty simple: you can specify the full name of the bean like java.lang:type=Memory , or you can fetch multiple beans in a single line using the wildcard * as in: java.lang:type=GarbageCollector,* (note that this is just a wildcard, not a regex).

    To get the list of all the beans and attributes that your application exports, you can use JVisualVM, Jmxterm, JConsole or other similar tools. Here is a screenshot from JConsole showing where to find the namespace, bean and attribute (metric) information (JConsole is available when you install the Java Development Kit):

    For each query, you have to specify the attributes that you want to retrieve, and for each of them a new metric will be created. We support the following JMX attributes types (For these attributes, all the subattributes will be retrieved):

    Attributes may be absolute values or rates. For absolute values, we need to calculate a per second rate before sending them. In this case, you can specify type: counter , the default is rate which can be omitted, so usually you can simply write the attribute name.

    Limits

    The total number of JMX metrics polled per host is limited to 500. The maximum number of beans queried per process is limited to 300. If more metrics are needed please contact your sales representative with your use case.

    In agents 0.46 and earlier, the limit was 100 beans for each process.

    Aliases

    JMX beans and attributes can have very long names. To avoid interface cluttering we added support for aliasing, you can specify an alias in the attribute configuration. For example:

      cassandra:
        pattern: "cassandra"
        beans:
          - query: "org.apache.cassandra.db:type=StorageProxy
            attributes:
              - name: RecentWriteLatencyMicros
                alias: cassandra.write.latency
              - name: RecentReadLatencyMicros
                alias: cassandra.read.latency
    

    In this way the alias will be used in Sysdig Monitor instead of the raw bean name. Aliases can be dynamic as well, getting data from the bean name - useful where you use pattern bean queries. For example, here we are using the attribute name to create different metrics:

          - query: "java.lang:type=GarbageCollector,*"
            attributes:
              - name: CollectionCount
                type: counter
                alias: jvm.gc.{name}.count
              - name: CollectionTime
                type: counter
                alias: jvm.gc.{name}.time
    

    This query will match multiple beans (All Garbage collectors) and the metric name will reflect the name of the Garbage Collector. For example: jvm.gc.ConcurrentMarkSweep.count . General syntax is: {<bean_property_key>} , to get all beans properties you can use a JMX explorer like JVisualVM or Jmxterm.

    To use these metrics in promQL queries, you have to add the prefix jmx_ and replace the dots (.) from metrics name by underscores (_). For example, the metric name jvm.gc.ConcurrentMarkSweep.count will be jmx_jvm_gc_ConcurrentMarkSweep_count in promQL.

    Troubleshooting: Why Can’t I See Java (JMX) Metrics?

    The Sysdig agent normally auto-discovers Java processes running on your host and enables the JMX extensions for polling them.

    JMX Remote

    If your Java application is not discovered automatically by the agent, try adding the following parameter on your application’s command line:

     -Dcom.sun.management.jmxremote
    

    For more information, see Oracle’s web page on monitoring using JMX technology.

    Java Versions

    Java versions 7 - 10 are currently supported by the Sysdig agents.

    For Java 11-14 you must be running minimum agent version 10.1.0 and must run the app with the JMX Remote option.

    Java-Based Applications and JMX Authentication

    For Java-based applications (Cassandra, Elasticsearch, Kafka, Tomcat, Zookeeper and etc.), the Sysdig agent requires the Java runtime environment (JRE) to be installed to poll for metrics (beans).

    The Sysdig agent does not support JMX authentication.

    If the Docker-container-based Sysdig agent is installed, the JRE is installed alongside the agent binaries and no further dependencies exist. However, if you are installing the service-based agent (non-container) and you do not see the JVM/JMX metrics reporting, your host may not have the JRE installed or it may not be installed in the expected location: usr/bin/java

    To confirm if the Sysdig agent is able to find the JRE, restart the agent with service dragent restart and check the agent’s /opt/draios/logs/draios.log file for the two Java detection and location log entries recorded during agent startup.

    Example if Java is missing or not found:

    2017-09-08 23:19:27.944, Information, java detected: false
    2017-09-08 23:19:27.944, Information, java_binary:
    

    Example if Java is found:

    2017-09-08 23:19:27.944, Information, java detected: true
    2017-09-08 23:19:27.944, Information, java_binary: /usr/bin/java
    

    If Java is not installed, the resolution is to install the Java Runtime Environment. If your host has Java installed but not in the expected location ( /usr/bin/java ) you can install a symlink from /usr/bin/java to the actual binary OR set the java_home: variable in the Sysdig agent’s configuration file: /opt/draios/etc/dragent.yaml

    java_home: /usr/my_java_location/
    

    Disabling JMX Polling

    If you do not need it or otherwise want to disable JMX metrics reporting, you can add the following two lines to the agent’s user settings configuration file /opt/draios/etc/dragent.yaml:

    jmx:
      enabled: false
    

    After editing the file, restart the native Linux agent via service dragent restart or restart the container agent to make the change take effect.

    If using our containerized agent, instead of editing the dragent.yaml file, you can add this extra parameter in the docker run command when starting the agent:

    -e ADDITIONAL_CONF="jmx:\n  enabled: false\n"