Monitor Log Files

You can search for particular strings within a given log file, and create a metric that is displayed in Sysdig Monitor’s Explore page. The metrics appear under the StatsD section:

Sysdig provides this functionality via a “chisel” script called “logwatcher”, written in Lua. You call the script by adding a logwatcher parameter in the chisels section of the agent configuration file (dragent.yaml). You define the log file name and the precise string to be searched. The results are displayed as metrics in the Monitor UI.

Caveats

The logwatcher chisel adds to Sysdig’s monitoring capability but is not a fully featured log monitor. Note the following limitations:

  • No regex support: Sysdig does not offer regex support; you must define the precise log file and string to be searched.

    (If you were to supply a string with spaces, forward-slashes, or back-slashes in it, the metric generated would also have these characters and so could not be used to create an alert.)

  • Limit of 12 string searches/host: Logwatcher is implemented as a LUA script and, due to resources consumed by this chisel, it is not recommended to have more than a dozen string searches configured per agent/host.

Implementation

Edit the agent configuration file to enable the logwatcher chisel. See Understanding the Agent Config Files for editing options.

Preparation

Determine the log file name(s) and string(s) you want to monitor.

Tip

To monitor the output of docker logs <container-name>, find the container’s docker log file with:

docker inspect <container-name> | grep LogPath

Edit dragent.yaml

  1. Access dragent.yaml directly at /opt/draios/etc/dragent.yaml.

  2. Add a chisels entry:

    Format:

    chisels:
      - name: logwatcher
        args:
          filespattern: YOURFILENAME.log
          term: YOURSTRING
    

    Sample Entry:

    customerid: 831f2-your-key-here-d69401
    tags: tagname.tagvalue
    chisels:
      - name: logwatcher
        args:
          filespattern: draios.log
          term: Sent
    

    In this example, Sysdig’s own draios.log is searched for the Sent string.

    The output, in the Sysdig Monitor UI, would show the StatsD metric logwatcher.draios_log.Sent and the number of ‘Sent’ items detected.

  3. Optional: Add multiple -name: sections in the config file to search for additional logs/strings.

    Note the recommended 12-string/agent limit.

  4. Restart the agent for changes to take effect.

    For container agent:

    docker restart sysdig-agent
    

    For non-containerized (service) agent:

    service dragent restart
    

Parameters

NameValueDescription
namelogwatcherThe chisel used in the enterprise Sysdig platform to search log files. (Other chisels are available in Sysdig’s open-source product.)
filespatternYOURFILENAME.logThe log file to be searched. Do not specify a path with the file name.
termYOURSTRINGThe string to be searched.

View Log File Metrics in the Monitor UI

To view logwatcher results:

  1. Log in to Sysdig Monitor and select Explore.

  2. Select Entire Infrastructure > Overview by Host.

  3. In the resulting drop-down, either scroll to Metrics > StatsD > logwatcher or enter “logwatcher” in the search field.

    Each string you configured in the agent config file will be listed in the format logwatcher.YOURFILENAME_log.STRING.

  4. The relevant metrics are displayed.

You can also Add an Alert on logwatcher metrics, to be notified when an important log entry appears.