Monitor Log Files

You can search for particular strings within a given log file, and create a metric that is displayed in Sysdig Monitor’s Explore page. The metrics appear under the StatsD section:

Sysdig provides this functionality via a “chisel” script called “logwatcher”, written in Lua. You call the script by adding a logwatcher parameter in the chisels section of the agent configuration file (dragent.yaml). You define the log file name and the precise string to be searched. The results are displayed as metrics in the Monitor UI.

Caveats

The logwatcher chisel adds to Sysdig’s monitoring capability but is not a fully featured log monitor. Note the following limitations:

• No regex support: Sysdig does not offer regex support; you must define the precise log file and string to be searched.

  (If you were to supply a string with spaces, forward-slashes, or back-slashes in it, the metric generated would also have these characters and so could not be used to create an alert.)

• Limit of 12 string searches/host: Logwatcher is implemented as a LUA script and, due to resources consumed by this chisel, it is not recommended to have more than a dozen string searches configured per agent/host.

Implementation

Edit the agent configuration file to enable the logwatcher chisel. See Understanding the Agent Config Files for editing options.

Preparation

Determine the log file name(s) and string(s) you want to monitor.

Edit dragent.yaml

1. Access dragent.yaml directly at /opt/draios/etc/dragent.yaml.

2. Add a chisels entry:

   Format:

   chisels:
     - name: logwatcher
       args:
         filespattern: YOURFILENAME.log
         term: YOURSTRING
   

   Sample Entry:

   customerid: 831f2-your-key-here-d69401
   tags: tagname.tagvalue
   chisels:
     - name: logwatcher
       args:
         filespattern: draios.log
         term: Sent
   

   In this example, Sysdig’s own draios.log is searched for the Sent string.

   The output, in the Sysdig Monitor UI, would show the StatsD metric logwatcher.draios_log.Sent and the number of ‘Sent’ items detected.

3. Optional: Add multiple -name: sections in the config file to search for additional logs/strings.

   Note the recommended 12-string/agent limit.

4. Restart the agent for changes to take effect.

   For container agent:

   docker restart sysdig-agent
   

   For non-containerized (service) agent:

   service dragent restart
   

Parameters

View Log File Metrics in the Monitor UI

To view logwatcher results:

1. Log in to Sysdig Monitor and select Explore.

2. Select Entire Infrastructure > Overview by Host.

3. In the resulting drop-down, either scroll to Metrics > StatsD > logwatcher or enter “logwatcher” in the search field.

   Each string you configured in the agent config file will be listed in the format logwatcher.YOURFILENAME_log.STRING.

   

4. The relevant metrics are displayed.