Create Custom Prometheus Jobs via API

Creating custom Prometheus jobs via the Sysdig Monitor API provides a flexible and efficient way to configure Prometheus scraping without redeploying the sysdig-agent. This is especially useful when small changes to scrape jobs are needed across multiple environments or clusters.

Overview

Custom Prometheus jobs created with the API behave exactly like regular Prometheus jobs defined traditionally through the prometheus.yaml. Using the API, you can define a job with all scrape_config parameters including metric_relabel_config. Once the job is created and enabled, job configuations are automatically pushed to relevant agents in your infrastructure without requiring manual configuration updates or agent redeployment. Custom Prometheus jobs support cluster-wide definitions and also allow for selective enablement or disablement on a per-cluster level.

Custom Prometheus jobs must be created and then enabled for the job to start collecting metrics

Job Priority when Names Conflict

When multiple Prometheus jobs share the same name, Sysdig Monitor uses the following precedence rules to determine which job takes effect:

PrioritySourceDescription
1 (Highest)prometheus.yaml agent configurationJobs defined directly in the agent’s prometheus.yaml override all others.
2Metrics Disablement APIMetrics disabled via the Metrics Disablement API will continue to be dropped, even if a custom job attempts to collect them.
3Custom Prometheus Jobs (API)Jobs created via the custom Prometheus Job API will override Enabled-by-Defualt Integrations if they have the same job name.
4 (Lowest)Enabled-by-Default IntegrationsJobs automatically created by Sysdig’s built-in integrations. This includes the default k8s-pods job that scrapes any pod with the prometheus.io/scrape=true annotation.

Best Practices

  • Validate your configuration before pushing: Use promtool check config to validate your job definition locally. This helps catch syntax errors that could cause the Prometheus scrape process to fail on the agent. The API does not perform strict validation. Use caution when deploying changes as metrics collection can fail when pushing invalid changes.

  • Use unique and descriptive job names: Use clear naming conventions to avoid unintentional name collisions with existing Prometheus jobs, especially those from built-in integrations or local agent configs (prometheus.yaml).

  • Verify deployment using the Agent Console: After creating or updating a job, use the Agent Console command prometheus scrape_jobs show to inspect the final scrape configuration applied to each agent. This helps confirm targeting and relabeling logic are working as expected.

  • Test in a single cluster first: If possible, roll out and validate a new job in a single cluster before enabling it globally.

Access Next Gen API Docs

For further detail regarding API usage, review the Metrics Collection section in the Next Gen API docs.