SignalFx

Alerting Overview

Detectors in SignalFx monitor signals (metric time series) for conditions or issues that you care about. Those conditions or issues are expressed as one or more rules that trigger alerts.

API supports only v2 detectors

Detector APIs are supported only for v2 detectors. A v2 detector is one that was created using the SignalFx v2 API.

V2 detectors can be created and managed via the API. For example, to create a detector which monitors the jvm.cpu.load metric and notifies person@example.org when it crosses a static threshold of 60, you can execute the following command in your terminal:

$ curl \
  --request POST \
  --header "X-SF-TOKEN: $SFX_TOKEN" \
  --header "Content-Type: application/json" \
  --data-binary @- \
  https://api.signalfx.com/v2/detector << EOF
{
  "name": "CPU load too high",
  "programText": "detect(data('jvm.cpu.load') > 60).publish('Load above 60%)",
  "rules": [{
    "severity": "Critical",
    "detectLabel": "Load above 60%",
    "notifications": [{ "type": "Email", "email": "person@example.org" }]
  }]
}
EOF

The created detector will be returned as a JSON object:

{
    "created": 1470704945599,
    "creator": "ABCDEFGHIJ",
    "customProperties": {},
    "description": null,
    "id": "ABCDEFGHIJ",
    "lastUpdated": 1470705776335,
    "lastUpdatedBy": "ABCDEFGHIJ",
    "maxDelay": null,
    "name": "CPU load too high",
    "programText": "detect(data('jvm.cpu.load') > 60).publish('Load above 60%')",
    "rules": [
        {
            "description": null,
            "detectLabel": "Load above 60%",
            "disabled": false,
            "notifications": [
                {
                    "type": "Email",
                    "email": "person@example.org"
                }
            ],
            "severity": "Critical"
        }
    ],
    "tags": [],
    "visualizationOptions": null
}

Now, by using the id field in this response, we can enable or disable this detector by calling /v2/detector/:id/enable or /v2/detector/:id/disable:

curl \
  --request PUT \
  --header "X-SF-TOKEN: $SFX_TOKEN" \
  --header "Content-Type: application/json" \
  --data '["Load above 60%"]' \
  https://api.signalfx.com/v2/detector/ABCDEFGHIJ/disable

You can also read, update, and delete the detector by issuing GET, PUT or DELETE against the /v2/detector/:id endpoint.

Detector Events and Incidents

The events and incidents that a detector produces can be queries using the /v2/detector/:id/events and /v2/detector/:id/incidents. For example, to find active incidents (alerts) for a detector:

curl \
  --header "X-SF-TOKEN: $SFX_TOKEN" \
  https://api.signalfx.com/v2/detector/ABCDEFGHIJ/incidents

To manually clear an incident, call the /v2/incident/:id/clear endpoint:

curl \
  --request PUT \
  --header "X-SF-TOKEN: $SFX_TOKEN" \
  --header "Content-Type: application/json" \
  https://api.signalfx.com/v2/incident/ABCDE1234/clear

Detector Templates

Alternatively, detectors can also be created by using Detector Templates. Detector Templates allow you to predefine analytics for detectors using the SignalFx web UI, while allowing you to vary metrics, filters, and rules via the API when instantiating detectors.

Detector templates are used to configure a dynamic endpoint specific to the template. This means the parameters for the endpoint will vary by how the template has been configured. Learn more about how to create detector templates in the web UI.

Alerting Overview