SignalFx Developers Guide

Detectors, Events, and Notifications

Detectors watch incoming data for anomalous conditions specified by SignalFlow calculations and other settings. In response to an event, detectors record an event, trigger an alert, and optionally send off notifications using third- party services. Detectors can also record events, alerts, and notifications when the initial condition is no longer present.

The topic Detectors, Events, and Alerts provides a detailed description of detectors, events, and notifications.

You can also create detectors that monitor µAPM metrics and alert you when they detect anomalies. To learn more about µAPM detectors, see the section Microservices APM detectors.

Detecting anomalous conditions with SignalFlow programs

The SignalFlow program for a detector defines a trigger condition based on MTS values. When SignalFx determines that the trigger condition is met, it generates an event, sends out an alert, and issues a notification.

To learn more about SignalFlow programs, see the section "SignalFlow programming language" in the topic Using SignalFlow.

Events, alerts and notifications

Events

Events provide a long-term record of anomalous conditions. Detectors record events, but you can also send your own custom events to SignalFx. For example:

  • SignalFx records an event when monitoring data meets an anomalous condition defined in a detector.

  • You can record an event when you change a server, upgrade an app, and so forth.

  • SignalFx records an event when the trigger condition is no longer true.

Alerts

An alert is an indicator that incoming data has triggered one of your detectors. The alert remains "on" until you clear it or the incoming data no longer exhibits the triggering condition. For example, if you have a detector with a trigger condition for cpu utilization above 50% for an MTS, SignalFx sets an alert when the MTS cpu utilization goes above 50%, and clears the alert when it drops below 50%.

Notifications

Detectors can send notifications using third-party systems such as PagerDuty and Slack. You can define custom messages for a detector notification, and these messages can include placeholders that SignalFx replaces with actual text when it sends the notification.

Clearing anomalous conditions

When the detector’s trigger condition is no longer true, SignalFx records a "clear" event, clears the alert associated with the event, and sends out another notification.

Incidents

Incidents are collections of related events generated by detectors. Each incident has a triggering (or "alerting") event that logs alert notifications sent to notification recipients and a clearing (or "resolving") event that logs the "ok" notification sent to the same recipients.

© Copyright 2019 SignalFx.

Third-party license information