SignalFx Developers Guide

Detectors, Events, and Alerts

Detectors watch incoming data for anomalous conditions specified by SignalFlow calculations and other settings. In response to an anomalous condition, detectors record an event, trigger an alert, and optionally send off notifications using third-party services. Detectors can also record events, alerts, and notifications when the anomalous condition clears (is no longer present).

Detectors

Specifically, detectors define the following:

  • A trigger condition, specified in a SignalFlow expression or program

  • An severity to set when the trigger condition occurs

  • Where and how notifications are sent

  • The content included in notifications

Detector actions

When SignalFx detects that a trigger condition exists, it does the following:

  • Generates an event

  • Sets off an alert

  • Sends one or more notifications to people to inform them of the alert

When SignalFx detects that the condition no longer exists, it does the following:

  • Generates a second event

  • Clears the alert

  • Sends a second set of notifications

You can view detectors in the web UI that you create using the API, by using the following URL pattern:

https://app.<REALM>.signalfx.com/#/detector/<DETECTOR_ID>/edit

The result is a chart of the trigger condition value over time, containing indicators that show when alerts were triggered and cleared.

Detectors also appear by name in the web UI Catalog.

Detector trigger conditions

Every detect() function must have a trigger or "on" condition. SignalFlow has three types:

Immediate conditions

Conditions that only care about the value of the input streams each time the condition is evaluated. As soon as condition x is true, SignalFlow triggers an alert.

Duration conditions. Monitor the value of the input streams for a specified duration prior to the point that the condition is evaluated. To define a duration condition, use when(<predicate>,<lasting>) function that triggers an alert if <predicate> is continuously true during duration <lasting>.

Percentage of duration conditions

Monitor the value of the input streams for a specified percentage of duration prior to the point that the condition is evaluated. To define a percentage of duration condition, use a when(<predicate>, <lasting>, <percent>) function that triggers an alert if <predicate> is true for <lasting> duration at least <percent> percent of the duration.

For more information about how to define conditions to trigger an alert, see the detect() function and the when() function documentation.

Clearing detectors

In general, an alert condition clears as soon as the trigger condiction in the detect() is no longer true; that is, as soon as the not condition is met. Each type of trigger condition has its own not condition criteria:

Evaluate immediately

Not condition is met as soon as x is false.

Duration trigger

Not condition is met as soon as <predicate> is false. Notice that <predicate> does not have to be false for the entire value of <lasting>.

Percentage trigger

Not condition isn’t met until <predicate> is false for longer than 100 - <percent> amount of <lasting>. For example, if <predicate> is cpu.utilization >= .50, <lasting> is 50 seconds, and percent is .50` (50%), the alert condition doesn’t clear until CPU utilization is less than 50 percent for 25 seconds.

If you don’t want any of these clear behaviors, you can set the clear condition as the second argument to detect(). For example, an alert condition that triggers when a > b continuously for the most recent 10 minutes clears as soon as a ⇐ b. If you prefer, you can require that a ⇐ b for more than 30 seconds before clearing the alert.

For more information about clear conditions and how they interact with trigger conditions, see the detect() function documentation.

To specify a detector using the API, you have to specify a SignalFlow program as one of the request properties. In general, SignalFlow program is simply one or more SignalFlow statements, but a SignalFlow detector program has specific requirements. The sections in this describe the SignalFlow program requirements for detectors.

Detector SignalFlow programs

The SignalFlow program for a detector must have:

  • One or more calls to detect().

  • A trigger condition for each detect() call.

  • A publish() call that publishes the results of the detector, with a label that’s unique within the program.

In addition, the program can optionally have:

  • Clear conditions

  • A mode designation that determines when and how the conditions are evaluated.

If you include more than one line of SignalFlow, terminate each line with either a semi-colon (";") or new line ("\n") characters.

Requirements for a detector SignalFlow program

The SignalFlow program must have at least one stream constructor that provides the base data. This is usually a call to the data() function.

You can use the resulting stream in the following ways:

  • Without change

  • Modified by one or more chained stream methods

  • Transformed into new streams using operators that perform a calculation on the input stream, such as returning the square root or natural log of input values, the mean of all current values, or dropping values that aren’t within a certain range.

After you modify and transform streams, you can construct boolean expressions from streams by comparing streams or by comparing streams to a constant value. You can further modify stream using when() functions if you you want to apply duration or percentage of duration conditions.

The result of these operations is a simple condition that you can use as it stands or combine with other simple conditions using the boolean logic operators and, or, and and not to construct a compound condition.

For example, this simple condition triggers an alert if the amount of deviation in CPU usage is high, using a base data stream modified by the stddev() function and compared to 2:

data('cpu.utilization').stddev() > 2

The following compound condition considers both the mean and the standard deviation. The base data stream is cpu.utilization, modified by stddev() function and mean() function. The resulting data streams are compared to constants, resulting in four boolean expressions that are combined into two. The overall expression is true if either of the two child expressions are true. This example also uses the when() function to create a duration requirement:

(data('cpu.utilization').stddev() > 3 and data('cpu.utilization').mean() > 50) or when(data('cpu.utilization').stddev() > 3 and data('cpu.utilization').mean() > 30,'5m')

SignalFlow calendar window transformations for detectors

Detector SignalFlow programs let you use calendar window transformations that calculate values over a time period specified as a clock or calendar value, such as an hour or a month.

SignalFlow provides calendar window transformations for the following methods:

To learn more about SignalFlow calendar window transformations, see Calendar window transformations.

Timezones and SignalFlow calendar window transformations

The following SignalFlow stream methods support calendar window transformations:

Calendar window transformations calculate values over a time period specified as a clock or calendar value, such as an hour or a month.

By default, SignalFlow interprets the time period relative to the time zone UTC. To change the time zone, use the timezone property when you create or update a detector. For example:

1
2
3
4
{
  "name": "European Detector"
  "timezone": "Europe/London"
}

You can see a list of permitted time zone values at Time zone values for calendar window transformations

Requirements for custom notifications

To support custom notification messages that include input data, you have to assign the data to variables in the detector trigger and clear conditions in the SignalFlow program. You can then retrieve the data using the variables.

For example, this SignalFlow expression creates a valid detector:

detect(data('cpu.utilization').mean() > 50).publish('highCpu')

If you want to include the mean CPU utilization value in notifications when the alert is triggered, you have to use this equivalent SignalFlow instead:

highMean = data('cpu.utilization').mean(); detect(highMean > 50).publish('highCpu')

This lets you refer to the value of highMean in your custom notification message definition.

Working with nulls and missing data

Data points in a stream evaluate to null for several reasons, such as performing an operation that results in dividing by 0. In addition, data points in a stream get delayed or dropped. When any of these conditions occur, SignalFx defaults to replacing the data with a null after waiting for the duration specified in the maxDelay property of the detector.

When SignalFlow evaluates a stream to determine if it should trigger or clear an alert, the null values aren’t considered valid, but they’re also not considered anomalous. Instead, for conditions that require specific criteria to be met for a set duration, SignalFlow resets and restarts the clock.

To avoid this behavior, set an explicit value for missing data using one of these common methods:

Last value extrapolation

Set the extrapolation for each call to data() to last_value. This option tells SignalFlow to wait for maxDelay to expire, and then sets missing data points to the value that immediately preceded the gap. As a result, a single missing data point doesn’t interrupt the duration measurement for the condition. By default, an extrapolation value of last_value causes SignalFlow to set up to 100 consecutive missing data points. To set fewer data points, set the maxExtrapolation argument. As soon as SignalFlow receives a valid data point, it resets the maximum number of extrapolations. For example, the following SignalFlow program lets a detector ignore up to three consecutive missing data points before resetting the time for ongoing durations:

detect(when(data('cpu.utilization', extrapolation='last_value',maxExtrapolations=3).mean() > 50),'10m')).publish('highCpu')

fill stream

This method is similar to last value extrapolation, but it lets you set an explicit replacement value for missing data points and uses a time-based rather than a number-based limitation to determine when to stop replacing null values. After maxDelay expires, SignalFlow replaces null values with the value in the fill stream method call for the duration specified in the call. SignalFlow resets the fill duration whenever it receives valid data. Using the fill stream method lets you ensure that the fill value matches or doesn’t match the detector condition. For example, the following SignalFlow statement sets any missing values to 75 for 15 seconds:

detect(when(data('cpu.utilization').fill(value=75,duration='15s').mean() > 50, '10m')).publish('highCpu')

Detector data resolutions

Detectors have two distinct types of resolutions:

Detector job resolution

The interval at which SignalFlow analyzes data to determine if it should trigger an alert. The type of data being analyzed and the transformations applied to the data determine this resolution. It remains constant throughout the life of the detector.

Data display resolution

Rate at which data populates the detector visualization. SignalFx sets this resolution to the coarsest resolution among all the publish() function calls associated with the detector, because all the results are displayed in the same visualization. The data display resolution may change when someone modifies the time window for the display.

Rules control how triggered and cleared alerts are processed. Each detect function is mapped to a severity and a set of notifications using the unique label inside the associated publish method.

Alert severity

Severity indicates how impactful a triggered alert might be. Alerts in the web application can be easily filtered by severity, allowing you to focus on the most important alerts first. SignalFx supports five severity levels: Critical, Warning, Major, Minor, and Info.

You are free to define your own meaning for each level of severity and judge which level is appropriate for different triggering conditions.

Notifications

Each rule can include one or more notification definitions indicating where and how to send notifications.

Notification recipients

Notifications can be sent to individual users or to teams and may be send via email, using a third-party messaging system, or using a third-party incident management system.

You can check if your organization has the desired integration available using the Get Integrations API call with a type query parameter set to the desired notification type. If so, you’ll need the ID of the relevant integration object to set up notifications using that integration; the same API call will return its value. If not, contact your local SignalFx administrator to set up the necessary integration.

Custom notification messages

If you prefer, you can specify custom notification messages. They are only available via the API and only for v2 detectors. Also, you can’t display or modify them in the web application.

Custom messages include separate subject and body sections, specified as parameterizedSubject and parameterizedBody properties when you create a rule for the notification. All notification types accept plain text containing valid ASCII characters, as well as any of the variables provided by SignalFx. Some notification types also render Markdown text in their messages. See the sections below for more information.

Variables in custom notification messages

Custom notification variable contain the following types of information: * Information about the detector * Current state of the detector at the time an alert is triggered or cleared * Detector job data including conditions and dimensions * Detector program information

Insert custom notification variables in a message definitions by inserting the variable name surrounded by curly braces { and }:

  • Double curly braces indicate a variable the API substitutes in place as is. Some characters in the value may be interpreted by the API rather than passed through and rendered in the output.

  • Triple curly braces indicate a variable for which the API escapes characters as needed. For example, use triple braces to escape quotation marks and angle brackets.

The following list is a sample of the available variables. The complete list is available in the :

  • detectorName - the name of the detector (as specified in the name property)

  • detectorId - the ID of the detector (as specified in the id property); permits notification recipient to use API calls to obtain further information about the detector

  • ruleSeverity - the severity defined for the rule (as specified in the rules[x].severity property corresponding to the rule controlling the notification)

  • runbookUrl - a link to more information about how to process the notification (as specified in the rules[x].runbookUrl property corresponding to the rule controlling the notification)

  • tip - a quick first step to try upon receipt of the notification (as specified in the rules[x].tip property corresponding to the rule controlling the notification)

  • anomalous - true if the alert is currently triggered

  • normal - true if the alert is currently cleared

  • #if, else, /if - provides the ability to conditionalize text; most useful for providing alternate text in alert triggered and alert cleared messages

  • inputs.variable.value - the value of the detector condition indicated by the specified variable corresponding to a defined data stream. See the SignalFlow Syntax section above for more information.

A more complete list of supported variables is available in the table

in the production documentation.

Markdown support in custom notification messages

GitHub-flavored Markdown is also supported in the body of custom notification messages, but not in the message subject or message tip. However, it may or may not be rendered depending on the intended output format of the messages. Markdown isn’t supported in notifications sent to any third-party incident management system or in notifications sent to third-party messaging systems other than Slack.

Email notifications support all Markdown formatting except tables. This includes:

  • Character styles.

  • Headers on a single line.

  • Links that use the syntax [description](url).

  • Images that use the syntax ![alt text](image-url).

  • Ordered and unordered lists.

  • Horizontal rules. Three or more consecutive underscores, hyphens, or asterisks on their own line render as a horizontal rule.

Slack notifications have limited markdown support: * Character styles are supported * Header notation is stripped out * The link constructs for URLs and images are replaced by the URL itself. * Unordered lists are displayed without the bullets * Ordered lists are statically numbered. * Horizontal rules are removed entirely.

Markup is stripped out entirely from other types of notifications: * [text](url) is rendered as text url * ![alt](image-url)] is rendered as alt image-url. * Headers and character styles are ignored; the notation symbols are removed. notation symbols removed. * Lists are rendered as in Slack. * Horizontal rules are removed entirely.

Examples

Create a detector

You can create and manage detectors using REST API calls. For example, to create a detector which monitors the jvm.cpu.load metric and notifies person@example.org when it crosses a static threshold of 60, run the following curl command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
$ curl --request POST \
    --header "X-SF-TOKEN:YOUR_SFX_TOKEN" \
    --header "Content-Type: application/json" \
    --data \
    '{
        "name": "CPU load too high",
        "programText": "detect(data('jvm.cpu.load') > 60).publish('Load above 60%)",
        "rules": [
            {
                "severity": "Critical",
                "detectLabel": "Load above 60%",
                "notifications": [
                    "type": "Email",
                    "email": "person@example.org" }
        ]
    }'
    https://api.<REALM>.signalfx.com/v2/detector

This call returns the created detector specification as a JSON object:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
    "created": 1470704945599,
    "creator": "ABCDEFGHIJ",
    "customProperties": {},
    "description": null,
    "id": "ABCDEFGHIJ",
    "lastUpdated": 1470705776335,
    "lastUpdatedBy": "ABCDEFGHIJ",
    "maxDelay": null,
    "name": "CPU load too high",
    "programText": "detect(data('jvm.cpu.load') > 60).publish('Load above 60%')",
    "rules": [
        {
            "description": null,
            "detectLabel": "Load above 60%",
            "disabled": false,
            "notifications": [
                {
                    "type": "Email",
                    "email": "person@example.org"
                }
            ],
            "severity": "Critical"
        }
    ],
    "tags": [],
    "visualizationOptions": null
}

Enable or disable a detector

By default, detectors you create are enabled, so they trigger an alert when their conditions are met.

You can disable and then re-enable a detector using the API.

Disable a detector

To disable a detector, use the operation PUT https://api.{REALM}.signalfx.com/v2/detector/{DETECTOR_ID}/disable. The value of DETECTOR_ID is returned in the response to the operation POST to https://api.{REALM}.signalfx.com/v2/detector that creates the detector.

For example:

1
2
3
4
5
6
$ curl \
    --request PUT \
    --header "X-SF-TOKEN: YOUR_SFX_TOKEN" \
    --header "Content-Type: application/json" \
    --data '["Load above 60%"]' \
    https://api.<REALM>.signalfx.com/v2/detector/<DETECTOR_ID>/disable

Enable a detector

To re-enable a disabled detector, use the operation PUT https://api.{REALM}.signalfx.com/v2/detector/{DETECTOR_ID}/enable. The value of {DETECTOR_ID} is returned in the response to the operation POST https://api.{REALM}.signalfx.com/v2/detector that creates the detector.

1
2
3
4
5
6
$ curl \
    --request PUT \
    --header "X-SF-TOKEN: YOUR_SFX_TOKEN" \
    --header "Content-Type: application/json" \
    --data '["Load above 60%"]' \
    https://api.<REALM>.signalfx.com/v2/detector/<DETECTOR_ID>/enable

Read, update, or delete a detector

You can also read, update, and delete detectors:

Detector events and incidents

You can retrieve the events and incidents that a detector has produced using the operations GET https://api.{REALM}.signalfx.com/v2/detector/{DETECTOR_ID}/events and GET** https://api.{REALM}.signalfx.com/v2/detector/{DETECTOR_ID}/incidents.

The following curl command demonstrates how to find active incidents (alerts) for a detector:

1
2
3
curl \
    --header "X-SF-TOKEN: YOUR_SFX_TOKEN" \
    https://api.<REALM>.signalfx.com/v2/detector/<DETECTOR_ID>/incidents

To manually clear an incident, use the operation GET https://api.{REALM}.signalfx.com/v2/detector/incident/{INCIDENT_ID}/clear endpoint. For example:

1
2
3
4
5
curl \
    --request PUT \
    --header "X-SF-TOKEN: YOUR_SFX_TOKEN" \
    --header "Content-Type: application/json" \
    https://api.<REALM>.signalfx.com/v2/incident/<INCIDENT_ID>/clear

Detector notifications

For example, you can specify a detector that checks for mean CPU usage over 60%. When it’s triggered, the detector sends alerts with Major severity to both a Slack channel and the "on call" email. Create the detector using the following request body:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
{
    "name":"High Mean CPU",
    "programText":"A = data('cpu.utilization').mean(); detect(A > 60).publish('highMeanCpu')",
    "tags":["CPU"],
    "rules":[{
        "detectLabel":"highMeanCpu",
        "notifications":[{
            "type":"Email",
            "email":"oncall@example.com"
        },{
            "type":"Slack",
            "credentialId":"_id_",
            "channel":"detector-alerts"
        }
        ],
        "runbookUrl":"http://runbook.example.com",
        "tip":"Add more machines!",
        "severity":"Major",
        "parameterizedSubject": "{{ruleSeverity}} Alert: {{{ruleName}}} {{{detectorName}}}",
        "parameterizedBody": "{{#if anomalous}}\nRule \"{{{ruleName}}}\" in detector \"{{{detectorName}}}\" triggered\n{{else}}\nRule \"{{{ruleName}}}\" in detector \"{{{detectorName}}}\" cleared{{/if}}\n\n{{#if anomalous}}\nTriggering condition: CPU utilization mean > 60\n{{/if}}\n\n{{#if anomalous}}Signal value: {{inputs.A.value}}\n{{else}}Current signal value: {{inputs.A.value}}\n{{/if}}\n\n{{#notEmpty dimensions}}\nSignal details:\n{{{dimensions}}}\n{{/notEmpty}}\n\n{{#if anomalous}}{{#if runbookUrl}}Runbook: {{{runbookUrl}}}{{/if}}\n\n{{#if tip}}Tip: {{{tip}}}{{/if}}\n{{/if}}"
  }]
}
When the alert triggers, you see the following notification delivered to the "on call" email address: email:

Alert triggered notification with custom email message
Alert triggered notification with custom email message

The alert also delivers the following message to Slack:

Alert triggered notification with custom Slack message
Alert triggered notification with custom Slack message

When you click the message, you see the following expanded text:

Alert triggered notification with expanded custom Slack message
Alert triggered notification with expanded custom Slack message

When the detector condition is no longer met, the alert "clears" and its status reverts to OK. You now see the following notification delivered to the on call email address: email

Alert cleared notification with custom email message
Alert cleared notification with custom email message

The corresponding notification appears in Slack:

Alert cleared notification with custom Slack message
Alert cleared notification with custom Slack message

Custom notification messages

To specify a detector with a simple custom notification message containing markdown, create a new detector with JSON that’s similar to the following request body:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
{
    "name":"Low Mean CPU - Minor Alert",
    "programText":"A = data('cpu.utilization').mean(); detect(A < 15).publish('lowMeanCpu')",
    "tags":["CPU"],
    "rules":[{
        "detectLabel":"lowMeanCpu",
        "notifications":[{
            "type":"Email",
            "email":"oncall@example.com"
        },{
            "type":"Slack",
            "credentialId":"_id_",
            "channel":"detector-alerts"
        }
        ],
        "tip":"Consolidate Machines if possible to save money",
        "severity":"Minor",
    "parameterizedBody": "{{#if anomalous}}\n *Low CPU Usage* triggered \n\n {{#if tip}}Tip: {{{tip}}}{{/if}}\n{{/if}}"
  }]
}

The notification message in email has the following form:

Email alert with custom Markdown
Email alert with custom Markdown

In Slack, the notification message has this form:

Slack alert with custom Markdown
Slack alert with custom Markdown

Full detector example

Create the stream

Streams are instantiated with the data() SignalFlow function, which takes as its main argument a query that select the metric time series you are interested in. The query string supports non-leading wildcards with the * character. You can also specify the optional filter argument. For example:

1
2
3
4
5
6
7
8
9
data('cpu.utilization').publish()
data('cpu.*').publish()

# With a filter
data('cpu.utilization', filter=filter('env', 'QA') and filter('serverType', 'API')).publish()

# Specifies the filter as a separate variable
qa_api_servers = filter('env', 'QA') and filter('serverType', 'API')
data('cpu.utilization', filter=qa_api_servers).publish()"

Apply analytics to the stream

To make calculations on the data in the stream, call analytics methods on it:

1
2
# Find the p90 of the 1h mean of total cache hits by datacenter
data('cache.hits').sum(by='datacenter').mean(over='1h').percentile(90)

Create a detector

Detectors are functions that continually examine a stream, looking for a specific condition you specify. When they detect the condition in the stream, they issue an alert and send out notifications. They can send notifications to incident management platforms such as PagerDuty or messaging systems such as Slack or email, or both.

To create a detector, call detect() on a data stream that you publish. For example:

1
2
# Send events when cpu.utilization is above 50 (and when it falls below again)
detect(data('cpu.utilization') > 50).publish('cpu_too_high')

Specify detector conditions

You specify conditions with predicates, which are expressions that evaluate to true or false. The input to a predicate is a data stream or streams; the predicate compares each value in the stream against a threshold value. If the comparison evaluates to true, then SignalFlow replaces the input stream value with True (Python style); otherwise it is replaced with False. If the result is True, SignalFlow keeps the metadata for the input value.

To combine simple predicates into more complex expressions, use the logical operators and, or, and not. You can also use parentheses ( ) to isolate an expression and evaluate it before any outer expressions.

The following code shows you some examples of SignalFlow predicates for detectors

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# True when any cpu.utilization timeseries' value is greater than 50
data('cpu.utilization') > 50

# True when cpu.utilization greater than 30 and memory.utilization less than 40
data('cpu.utilization') > 50 and data('memory.utilization') < 40

# Complex example of a condition. Checking if the moving average of the memory
# utilization is greater than 2 times the stddev
mem = data('memory.utilization')
mem_mean = mem.mean(over='1h')
mem_stddev = mem.stddev()mem_mean > 2 * mem_stddev

Use when() to specify predicate durations

By default, the state of the predicate changes when its predicate value changes from False to True or from True to False. To alter this behavior, use the when() function to require that the condition is true for a specified amount of time or a percentage of a specified amount of time. Use duration controls to create detectors that ignore temporary spikes. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Returns True at the moment cpu.utilization is above 50
# Returns False at the moment cpu.utilization is less than or equal to 50
when(data('cpu.utilization') > 50)

# Returns True when cpu.utilization has been above 50 for 5 minutes
# continuously;
# Returns False if cpu.utilization drops below 50 at
# any point during the next 5 minutes, that is, even if it dips instantaneously
# below 50.
when(data('cpu.utilization') > 50, '5m')

# Returns True when cpu.utilization has been above 50 for 75% of 5 minutes
# continuously; will generate False if the cpu.utilization drops below 50 for
# more than 25% of 5 minutes.
when(data('cpu.utilization') > 50, '5m', .75)"

Conditions are like filters, because you can specify one without using it. To use it, you have to pass its value to detect().

Specify a detector using detect()

The detect() function takes the output of predicate objects as inputs and sends out events based on the predicate states and the specific arguments you pass to detect().

The detect() function has this syntax:

detect(<on_predicate>, off=<off_predicate>, mode=<evaluation_mode>

  • The only required argument is <on_predicate>, which specifies a predicate that causes the Anomalous event to fire.

  • The optional off=<off_predicate> specifies a predicate that causes the Ok event to fire.

  • The optional mode=<evaluation_mode> controls how detect() evaluates predicates.

Using <on_predicate> and off=<off_predicate>

Table 1. Result of detect() for different number of arguments
Call Argument meaning Result

detect(<on_predicate>)

Specifies when to fire Anomalous

* When <on_predicate> changes from False to True, detect() fires the Anomalous event * When <on_predicate> changes from True to False, detect() fires the Ok event

detect(<on_predicate>, off=<off_predicate>)

Specifies when to fire Anomalous and when to fire Ok

* When <on_predicate> changes from False to True, detect() fires the Anomalous event * When <off_predicate> changes from False to True, detect() fires the Ok event This form lets you specify separate conditions for sending the events

detect(<on_predicate>, off=<off_predicate>, mode=<evaluation_mode>)

Controls how detect() evaluates predicates

* paired: If <on_predicate> is True and <off_predicate> is False simultaneously, detect() fires Anomalous; otherwise it doesn’t fire an alert. * split: detect() only evaluates <on_predicate> if no alert is currently set. If <on_predicate> is true, detect() fires Anomalous. Similarly, detect() only evaluates <off_predicate> if an alert is currently set. If <off_predicate> is true, detect() fires Ok.

Publish detect() events

To send events, you need to call publish() on the results of detect(). For example:

1
2
3
4
5
6
7
8
# Send events when cpu.utilization is above 50 (and when it falls below again)
detect(data('cpu.utilization') > 50).publish('cpu_too_high')

# Send events when cpu.utilization is above 50C
# Clear events when the cpu.utilization is below 40 and memory.utilization is below 20 for 5 minutes
cpu = data('cpu.utilization')
mem = data('memory.utilization')
detect(cpu > 50, when(cpu < 40 and mem < 20,'5m')).publish('cpu_too_high')

© Copyright 2019 SignalFx.

Third-party license information