SignalFlow Stream Messages Reference

Realms in endpoints

When making a request to the REST API, you need to use the endpoint specific to your SignalFx realm.

SignalFlow client libraries

SignalFx provides open-source, language-specific client libraries that encapsulate the following tasks:

Choosing the best connection method (HTTP or WebSocket) to use
Parsing and decoding the stream of the messages coming from your SignalFlow computations
High-level abstractions that simplify and speed up the process of building SignalFlow analytics applications

In most cases, you should use a client library rather than writing your own low-level connection, message processing, and request routines. The information in this topic is provide for developers who want to handle SignalFlow requests and responses at a low level.

You can receive messages using either the SSE (Server-Sent Event) or WebSocket protocol. The format of the message you receive depends on the protocol you use.

The following are links to the client libraries in GitHub:

Receiving messages via Server-Sent Events

If you send a computation to SignalFlow using the HTTP REST API, SignalFlow sends you response messages using the Server-Sent Event (SSE) protocol. The content-type is text/event-stream. Each SSE message has the name event, which is also the message type, and a data payload formatted in JSON. In accordance with the SSE standard, SignalFlow puts an empty new line between each event.

Receiving messages via WebSocket

If you send a computation to SignalFlow using a WebSocket connection to SignalFx, SignalFlow sends you response messages in either JSON text or binary format. The format SignalFlow uses depends on these factors:

Message type: The messageType SignalFlow assigns to the message. Data messages are binary-encoded, but all other messages use JSON.
Message compression settings: Whether you requested message compression when you connected to a channel using WebSocket.

SignalFlow identifies all WebSocket response messages with the channel value you specify when you make an execute request. SignalFlow also identifies the type of message in one of the top-level message properties. In all other respects, WebSocket response messages formatted in JSON have the same structure as SSE payloads.

Message compression

When you request message compression, SignalFlow attempts to compress response messages for the channel and client that sent the request. If the compressed message is smaller, SignalFlow sends it as a compressed binary message. The base format of this message is JSON except for data messages, which use the data message format.

Response message examples

SSE response messages

The following JSON contains the first SSE control messages you usually see when you request a SignalFlow computation using HTTP:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


event: control-message
data: {
data:   "event" : "STREAM_START",
data:   "timestampMs" : 1461351769440
data: }

event: control-message
data: {
data:   "event" : "JOB_START",
data:   "handle" : "ChkVxy0AEAA",
data:   "timestampMs" : 1461351771160
data: }

event: control-message
data: {
data:   "event" : "JOB_PROGRESS",
data:   "timestampMs" : 1461351771168,
data:   "progress" : 10
data: }

WebSocket response messages

The following JSON contains the first SSE control messages you usually see when you request a SignalFlow computation using HTTP:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{
  "type" : "control-message",
  "channel" : "channel-1",
  "event" : "STREAM_START",
  "timestampMs" : 1461351769440
}

{
  "type" : "control-message",
  "channel" : "channel-1",
  "event" : "JOB_START",
  "handle" : "ChkVxy0AEAA",
  "timestampMs" : 1461351771160
}

{
  "type" : "control-message",
  "channel" : "channel-1",
  "event" : "JOB_PROGRESS",
  "timestampMs" : 1461351771168,
  "progress" : 10
}

Binary encoding of WebSocket messages

Binary messages coming from a SignalFlow WebSocket connection have a 4-byte preamble followed by the 16-byte ASCII channel name.

All binary messages use big endian encoding.

This text graphic shows the layout of the first 20 bytes of a WebSocket message:

0 1 2 3 4 5 6 7 8 A B C D E F 0 1 2 3 4 5 6 7 8 A B C D E F
+-------------+-------------+---------------+-------------+
| Version     | Message type| Flags         | Reserved    |
+---------------------------------------------------------+
|   Channel name (bytes 0-3)                              |
+---------------------------------------------------------+
|   Channel name (bytes 4-7)                              |
+---------------------------------------------------------+
|   Channel name (bytes 8-11)                             |
+---------------------------------------------------------+
|   Channel name (bytes 12-15)                            |
+---------------------------------------------------------+

Version: Always 1
Message type: Always 5 (data batch) for binary-encoded messages
Flags: Bitfield that contains the following bit flags:
- 000000x0 : message payload isn’t compressed.
- 000000x1 : message payload is compressed, so you have to decompress it with a tool such as gzip -d.
- 0000001x : message payload is UTF-8 encoded JSON
- 0000000x : message payload is binary SignalFlow only compresses data messages. If the compressed payload is smaller, it’s sent compressed and SignalFlow sets the compressed bit; otherwise, it’s sent uncompressed and SignalFlow leaves the compressed bit unset.

Message types

The following sections describe each response message type in more detail.

Control messages

Control messages provide information about the status and progress of a SignalFlow computation. Each control message has an event property and a timestampMs property. The event defines the type of control message ("what happened"), while timestampMs contains a timestamp in Unix time format noting the time the event occurred.

SSE example

1
2
3
4
5


event: control-message
data: {
data:   "event" : "...",
data:   ...
data: }

WebSocket example

1
2
3
4


  "type" : "control-message",
  "channel" : "channel-1",
  "event" : "...",
  ...

`STREAM_START` control message

STREAM_START indicates that SignalFlow has opened the response stream and started the computation.

SSE example

1
2
3
4
5


event: control-message
data: {
data:   "event" : "STREAM_START",
data:   "timestampMs" : 1461360399704
data: }

WebSocket example

1
2
3
4
5
6


{
  "type" : "control-message",
  "channel" : "channel-1",
  "event": "STREAM_START",
  "timestampMs" : 1461360399704
}

`JOB_START` control message

JOB_START indicates that the SignalFlow computation is initialized and is starting to process data. The message contains the computation handle ID, which you can use to control the computation using request messages such as keep-alive, stop, and so forth.

SSE example

1
2
3
4
5
6


event: control-message
data: {
data:   "event" : "JOB_START",
data:   "handle" : "ChkVxy0AEAA",
data:   "timestampMs" : 1461360399772
data: }

WebSocket example

1
2
3
4
5
6
7


{
  "type" : "control-message",
  "channel" : "channel-1",
  "event" : "JOB_START",
  "handle" : "ChkVxy0AEAA",
  "timestampMs" : 1461360399772
}

`JOB_PROGRESS` control message

If your computation contains a moving transformation such as a one-hour moving average, SignalFlow has to feed initialization data into the computation to seed the data window. As a result, it may take longer for SignalFlow to generate the first data batch, especially if the data window is large. In this situation, SignalFlow sends JOB_PROGRESS control messages during the seeding phase.

The control message contains the progress property, which is an integer between 0 and 100 inclusive representing the percent completion of the operation. SignalFlow usually sends the message after it has completed an additional 10% of the operation.

SSE example

1
2
3
4
5
6


event: control-message
data: {
data:   "event" : "JOB_PROGRESS",
data:   "timestampMs" : 1461360399784,
data:   "progress" : 10
data: }

WebSocket example

1
2
3
4
5
6
7


{
  "type" : "control-message",
  "channel" : "channel-1",
  "event" : "JOB_PROGRESS",
  "timestampMs" : 1461360399784,
  "progress" : 10
}

`CHANNEL_ABORT` control message

If your computation fails, SignalFlow sends you the CHANNEL_ABORT message. This is the last message you receive for the computation.

The message contains an abortInfo object property that contains child properties that provide additional information about the error cause. The following table describes these properties:

Table 1. `abortInfo` child properties
Property	Description
`sf_job_abortReason`	Why SignalFlow aborted the computation job
`sf_job_abortState	Code that indicates the current state of the computation

SSE example

1
2
3
4
5
6
7
8
9


event: control-message
data: {
data:   "event" : "CHANNEL_ABORT",
data:   "timestampMs" : 1461360400235,
data:   "abortInfo" : {
data:     "sf_job_abortReason" : "job expired",
data:     "sf_job_abortState" : "EXPIRED"
data:   }
data: }

WebSocket example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
  "type" : "control-message",
  "channel" : "channel-1",
  "event" : "CHANNEL_ABORT",
  "timestampMs" : 1461360400235,
  "abortInfo" : {
    "sf_job_abortReason" : "job expired",
    "sf_job_abortState" : "EXPIRED"
  }
}

`END_OF_CHANNEL` control message

You receive END_OF_CHANNEL when your computation reaches the end of its execution time range. This only happens when you explicitly specify a stop time. This mesage is the last you receive for the computation.

SSE example

1
2
3
4
5


event: control-message
data: {
data:   "event" : "END_OF_CHANNEL",
data:   "timestampMs" : 1461360400235
data: }

WebSocket example

1
2
3
4
5
6


{
  "type" : "control-message",
  "channel" : "channel-1",
  "event" : "END_OF_CHANNEL",
  "timestampMs" : 1461360400235
}

Information messages

During the initialization and execution of your computation, SignalFlow captures about decisions taken by the system, anomalies, and other conditions of interest. SignalFlow sends these messages during the computation, and in particular at the end of the first compute iteration. SignalFlow also sends messages at the end of the last compute iteration if you set an explicit stop time..

The reference documentation for all the information messages is in the SignalFlow Information Messages Reference.

Metadata messages

Metadata messages contain the metadata from the output time series of your computation. Each message contains the metadata for a single time series, identified by the tsId property of the message. The message also contains a properties object property that contains the metadata. The properties object contains these child properties:

Table 2. Metadata message child properties
Property	Description
`jobId`	Identifier assigned by SignalFlow to the job that’s running the computation
`sf_OrganizationId`	The organization identifier for the access token used to send the SignalFlow API request. This is either the organization that owns the org token, or the organization that the owner of the user token belongs to
`sf_key`	An array of property keys that identify the computation
`sf_metric`	For the metric time series that’s the source of data for this computation, `sf_metric` is the name of the metric.
`sf_resolutionMs`	The resolution of the computation, in milliseconds. The section More about resolution in the SignalFlow Overview topic describes how SignalFlow determines this resolution.
`sf_type`	The type of the incoming data, for example "Metric Time Series"
`sf_isPreQuantized`	For SignalFx internal use

The following code contains examples of metadata messages in SSE and WebSocket format:

Metadata message in SSE format

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


event: metadata
data: {
data:   "tsId" : "CgrT2EkAAAA",
data:   "properties" : {
data:     "jobId" : "CgrS6A4AIAM",
data:     "sf_organizationID" : "BqDQY5OAAAA",
data:     "sf_key" : [ "jobId", "sf_metric" ],
data:     "sf_metric" : "mean_latency",
data:     "sf_resolutionMs" : 1000,
data:     "sf_type" : "MetricTimeSeries",
data:     "sf_isPreQuantized" : true
data:   }
data: }

Metadata message in WebSocket format

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
  "type" : "metadata",
  "channel" : "channel-1",
  "tsId" : "CgrT2EkAAAA",
  "properties" : {
    "jobId" : "CgrS6A4AIAM",
    "sf_organizationID" : "BqDQY5OAAAA",
    "sf_key" : [ "jobId", "sf_metric" ],
    "sf_metric" : "mean_latency",
    "sf_resolutionMs" : 1000,
    "sf_type" : "MetricTimeSeries",
    "sf_isPreQuantized" : true
  }
}

Metadata about a particular time series is always emitted before any data for that time series.

Expired `tsId` messages

Expired TSID messages indicate that a specific output timeseries is probably no longer useful for the computation. This message helps a client that maintains a repository of received metadata, organized by tsId, to remove unnecessary entries.

SignalFlow sends this message if the computation has not sent any data for this output timeseries for a long time. In this case, the timeseries itself is no longer relevant. If the same timeseries reappears, SignalFlow sends a new metadata message to describe it again.

Expired `tsId` message in SSE format

1
2
3
4


event: expired-tsid
data: {
data:   "tsId" : "CgrT2EkAAAA"
data: }

Expired `tsId` message in WebSocket format

1
2
3
4
5


{
  "type" : "expired-tsid",
  "channel" : "channel-1",
  "tsId" : "CgrT2EkAAAA"
}

Data messages

Data messages contain the actual timeseries results generated by the computation. In general, SignalFlow issues a data message for each compute iteration, which occurs at the end of each period defined by the resolution of the computation. If you call publish() multiple times for a computation, you get a separate data message for each compute iteration for each publish() call.

Each data message contains the logical timestamp for the data and a list of datapoints identified by a timeseries identifier.

SSE example

The following listing is an example of a data message in SSE format. See the next section for the WebSocket binary format.

1
2
3
4
5
6
7
8
9


event: data
id: data-1461353198000
data: {
data:   "data" : [ {
data:     "tsId" : "CgrT2EkAAAA",
data:     "value" : 199.53076547689204
data:   } ],
data:   "logicalTimestampMs" : 1461353198000
data: }

Binary data message format for WebSocket

When you send computation requests to SignalFlow using a WebSocket connection, you receive data messages in an efficient binary data encoding. Because data messages constitute most of the messages sent by SignalFlow computations, binary encoding helps reduce the data bandwidth, in some cases by more than 60%.

The following text diagram describes the format of the payload:

0                               1
0 1 2 3 4 5 6 7 8 9 A B C D E F 0 1 2 3 4 5 6 7 8 9 A B C D E F
+-------------------------------------------------------------+
|           Data batch (logical millisecond timestamp)        |
+- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+
|                      Timestamp (continued)                  |
+-------------------------------------------------------------+
|                     Payload element count                   |
+-------------------------------------------------------------+
| Payload data, series of 17-byte 3-uples: 1-byte value type, |
+- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+
| 8-byte timeseries ID (long) and 8-byte datapoint value,     |
+- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+
| either long (0x01), double (0x02), or int (0x03), depending |
+- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+
| on value type.                                              |
+-------------------------------------------------------------+

Table 3. Data message payload fields
Field	Length	Description
Data batch time stamp	64 bits (`int64`)	The logical timestamp in Unix time (milliseconds)
Payload element count	32 bits (`int`)	Number of datapoints in the payload
Payload data	136 bits (17 bytes) per datapoint	Bits 0 - 7 (byte 1): value type with one of the following values: 0x01 — long 0x02 — double 0x03 — int Bits 8 - 71 (bytes 2 - 9) : timeseries ID (long) Bits 72 - 135 (bytes 10 - 16): datapoint value, with size determined by value type

Both int and long are 64-bit integers, but the value type int indicates that the value fits in a 32-bit integer, which means that the 32 high-order bits are zero.

You can convert the time series ID to its string representation by Base64-decoding the bytes and removing the = padding characters.

The data is sent "big-endian".

Event messages

SignalFlow sends an event message when a detect() function detects or clears an incident based on the rule for the detect. The metadata for the event time series, identified by the tsId specified in the event message, always arrives before the event message. The event message itself contains a Unix time timestamp in milliseconds and a properties object.

The inputValues child property of properties is an object that contains the names and values of each detect condition variables that, when evaluated, caused the alert to fire or to clear. inputValues is a JSON-encoded map of the detect condition’s variables and their values which, when evaluated, caused the alert to fire (was: ok, is: anomalous) or clear (was: anomalous, is: ok).

The following table describes each property of the event message:

Table 4. Event message properties
Property	Description
`timestampMs`	Timestamp (in Unix time format) of the incident that resulted in the event message
`tsId`	Identifier of the input timeseries for the computation that caused the event message
`incidentId`	Identifier for the incident that resulted in the event message
`inputValues`	Object that containing the variables and values that caused the alert to fire or to clear
`was`	State of the alert before the computation
`is`	State of the alert after the computation

SSE example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


event: event
data: {
data:   "properties" : {
data:     "incidentId" : "Cj5VGRiAEAM",
data:     "inputValues" : "{'a':4}",
data:     "is" : "anomalous",
data:     "was" : "ok"
data:   },
data:   "timestampMs" : 1464816908000,
data:   "tsId" : "AAAAAOsfgK8"
data: }

WebSocket example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
  "type" : "event",
  "channel" : "channel-1",
  "properties" : {
    "incidentId" : "Cj5VGRiAEAM",
    "inputValues" : "{'a':4}",
    "is" : "anomalous",
    "was" : "ok"
  },
  "timestampMs" : 1464816908000,
  "tsId" : "AAAAAOsfgK8"
}

Third-party license information

SignalFlow Stream Messages Reference

Realms in endpoints

SignalFlow client libraries

Receiving messages via Server-Sent Events

Receiving messages via WebSocket

Message compression

Response message examples

SSE response messages

WebSocket response messages

Binary encoding of WebSocket messages

Message types

Control messages

SSE example

WebSocket example

STREAM_START control message

SSE example

WebSocket example

JOB_START control message

SSE example

WebSocket example

JOB_PROGRESS control message

SSE example

WebSocket example

CHANNEL_ABORT control message

SSE example

WebSocket example

END_OF_CHANNEL control message

SSE example

WebSocket example

Information messages

Metadata messages

Metadata message in SSE format

Metadata message in WebSocket format

Expired tsId messages

Expired tsId message in SSE format

Expired tsId message in WebSocket format

Data messages

SSE example

Binary data message format for WebSocket

Event messages

SSE example

WebSocket example

Information

Last Changed: 2019-03-27

Page Version:2.1.0

Writer:Joe Malin

BaseURL: https://developers.signalfx.com

Hugo Version: 0.54.0

`STREAM_START` control message

`JOB_START` control message

`JOB_PROGRESS` control message

`CHANNEL_ABORT` control message

`END_OF_CHANNEL` control message

Expired `tsId` messages

Expired `tsId` message in SSE format

Expired `tsId` message in WebSocket format