SignalFx Developers Guide

Integrating with AWS

Amazon Web Services (AWS) integration connects SignalFx data collection with Amazon Web Services Cloud Watch monitoring.

Only SignalFx administrators are allowed to create or update AWS integrations, and they must use an access token associated with a user that has administrative access. Other users may view but not edit existing integrations.

AWS integrations with SignalFx support two authentication options:

  1. External ID and Amazon Resource Name (ARN)

  2. Key and token

To learn more about these options, see the topic Monitor Amazon Web Services in the product documentation

To learn how to set up integrations using these methods, see the examples in the topic Working with Integrations.

AWS integration namespaces and filters

An AWS integration can sync predefined AWS namespaces corresponding to various AWS services such as EC2 and S3. The integration can also work with custom namespaces set up to sync additional metrics you define. By default, SignalFx syncs all AWS data from all namespaces. If you only need a specific part of the data, you can choose to import specific namespaces, or you can set up your integration to include or exclude specific AWS tags or metrics.

To learn more about custom namespaces and metrics, refer to Amazon’s documentation for AWS.

Choosing namespaces

You configure support for namespaces using properties of the AWS Cloud Watch Integration Model defined by SignalFx:

  • To configure predefined AWS namespace support, use the namespaceSyncRules property.

  • To configure custom namespace support, use the customNamespaceSyncRules property.

These properties are arrays of objects in which each object specifies a single namespace and its properties:

namespace

Name of the namespace you’re configuring. This property is required for each object. If you wish to sync the entire namespace, you don’t need to specify any other properties.

defaultAction

Controls whether the default for this namespace should be to sync all data or sync no data. If you don’t specify a filter, the API ignores this property.

filter

Specifies a filter that overlays the default action. Individual filters can either specify data to include or data to exclude in the sync. Useful filters usually add data to a namespace that otherwise doesn’t sync, or they exclude data from a namespace that otherwise includes all data. This property is an object that contains two properties:

action

Indicates whether the filter includes or excludes data

source

Defines the filter

Specifying filter criteria

To specify a filter, you add SignalFlow filter expressions to the source property. These expressions use the syntax of the SignalFlow filter() function; they define filter criteria based on AWS tags and metrics. You can only specify one filter for a namespace, but you can make it as complex as you want.

The filter expression specifies how to compare the value of a property listed in the first argument to all the subsequent arguments. The comparisons are joined by an implicit OR; if the value matches any one of the arguments, the expression is true.

AWS namespaces with synced tags

SignalFx doesn’t accept all AWS tags as filter criteria. You can only filter on a tag when it comes from an AWS namespace for which SignalFx syncs tags. The section Importing account metadata and custom tags in the product documentation has a subsection for each of the AWS namespaces (services) that sync tags with SignalFx.

Specifying AWS regions

The Integrations API lets you specify the AWS regions that you’re monitoring. Use the "regions" array property to specify one or more regions. For example, the following curl command creates an AWS integration for the "ap-northeast-1" and "ap-northeast-2" AWS regions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
curl \
--request POST \
--header "X-SF-TOKEN: <YOUR_ORG_TOKEN>" \
--header "Content-Type: application/json; charset=utf8" \
--data \
    '{
        "type": "AWSCloudWatch",
        "authMethod": "ExternalId",
        "enabled": true,
        "externalId": "<EXTERNAL_ID>",
        "id": "<INTEGRATION_ID>",
        "name": "AWS-IDauth",
        "pollRate": 60000,
        "regions": ["ap-northeast-1", "ap-northeast-2"],
        "roleArn": "<YOUR_ARN>",
        "services": ["AWS/EC2"],
        ...
    }' \
https://api.<REALM>.signalfx.com/v2/integration

If you omit the "regions" property or specify regions: [], the API adds all of the regular AWS regions to your integration. These regions are listed in the next section.

Supported regions

The Integrations API supports the following sets of regions:

Regular AWS regions

  • ap-northeast-1

  • ap-northeast-2

  • ap-south-1

  • ap-southeast-1

  • ap-southeast-2

  • ca-central-1

  • eu-central-1

  • eu-north-1

  • eu-west-1

  • eu-west-2

  • eu-west-3

  • sa-east-1

  • us-east-1

  • us-east-2

  • us-west-1

  • us-west-2

GovCloud AWS regions

  • us-gov-east-1

  • us-gov-west-1

China AWS regions

  • cn-north-1

  • cn-northwest-1

Refer to Amazon’s AWS documentation for more information.

Referencing AWS tags and metrics

Refer to an AWS tag defined by the key-value pair "tagName": "tagValue" using the SignalFlow syntax:

filter('aws_tag_tagName', 'tagValue')

Notice that you need to prepend aws_tag_ to tagName; this differentiates from SignalFx tag names.

To test for the existence of tagName, set tagValue to * to indicate that all values match. To learn more about defining AWS tags, refer to Amazon’s AWS documentation.

The following snippet shows you a simple filter expression that matches the AWS tag environment to the value Prod.

filter('aws_tag_environment', 'Prod')

Individual simple filter expressions may reference AWS metrics using this syntax:

filter('sf_metric', 'metricName')

where sf_metric is constant and metricName is the name of the metric as defined in AWS. For information on how to determine which metrics are available within specific AWS namespaces, see the AWS documentation on Viewing Metrics with CloudWatch and for information on defining metrics for your custom namespaces see the AWS documentation on Publishing Metrics.

The following snippet demonstrates how to specify a simple filter expression that matches the NetworkIn metric defined in many AWS namespaces:

filter('sf_metric', 'NetworkIn')

For a more complex example of filter expressions, see the examples in the topic Working with Integrations.

Examples

These examples demonstrate how to create an AWS integration using the two available authentication methods:

Create an AWS integration using an external ID and ARN

Specify external ID/ARN authentication

Create an AWS integration object using the operation POST https://api.{REALM}.signalfx.com/v2/integration and specify "authMethod": "ExternalId" in the request body. The following curl command demonstrates how to do this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
curl \
--request POST \
--header "X-SF-TOKEN: <YOUR_ORG_TOKEN>" \
--header "Content-Type: application/json; charset=utf8" \
--data \
    '{
        "type": "AWSCloudWatch",
        "name": "AWS-IDauth",
        "pollRate": 60000,
        "authMethod": "ExternalId",
        ...
    }' \
https://api.<REALM>.signalfx.com/v2/integration

The response object for a successful request is similar to the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
    "type": "AWSCloudWatch",
    "authMethod": "ExternalId",
    "enabled": true,
    "externalId": "<EXTERNAL_ID>",
    "id": "<INTEGRATION_ID>",
    "name": "AWS-IDauth",
    "pollRate": 60000,
    "regions": [],
    "roleArn": null,
    "services": [],
   ...
}

Create an AWS Cloud Watch role

After you have the external ID, use it and your SignalFx account ID to create a role in AWS Cloud Watch. Remember that the account ID must belong to a user who has administrative privileges.

To learn how to create the role, see the topic Connect to AWS CloudWatch in the product documentation.

Update the integration object with the Cloud Watch ARN

Now that you have an external ID and ARN for the AWS integration, update the integration with the ARN. The following curl command shows you how to do this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
curl \
--request PUT \
--header "X-SF-TOKEN: <YOUR_ORG_TOKEN>" \
--header "Content-Type: application/json; charset=utf8" \
--data \
    '{
        "type": "AWSCloudWatch",
        "authMethod": "ExternalId",
        "enabled": true,
        "externalId": "<EXTERNAL_ID>",
        "id": "<INTEGRATION_ID>",
        "name": "AWS-IDauth",
        "pollRate": 60000,
        "regions": [],
        "roleArn": "<YOUR_ARN>",
        "services": [],
        ...
    }' \
https://api.<REALM>.signalfx.com/v2/integration/<INTEGRATION_ID>

The response object for this request is the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
    "type": "AWSCloudWatch",
    "authMethod": "ExternalId",
    "enabled": true,
    "externalId": "<INTERNAL_ID>",
    "id": "<INTEGRATION_ID>",
    "name": "AWS-IDauth",
    "pollRate": 60000,
    "regions": [],
    "services": [],
   ...
}

Create an AWS integration using a key and a token

To create an AWS integration using a key and token, create an AWS integration object using the operation POST https://api.{REALM}.signalfx.com/v2/integration. Specify the following in the request body:

`"authMethod": "SecurityToken"

Use the security token authentication method

`"token": "<YOUR_TOKEN>"

A security token you create

`"key": "<YOUR_KEY>"

A key you create

The following curl command demonstrates how to do this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
$ curl \
--request POST \
--header "X-SF-TOKEN: <YOUR_ORG_TOKEN>" \
--header "Content-Type: application/json; charset=utf8" \
--data \
    '{
        "type": "AWSCloudWatch",
        "name": "AWS-tokenauth",
        "authMethod": "SecurityToken",
        "token": "<YOUR_TOKEN>",
        "key": "<YOUR_KEY>",
        ...
    }' \
https://api.<REALM>.signalfx.com/v2/integration

The response object for a successful request is similar to the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
    "type": "AWSCloudWatch",
    "authMethod": "SecurityToken",
    "enabled": true,
    "id": "<INTEGRATION_ID>",
    "name": "AWS-IDauth",
    "pollRate": 60000,
    "regions": [],
    "services": [],
   ...
}

Full example

Jane is a SignalFx administrator at Awesomesauce Company. She wants to monitor their EC2, S3, and DynamoDB activity in AWS as well as their custom namespace Users containing information about users of various services. Each user has a primary account owner responsible for managing their engagement with the company. Awesomesauce uses an AWS tag to indicate the environment each resource supports (QA, Staging, or Prod) and two tags in the Users namespace to indicate the owner for that user and that owner’s team.

Jane decides to monitor everything in DynamoDB but only cares about the production and staging environments for EC2 and the production environment for S3. Because the data stored in S3 is fairly stable and updated via controlled pathways, she only wants to monitor error cases using the 4xxErrors and 5xxErrors metrics. In addition, Fred volunteers to use SignalFx to monitor activity and data for his users as a pilot program.

Jane is using a key and token to authenticate with AWS. She creates the following integration to sync the data described above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
$ curl \
--request POST \
--header "X-SF-TOKEN: <YOUR_ORG_TOKEN>" \
--header "Content-Type: application/json; charset=utf8" \
--data \
    '{
        "type": "AWSCloudWatch",
        "name": "EC2-S3-Dynamo-User AWS Integration",
        "authMethod": "SecurityToken",
        "token": "<YOUR_AWS_TOKEN>",
        "key":"<YOUR_AWS_KEY",
        "pollRate": 60000,
        "importCloudWatch": true,
        "customNamespaceSyncRules": [
            {
                "namespace":"User",
                "defaultAction":"Exclude",
                "filter": {
                    "action":"Include",
                    "source": "filter(\"aws_tag_owner\",\"Fred\")"
                }
            }
        ],
        "namespaceSyncRules": [
            {
                "namespace":"AWS/EC2",
                "defaultAction":"Include",
                "filter": {
                    "action":"Exclude",
                    "source": "filter(\"aws_tag_environment\",\"QA\")"
                }
            },
            {
                "namespace": "AWS/S3",
                "defaultAction": "Exclude",
                "filter": {
                    "action":"Include",
                    "source":"filter(\"aws_tag_environment\",\"Staging\",\"Prod\") and filter(\"sf_metric\",\"4xxErrors\",\"5xxErrors\")"
                }
            },

            {
                "namespace": "AWS/DynamoDB"
            }
        ]
    }'
    https://api.{REALM}.signalfx.com/v2/integration

which returns the following integration object in the response:

200 OK

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
{
  "type": "AWSCloudWatch",
  "name": "EC2-S3-Dynamo-User AWS Integration",
  "id": "<INTEGRATION_ID>",
  "authMethod": "SecurityToken",
  "created": <CREATED_TIMESTAMP>,
  "creator" : "<CREATOR_ID>"
  "lastUpdated": <UPDATED_TIMESTAMP>,
  "lastUpdatedBy" : "<UPDATER_ID>",
  "externalId": null,
  "roleArn": null,
  "customCloudWatchNamespaces": null,
  "enableAwsUsage": false,
  "enabled": true,
  "pollRate": 60000,
  "importCloudWatch":true,
  "customNamespaceSyncRules": [
    {
        "namespace":"User",
        "defaultAction":"Exclude",
        "filter":{
            "action":"Include",
            "source":"filter(\"aws_tag_owner\",\"Fred\")"
        }
    }
  ],
  "namespaceSyncRules": [
    {
        "namespace": "AWS/EC2",
        "defaultAction": "Include",
        "filter": {
            "action": "Exclude",
            "source": "filter(\"aws_tag_environment\",\"QA\")"
        }
    },
    {
        "namespace": "AWS/S3",
        "defaultAction": "Exclude",
        "filter": {
            "action": "Include",
            "source": "filter(\"aws_tag_environment\",\"Staging\",\"Prod\") and filter(\"sf_metric\",\"4xxErrors\",\"5xxErrors\")"
        }
    },
    {
        "namespace":"AWS/DynamoDB"
    }
  ]
}

To ensure security, SignalFx doesn’t return the token and key in the response.

Fred is very happy with the experiment and more owners are interested in using SignalFx. Everyone on his team except Prasad is willing to give it a try themselves. Jane updates the integration to sync Fred’s entire team, team X23f1, except for Prasad.

The following curl command shows you how Jane updates the integration. The path parameter <INTEGRATION_ID> in the endpoint comes from the previous response body.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
$ curl \
--request PUT \
--header "X-SF-TOKEN: <YOUR_ORG_TOKEN>" \
--header "Content-Type: application/json; charset=utf8" \
--data \
    '{
        "type": "AWSCloudWatch",
        "customNamespaceSyncRules": [
            {
                "namespace": "User",
                "defaultAction": "Exclude",
                "filter": {
                    "action": "Include",
                    "source":"filter(\"aws_tag_team\",\"X23f1\") and not filter(\"aws_tag_owner\",\"Prasad\")"
                }
            }
        ],
    ...
    }'
    https://api.{REALM}.signalfx.com/v2/integration/<INTEGRATION_ID>

Note that the new filter on the User namespace will not display in the web UI.

© Copyright 2019 SignalFx.

Third-party license information