Amazon Kinesis

Step-by-step guide to set up Amazon Kinesis as a destination in RudderStack.

Amazon Kinesis enables you to ingest, buffer and process streaming data in real-time. It can handle any amount of streaming data and process data from hundreds of thousands of sources with very low latencies along with the flexibility to choose the tools that best suit the requirements of your application.

RudderStack allows you to configure Amazon Kinesis as a destination to which you can send your event data seamlessly.

Find the open-source transformer code for this destination in our GitHub repo.

Getting Started

In order to enable dumping data to Amazon Kinesis, you will first need to add it as a destination to the source from which you are sending event data. Once the destination is enabled, events from RudderStack will start to flow to Amazon Kinesis. Before configuring your source and destination on the RudderStack app, please check whether the platform you are working on is supported by Amazon Kinesis. Refer to the table below:

Connection Mode	Web	Mobile	Server
Device mode	-	-	-
Cloud mode	Supported	Supported	Supported

To know more about the difference between Cloud mode and Device mode in RudderStack, read the RudderStack connection modes guide.

Once you have confirmed that the platform supports sending events to Kinesis, perform the steps below:

To do so, please follow these steps:

• Select Amazon Kinesis as a destination in the RudderStack dashboard.

• Next, in the Connection Settings, fill all the fields with the relevant information and click Next.

Add Kinesis as destination

• Region: AWS Region in which you have created the Kinesis stream

• Stream name: The name of your Kinesis stream

• Credentials: AWS Access Key ID and Secret Access Key to authorize RudderStack to write to your stream. For that, you need to create an IAM Policy that will provide permission to write to your data stream. Please refer to Controlling Access to Amazon Kinesis Data Streams Resources Using IAM for the applicable policies. A sample policy is as below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kinesis:PutRecord"],
      "Resource": ["arn:aws:kinesis:{region}:{account-id}:stream/{stream-name}"]
    }
  ]
}

If the AWS credentials are already configured on your machine where the RudderStack server is set up, you will not need the security credentials.

• There is an option of messageId being used as a Partition Key. By default, we will use userId ( if userId is not present in the payload, then anonymousId ) as a partition key, but if you enable this, we will use messageId as the partition key. This enables your data to be more evenly distributed across the shards in the Kinesis stream.

FAQs

Why are the events not published to my Kinesis stream

Verify that you are sending the events in the Live Events tab of your source.

Check if there are any delivery failures in the Live Events tab of your destination.

Contact us

In case of any queries, you can always contact us, or feel free to open an issue on our GitHub Issues page in case of any discrepancy. You can also start a conversation in our Slack community; we will be happy to talk to you!