Amazon Kinesis

Step-by-step guide to set up Amazon Kinesis as a destination in RudderStack.

enables you to ingest, buffer and process streaming data in real-time. It can handle any amount of streaming data and process data from hundreds of thousands of sources with very low latencies along with the flexibility to choose the tools that best suit the requirements of your application.

RudderStack allows you to configure Amazon Kinesis as a destination to which you can send your event data seamlessly.

Find the open-source transformer code for this destination in our .

Getting Started

In order to enable dumping data to Amazon Kinesis, you will first need to add it as a destination to the source from which you are sending event data. Once the destination is enabled, events from RudderStack will start to flow to Amazon Kinesis. Before configuring your source and destination on the RudderStack app, please check whether the platform you are working on is supported by Amazon Kinesis. Refer to the table below:

Connection Mode

Web

Mobile

Server

Device mode

Cloud mode

Supported

To know more about the difference between Cloud mode and Device mode in RudderStack, read the guide.

Once you have confirmed that the platform supports sending events to Kinesis, perform the steps below:

To do so, please follow these steps:

Select Amazon Kinesis as a destination in the RudderStack dashboard.
Next, in the Connection Settings, fill all the fields with the relevant information and click Next.

Add Kinesis as destination

Region: AWS Region in which you have created the Kinesis stream
Stream name: The name of your Kinesis stream
Credentials: AWS Access Key ID and Secret Access Key to authorize RudderStack to write to your stream. For that, you need to create an IAM Policy that will provide permission to write to your data stream. Please refer to for the applicable policies. A sample policy is as below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kinesis:PutRecord"],
      "Resource": ["arn:aws:kinesis:{region}:{account-id}:stream/{stream-name}"]
    }
  ]
}

If the AWS credentials are already configured on your machine where the RudderStack server is set up, you will not need the security credentials.

There is an option of messageId being used as a Partition Key. By default, we will use userId ( if userId is not present in the payload, then anonymousId ) as a partition key, but if you enable this, we will use messageId as the partition key. This enables your data to be more evenly distributed across the shards in the Kinesis stream.

FAQs

Why are the events not published to my Kinesis stream

Verify that you are sending the events in the Live Events tab of your source.

Check if there are any delivery failures in the Live Events tab of your destination.

Contact us

PreviousAmazon Kinesis Firehose NextAzure Event Hubs

Last updated 3 years ago

Was this helpful?

Getting Started

Connection Mode

Web

Mobile

Server

Device mode

Cloud mode

Supported

To know more about the difference between Cloud mode and Device mode in RudderStack, read the guide.

Once you have confirmed that the platform supports sending events to Kinesis, perform the steps below:

To do so, please follow these steps:

Select Amazon Kinesis as a destination in the RudderStack dashboard.

Next, in the Connection Settings, fill all the fields with the relevant information and click Next.

Add Kinesis as destination

Region: AWS Region in which you have created the Kinesis stream

Stream name: The name of your Kinesis stream

Credentials: AWS Access Key ID and Secret Access Key to authorize RudderStack to write to your stream. For that, you need to create an IAM Policy that will provide permission to write to your data stream. Please refer to for the applicable policies. A sample policy is as below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kinesis:PutRecord"],
      "Resource": ["arn:aws:kinesis:{region}:{account-id}:stream/{stream-name}"]
    }
  ]
}

If the AWS credentials are already configured on your machine where the RudderStack server is set up, you will not need the security credentials.

There is an option of messageId being used as a Partition Key. By default, we will use userId ( if userId is not present in the payload, then anonymousId ) as a partition key, but if you enable this, we will use messageId as the partition key. This enables your data to be more evenly distributed across the shards in the Kinesis stream.