---
description: Step-by-step guide on setting up S3 Data Lake as a destination in RudderStack.
---

# Amazon S3 Data Lake

Amazon S3 is a popular object storage service used to store both structured and unstructured data. You can leverage S3 to securely and cost-effectively build a data lake of any size or scale. With an S3-powered data lake, you can easily use the native AWS services for data processing, analytics, machine learning, and more.

RudderStack lets you configure S3 data lake as a destination to which you can send your event data seamlessly.

Refer to the [Warehouse Schemas](https://rudderstack.com/docs/data-warehouse-integrations/warehouse-schemas/) guide for more information on how the events are mapped to the tables in S3 data lake.

Find the open source transformer code for this destination in the [GitHub repository](https://github.com/rudderlabs/rudder-transformer/tree/master/v0/destinations/s3_datalake).

## Configuring S3 Data Lake destination in RudderStack

To send event data to SQL Server, you first need to add it as a destination in RudderStack and connect it to your data source. Once the destination is enabled, events will automatically start flowing to SQL Server via RudderStack.

To configure SQL Server as a destination in RudderStack, follow these steps:

1. In your [RudderStack dashboard](https://app.rudderstack.com), set up the data source. Then, select **S3 Data Lake** from the list of destinations.
2. Assign a name to your destination and then click on **Next**.

### Connection settings

![S3 data lake destination settings in RudderStack](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-fe917d9ba6e898469b178ef6c71444c4b3dbd47c%2FconnectionScreen.png?alt=media)

Enter the following credentials in the **Connection Credentials** page:

* **S3 Storage Bucket Name**: The name of the S3 bucket that will be used to store the data before loading it into the S3 data lake.
* **Register schema on AWS Glue**: If you enable this option, RudderStack registers the schema of your incoming data on AWS Glue's data catalog.

For more information on registering your schema in AWS Glue, refer to the [AWS Glue documentation](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html).

* **AWS Glue Region**: Your AWS Glue region. For example, for `N.Virginia`, it would be `us-east-1`.
* **S3 Prefix**: If specified, RudderStack creates a folder in the bucket with this prefix and push all the data within that folder.
* **Namespace**: If specified, all the data for the destination will be pushed to `s3://<bucketName>/<prefix>/rudder-datalake/<namespace>`. If you don't specify a namespace in the settings, it is set to the source name, by default.

If AWS Glue is enabled, all the table definitions are created in a database with the name set to this namespace.

* **AWS Access Key ID**: Your AWS access key ID.
* **AWS Secret Access Key**: Your AWS secret access key.

Make sure the above credentials (**Access Key ID** and **Secret Access Key**) have the permissions to read and write into the configured bucket.

If AWS Glue is enabled, make sure that the following permissions are granted to it:

* glue:CreateTable
* glue:UpdateTable
* glue:CreateDatabase
* glue:GetTables

### Finding your data in S3 data lake

RudderStack converts your events into Parquet files and dumps them to the configured S3 bucket. Before dumping the events, RudderStack groups them by the event name based on the time (UTC) they were received.

The folder structure is shown below:

```
s3://<bucketName>/<prefix>/rudder-datalake/<namespace>/<tableName>/YYYY/MM/DD/HH
```

As mentioned in the [Connection Settings](#connection-settings) section:

* `prefix`: This is the S3 prefix in the destination settings. If not specified, RudderStack will omit this value.
* `namespace`: The namespace specified in the destination settings. If not specified, RudderStack sets this field to the source name by default.
* `tableName`: RudderStack sets this to the event name.

**`YYYY`**, **`MM`**, **`DD`**, and **`HH`** are replaced by actual time values. A combination of these values represents the UTC time.

Suppose that RudderStack tracks the following two events:

| Event name          | Timestamp                       |
| ------------------- | ------------------------------- |
| `Product Purchased` | `"2019-10-12T08:40:50.52Z" UTC` |
| `Cart Viewed`       | `"2019-11-12T09:34:50.52Z" UTC` |

RudderStack will convert these events into Parquet files and dump them into the following folders:

| Event Name          | Folder Location                                                                          |
| ------------------- | ---------------------------------------------------------------------------------------- |
| `Product Purchased` | `s3://<bucketName>/<prefix>/rudder-datalake/<namespace>/product_purchased/2019/10/12/08` |
| `Cart Viewed`       | `s3://<bucketName>/<prefix>/rudder-datalake/<namespace>/cart_viewed/2019/11/12/09`       |

If AWS Glue is enabled, all the table definitions are created in a database with the name set to the namespace specified in the destination settings.

## Creating a crawler

Refer to this section **only** if you haven't enabled the **Register Schema on AWS Glue** setting while configuring the S3 data lake destination in RudderStack.

In the absence of AWS Glue, you can create a crawler to go through your data and create table definitions out of it.

Follow these steps to create a crawler:

1. Go to your AWS Glue console and select **Crawler** from the left pane.
2. Select **Add Crawler**.
3. Specify a name for your crawler and click **Next**, as shown:

![Add Crawler](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-7d1a7e3630d5f927727f9221a6b81d15e19266e8%2FcrawlerName.png?alt=media)

* Next, under the **Crawler source type** section, choose **Data stores**.

![Crawler source type](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-ae9fd1836934c23673e887f33e75e2bf01db7f4a%2FcrawlerSourceType.png?alt=media)

1. Configure the **Repeat crawls of S3 data stores** based on your requirement.
2. Then, under the **Data store** section, select **S3** from the dropdown for the **Choose a data store** setting, as shown:

![Choose a data store](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-c4f24d7976ac60d5c05695f5f4466427339232a3%2FaddDataStore.png?alt=media)

1. For the **Crawl data in** setting, choose the option **Specified path in my account**.
2. In the **Include path** setting, enter the S3 URI of your configured bucket followed by the suffix `/<prefix>/rudder-datalake/<namespace>/`.

If your S3 bucket name is `testBucket` and the configured prefix and namespace are `testPrefix` and `testNameSpace` respectively, then your path should be: `s3://testBucket/testPrefix/rudder-datalake/testNameSpace/`

If you have not configured any prefix while setting up the S3 data lake destination in RudderStack, omit the prefix. The path would then be: `s3://testBucket/rudder-datalake/testNameSpace/`.

1. Then, under the **Add another data store** setting, select **No**, as shown:

![Add another data store](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-fe6c01a3cb8b3af5dd0fd40e060642202c83343d%2FanotherDataStore.png?alt=media)

1. In the **IAM Role** section, configure a suitable IAM role.

![IAM Role](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-5eab40ceb72d8b30a0272888296c0741aa02b100%2Fiam.png?alt=media)

1. In the **Schedule** section, select the frequency of your crawler from the dropdown options, as shown:

![Scheduler](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-b3e2a3d33350f20d703652fccce577ba7e762516%2Fscheduler.png?alt=media)

1. In the **Output** section, configure the database that stores all the tables. Under the Grouping behavior for S3 data section, enable the **Create a single schema for each S3 path** option, as shown:

![Output](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-fb0706b76f60c11a758b09ee2e77d28c97605590%2Foutput.png?alt=media)

1. Specify the **Table level** as **5** or **4** (refer to the tips below). This value indicates the absolute level of the table location in the bucket.

The level for the top-level folder is 1. For example, for the path `mydataset/a/b`, if the level is set to 3, the table will be created at the location `mydataset/a/b`. Similarly, if the level is set to 2, the table will be created at the location `mydataset/a`.

Since all tables are created in the path `s3://testBucket/<prefix>/rudder-datalake/<namespace>/`, make sure the table level is set to:

* **5**: If a prefix is configured.
* **4**: If a prefix is **not** configured.

1. Review your crawler configuration, as shown:

![Review](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-7135d0c0bddba66ef78bff07226847f310eb560e%2Freview.png?alt=media)

1. Click on **Finish** to confirm the configuration.
2. Finally, click on your crawler and run it. Wait for the process to finish - you should see some tables created in your configured database.

## Querying data using AWS Athena

You can query your S3 data using a tool like [AWS Athena](https://aws.amazon.com/athena/) which lets you run SQL queries on S3.

Before querying your data on S3, make sure that you have sent some data to S3 and that the sync has been completed.

Follow these steps to start querying your data on s3 -

1. Open your AWS Athena console. Then, go to the same AWS region which was used while to configure AWS Glue.
2. In the left pane, select `AwsDataCatalog` as your data source, as shown:

![AwsDataCatalog](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-142298a34f3f829969c0fef21b70c6dbd6040d8b%2Fdatasource.png?alt=media)

1. Select your configured namespace (or the database you specified while configuring the crawler) from the database dropdown menu.

![Database](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-e1895c5452ed4e3a541df5d41b97ce1717925251%2Fdatabase.png?alt=media)

By default, the namespace is set to your source name if you did not specify it in the destination settings.

1. You should see some tables already created under the **Tables** section in the left pane.
2. You can preview the data by clicking on the three dots next to the table and selecting the **Preview Data** option. Alternatively, you can run your own SQL queries in the workspace on the right, as shown:

![Preview Data](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-c52934330332771906dab69828327e2bcbdab790%2FpreviewTable.png?alt=media)

## IPs to be whitelisted

To enable network access to RudderStack, you will need to whitelist the following RudderStack IPs:

* 3.216.35.97
* 34.198.90.241
* 54.147.40.62
* 23.20.96.9
* 18.214.35.254
* 35.83.226.133
* 52.41.61.208
* 44.227.140.138
* 54.245.141.180
* 3.66.99.198
* 3.64.201.167

If you have your deployment in the EU region, you can whitelist only the following two IPs:

* 3.66.99.198
* 3.64.201.167

All the outbound traffic is routed through these RudderStack IPs.

## Contact us

For queries on any of the sections covered in this guide, you can [contact us](mailto:%20docs@rudderstack.com) or start a conversation in our [Slack](https://rudderstack.com/join-rudderstack-slack-community) community.
