---
description: Detailed technical description of the RudderStack Airflow Provider.
---

# Airflow Provider

RudderStack's Airflow Provider lets you programmatically schedule and trigger your [Reverse ETL](https://rudderstack.com/docs/reverse-etl/) syncs from outside RudderStack and integrate them with your existing Airflow workflows.

For more information on the codebase and sample implementation, refer to our [GitHub Repository](https://github.com/rudderlabs/rudder-airflow-provider).

## Using the Airflow Provider

To use the Airflow Provider, you must have a working Apache Airflow installation. For more information, refer to the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html).

Follow the steps in the below sections to use the RudderStack Airflow Provider:

### Running Airflow

Initialize all the dependencies by running Apache Airflow via the following command:

```bash
airflow standalone
```

The Airflow standalone server is not meant for use in production. It is highly recommended using alternate methods to install and run Airflow in a production environment.

### Installing Airflow Provider

Install the RudderStack Airflow Provider by running the following command:

```bash
pip install rudderstack-airflow-provider
```

### Creating an Airflow connection

To create a new Airflow connection, follow these steps:

1. In your Airflow dashboard, go to **Admin** > **Connections**, as shown:

![Airflow dashboard Connections option](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-afe287940a3ed5859d5ad498fe6c21d592b42b31%2Fairflow-provider-1.png?alt=media)

1. Add a new connection by configuring the following details:

![Airflow dashboard edit connection](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-a969d9fbfbe7de38820bda37f133f98c59450421%2Fairflow-provider-2.png?alt=media)

* **Connection ID**: Specify a unique connection name. `RudderstackOperator` will pick the connection with the name `rudderstack_default` by default. If you have created a connection with a different name, make sure that name is passed as a parameter to `RudderstackOperator`.
* **Connection Type**: For this field, select **HTTP** from the list.
* **Host**: Set the value for this field to `https://api.rudderstack.com`.
* **Password**: Enter your Personal Access Token here.

For more information on generating an access token, refer to the [Personal Access Tokens](https://rudderstack.com/docs/rudderstack-api/personal-access-tokens/) guide.

### Defining a DAG

Next, define a DAG with the tasks as per your requirement.

The following code snippet highlights an Airflow DAG with one task named `rs_trigger_sync` for the source ID `20dQV6yuUDUw31peWA8f7xxgHdN`:

For more information on obtaining the source ID, refer to the [FAQ](#faq) section below.

```python
from datetime import datetime, timedelta

from airflow import DAG
from rudder_airflow_provider.operators.rudderstack import RudderstackOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['airflow@example.com'],
    'email_on_failure': False
}

with DAG(
    'rudderstack-sample',
    default_args=default_args,
    description='A simple tutorial DAG',
    schedule_interval=timedelta(days=1),
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=['rs']
) as dag:
    rs_operator = RudderstackOperator(
        source_id='20dQV6yuUDUw31peWA8f7xxgHdN',
        connection_id='rudderstack_sync_conn',
        task_id='rs_trigger_sync'
    )
```

For more information on where to find the source ID, refer to [Sources](https://www.rudderstack.com/docs/rudderstack-cloud/sources/) guide.

### Running the DAG

Once you have defined a DAG and configured an Airflow connection, run the following commands to allow Airflow to pick up and run the DAG:

```bash
export AIRFLOW_HOME=</path/to/airflow_home>
mkdir $AIRFLOW_HOME/dags
cp rudderstack_dag.py $AIRFLOW_HOME/dags
```

**Make sure the Airflow scheduler is running in the background**. Also, you must enable the DAG in the Airflow dashboard, as shown:

![enabling Airflow DAG in dashboard](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-e39af3bec375711c65ad07f35efc73b23c99c93c%2Fairflow-provider-3.png?alt=media)

You can trigger a DAG by clicking on the play button on the right as seen above, and selecting **Trigger DAG**. Note that **stopping the DAG will not cancel the ongoing sync**.

## FAQ

### Where can I find the source ID for my Reverse ETL source?

The source ID is a unique identifier for any source set up in RudderStack.

To obtain the source ID, go to the configured Reverse ETL source in your RudderStack dashboard and navigate to the **Settings** tab.

![Source ID for Reverse ETL sources](https://876606571-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lq5Ea6fHVg3dSxMCgyQ%2Fuploads%2Fgit-blob-9c745c7ef310a802092e99d075f170afb4d50e04%2Fsource-id.png?alt=media)

## Contact us

For queries on any of the sections covered in this guide, you can [**contact us**](mailto:%20docs@rudderstack.com) or start a conversation in our [**Slack**](https://rudderstack.com/join-rudderstack-slack-community) community.
