RudderStack Architecture
Detailed technical description of the RudderStack architecture and its major components - the control plane and data plane.
Last updated
Was this helpful?
Detailed technical description of the RudderStack architecture and its major components - the control plane and data plane.
Last updated
Was this helpful?
RudderStack is a stand-alone system dependent only on a database (PostgreSQL). Its backend is written in Go, with a rich UI written in React.js.
RudderStack's architecture consists of 2 major components: the control plane and data plane, as seen in the following diagram:
The control plane offers a UI to configure your event data sources and destinations. It comprises of:
RudderStack web app: This is the that lets you set up your data pipelines in RudderStack.
Configuration backend: The web app leverages this module to store all the relevant information around your configured sources, destinations, and the connections between them.
The data plane(backend) is RudderStack's core engine responsible for:
Receiving and processing the event data
Transforming the events in the required destination format
Relaying the events to the destination
The RudderStack data plane consists of three major components:
RudderStack server ()
Transformations module
Standalone streaming database(PostgreSQL) for the event data
RudderStack's data plane is responsible for receiving, transforming, and routing the event data to the destination. To do so, it receives the event data from sources like websites, mobile apps, server-side applications, cloud apps, and data warehouses.
A simplified version of the RudderStack data plane architecture is as shown:
The following sections give a detailed overview of each of the components of the RudderStack data plane.
The gateway module is primarily responsible for receiving the event data from a source (web, mobile, or server-side) and forwarding it for processing and transformation.
It accepts the event requests and sends an acknowledgment back to the source depending on the acceptance (HTTP 200 response) or rejection of the event data.
The gateway rejects the event data in case of the following scenarios:
Invalid JSON
Invalid write key
Improper request size
The gateway also temporarily stores all the received event data into an internal PostgreSQL database before acknowledging successful receipt. Once the event is successfully sent to the destination, it is then deleted from the database.
The processor fetches the data from the gateway and forwards it to the transformation module. Once the event data is transformed, the processor forwards it to the router module that sends it to the required destination.
RudderStack's Transformations module transforms the received event data into a suitable destination-specific format. All the transformation codes are written in JavaScript.
RudderStack also supports user transformations that let you code custom JavaScript functions to implement specific use-cases on your event data. These include - but are not limited to - event enrichment, filtering, removing sensitive PII information, etc.
Once the events are transformed, the transformation module sends them back to the processor. The processor then forwards this data to the router, which in turn relays it to the desired destination.
The router module sends the processed and transformed event data to your desired destinations, like marketing and analytics platforms, CRMs, data warehouses, etc.
The data plane uses PostgreSQL as a streaming database for the event data. RudderStack temporarily stores the events in the database so that it can retry sending them in case of delivery failures. The events are deleted once they are successfully delivered.
RudderStack does not persist any events or store any user information in the database.
The following steps detail the RudderStack data plane workflow:
The gateway module receives the event data from the source.
The gateway then:
Stores the event data to the router database (PostgreSQL).
Sends an HTTP 200 status acknowledging receipt of the data.
The processor module picks the data from the gateway and forwards it to the transformation module.
The transformation module transforms the event and sends it back to the processor.
The processor forwards the transformed event to the router. This event is then deleted from the gateway store.
The router then: i. Sends the transformed event data to the specified destinations. ii. Stores the event information in a separate table in the database.
Once the event data reaches the destination, the router deletes it from the database.
Although the default configuration works just fine for most use cases, RudderStack also gives you the flexibility to customize the data plane with a variety of configuration options. Some of these include backing up events to S3, rejecting malicious requests by defining the maximum size of the event, and more.
Refer to the section below for more information on each of these components.
Refer to the guide for more information on the user transformations.
You can do so by tweaking the file to suit your application's needs.
Refer to the guide for more information on these configuration options.
For more information on any of the sections covered in this guide, you can or start a conversation in our community.
