Cloud

Redpanda Terraform Provider

Use the Redpanda Terraform provider to define, automate, and track changes to your Redpanda Cloud infrastructure as code.

With the Redpanda Terraform provider, you can manage:

After reading this page, you will be able to:

  • Create a BYOC cluster using Terraform

  • Enable Redpanda SQL on a BYOC cluster using the rpsql block

  • Create Redpanda Connect pipelines and allow pipeline egress to custom destinations

Why use Terraform with Redpanda?

  • Simplicity: Manage all your Redpanda Cloud resources in one place.

  • Automation: Create and modify resources without manual intervention.

  • Version control: Track and roll back changes using version control systems, such as GitHub.

  • Scalability: Scale your infrastructure as your needs grow with minimal effort.

Understand Terraform configurations

Terraform configurations are written in HCL (HashiCorp Configuration Language), which is declarative. Here are the main building blocks of a Terraform configuration:

Providers

Providers tell Terraform how to communicate with the services you want to manage. For example, the Redpanda provider connects to the Redpanda Cloud API using client credentials.

provider "redpanda" {
  client_id     = "<client_id>"
  client_secret = "<client_secret>"
}

Resources

Resources define the infrastructure components you want to create, such as networks, clusters, or topics. Each resource block specifies the type of resource and its configuration.

resource "redpanda_network" "example" { (1)
  name           = "example-network" (2)
  cloud_provider = "aws" (3)
  region         = "us-east-1" (4)
  cidr_block     = "10.0.0.0/20" (5)
}
1 The resource type (redpanda_network) and internal name (example). Terraform uses the internal name to reference the resource elsewhere in your configuration; for example, redpanda_network.example.id returns the network’s unique ID. The internal name does not appear in Redpanda Cloud.
2 A user-defined name for the resource as it appears in Redpanda Cloud. This is the user-facing name visible in the Redpanda UI and API.
3 The cloud provider where the network is deployed, such as AWS or GCP.
4 The region where the resource is provisioned.
5 The IP address range for the network.

Variables

Variables allow you to parameterize your configuration, making it reusable and customizable for different environments. Use variable blocks to define reusable values, like region, which can be overridden when running Terraform.

variable "region" {
  default = "us-east-1"
}

resource "redpanda_network" "example" {
  name           = "example-network"
  cloud_provider = "aws"
  region         = var.region
  cidr_block     = "10.0.0.0/20"
}

Outputs

Outputs let you extract information about your infrastructure, such as cluster URLs, to use in other configurations or scripts.

This example displays the cluster’s API URL after Terraform provisions the resources:

output "cluster_api_url" {
  value = data.redpanda_cluster.example.cluster_api_url
}

Limitations

The following functionality is supported in the Cloud API but not in the Redpanda Terraform provider:

  • Creating or deleting BYOVNet clusters on Azure

  • Kafka Connect

If allow_deletion is set to true, modifying throughput_tier causes Terraform to destroy and replace the existing cluster, causing data loss.

To prevent this, apply one or both of the following:

  • Set allow_deletion to false.

  • Add a lifecycle block instructing Terraform to ignore changes to throughput_tier:

    lifecycle {
  ignore_changes = [
    throughput_tier
  ]
}

    This setting protects your cluster in the event Redpanda Support upgrades your cluster’s throughput tier. Without it, the next terraform apply command triggers replacement.

Prerequisites

Redpanda Terraform Provider - Windows Support Notice

The Redpanda Terraform provider does not support Windows. To use it on Windows, install Windows Subsystem for Linux 2 (WSL2).

To use WSL2 with the Redpanda Terraform provider:

  1. If WSL2 is not already installed, install it by running the following command in PowerShell as Administrator:

    wsl --install

    Then restart your computer.

  2. Open your WSL2 Linux distribution (such as Ubuntu) from the Start menu or by running wsl in PowerShell.

  3. Navigate to your project directory within WSL2.

  4. Run all Terraform commands from within your WSL2 environment:

    # Initialize Terraform and download the Redpanda provider
terraform init

# Plan your Redpanda infrastructure changes
terraform plan

# Apply the configuration to create Redpanda resources
terraform apply

# View created resources
terraform show

  1. Install Terraform 1.0.0 or later using the official guide.

  2. Create a service account in Redpanda Cloud:

    1. Log in to Redpanda Cloud.

    2. Navigate to the Organization IAM page and select the Service account tab. Click Create service account and provide a name for the new service account.

    3. Save the client ID and client secret for authentication.

Set up the provider

  1. Add the Redpanda provider to your Terraform configuration:

    terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = ">= 2.1.1"
    }
  }
}

  2. Initialize Terraform to download the provider:

    terraform init

  3. Add the service account credentials you saved in Prerequisites as environment variables or in your Terraform configuration file:

    • Environment variables

    • Static credentials

    REDPANDA_CLIENT_ID=<client_id>
REDPANDA_CLIENT_SECRET=<client_secret>
    provider "redpanda" {
  client_id      = "<client_id>"
  client_secret  = "<client_secret>"
}

Manage sensitive attributes with write-only fields

You can use Terraform 1.11+ write-only attributes to keep sensitive values out of your Terraform state file. By default, Terraform persists sensitive attributes such as passwords to .tfstate when you run terraform apply. When you store state in a remote backend or in CI runner artifacts, this can leak credentials.

Write-only attributes require Terraform CLI 1.11 or later and Redpanda Terraform provider v1.6.0 or later.

How write-only attributes work

For each supported sensitive field, the provider exposes two new attributes alongside the existing one:

  • <field>_wo: A write-only attribute. Terraform sends the value to the provider during apply but never persists it to state.

  • <field>_wo_version: An integer version. Because Terraform cannot detect changes in a write-only value (there is nothing to compare against in state), you increment this number to signal that the value has changed and to trigger an update on the next apply.

redpanda_pipeline and redpanda_secret are exceptions to this naming convention. For redpanda_pipeline, the existing client_secret attribute is the write-only attribute (no separate client_secret_wo field), and is paired with secret_version instead of client_secret_wo_version. For redpanda_secret, the secret_data attribute is write-only by design (there is no plaintext variant), and is paired with secret_data_version.

The provider retains the original plaintext attributes for backward compatibility. You can migrate to the write-only variants on your own schedule. Avoid setting both the plaintext attribute and its write-only counterpart on the same resource. If both are set, the provider uses the write-only value.

Supported attributes

Resource Plaintext attribute (deprecated) Write-only attribute Version attribute

redpanda_user

password

password_wo

password_wo_version

redpanda_schema

password

password_wo

password_wo_version

redpanda_schema_registry_acl

password

password_wo

password_wo_version

redpanda_pipeline

client_secret

client_secret (write-only)

secret_version

redpanda_secret

None (write-only by design)

secret_data

secret_data_version

Set a write-only attribute

Inject the sensitive value through a sensitive Terraform variable, an environment variable, or your secrets manager. The following example uses a TF_VAR_ environment variable to populate var.schema_password:

variable "schema_password" {
  description = "Password for the Schema Registry user"
  sensitive   = true
}

resource "redpanda_user" "schema_user" {
  name                = "schema-user"
  password_wo         = var.schema_password
  password_wo_version = 1
  mechanism           = "scram-sha-256"
  cluster_api_url     = data.redpanda_cluster.byoc.cluster_api_url
  allow_deletion      = true
}

Set the variable before running Terraform:

export TF_VAR_schema_password="your-secret-password"
terraform apply

Terraform sends the value to Redpanda Cloud during apply but never writes it to .tfstate.

Rotate a write-only attribute

Because Terraform cannot detect a change in the write-only value itself, increment the corresponding _wo_version to trigger an update:

resource "redpanda_user" "schema_user" {
  name                = "schema-user"
  password_wo         = var.schema_password   # Set TF_VAR_schema_password to the new value
  password_wo_version = 2                     # Increment from 1 to trigger update
  mechanism           = "scram-sha-256"
  cluster_api_url     = data.redpanda_cluster.byoc.cluster_api_url
  allow_deletion      = true
}

After running terraform apply, the provider sends the new password to Redpanda Cloud. Neither the old nor the new value is written to state.

Create Redpanda Connect pipelines

Use the redpanda_pipeline resource to create and manage Redpanda Connect pipelines as code. Pass the pipeline configuration as YAML in the config_yaml attribute.

resource "redpanda_pipeline" "example" {
  cluster_api_url = redpanda_cluster.example.cluster_api_url
  display_name    = "example-pipeline"
  description     = "Generates data and writes it to a topic"
  state           = "running"
  allow_deletion  = true

  config_yaml = <<-YAML
    input:
      generate:
        interval: 5s
        mapping: |
          root.message = "hello world"
          root.timestamp = now()

    output:
      redpanda:
        seed_brokers:
          - $${REDPANDA_BROKERS}
        tls:
          enabled: true
        sasl:
          - mechanism: SCRAM-SHA-256
            username: $${secrets.KAFKA_USER_CONNECT}
            password: $${secrets.KAFKA_PASSWORD_CONNECT}
        topic: example-topic
  YAML

  resources = {
    memory_shares = "128Mi"
    cpu_shares    = "100m"
  }

  tags = {
    environment = "dev"
  }
}

  • cluster_api_url: The URL of the data plane API for the cluster that runs the pipeline.

  • config_yaml: The pipeline configuration in YAML format. Escape literal $\{…​} references, such as secrets and contextual variables, as $$\{…​} so that Terraform doesn’t interpret them as its own interpolation syntax. Redpanda Connect resolves them at runtime.

  • state: The desired state of the pipeline: running or stopped. The provider ensures the pipeline reaches this state after create and update operations.

  • resources (optional): The amount of memory and CPU to allocate for the pipeline. See Manage Pipeline Resources on Clusters.

  • allow_deletion (optional): Whether Terraform may destroy the pipeline. Defaults to false.

This example references the KAFKA_USER_CONNECT and KAFKA_PASSWORD_CONNECT cluster secrets for SASL authentication. You can create these secrets with the redpanda_secret resource. See Manage cluster secrets.

For the full schema and import syntax, see the redpanda_pipeline resource documentation.

Enable egress to custom destinations

On BYOC clusters, Redpanda Connect pipelines run inside the Redpanda data plane VPC in your cloud account. By default, pipelines can connect to your Redpanda cluster and to publicly routable endpoints. Outbound connections to other destinations, such as a database with a private address in a peered VPC, are blocked.

To allow pipelines to reach these destinations, add the redpanda_connect.allowed_destination_cidr_ports attribute to the redpanda_cluster resource (requires provider version >= 2.1.1). The attribute accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range:

resource "redpanda_cluster" "example" {
  # ... other cluster arguments ...

  redpanda_connect = {
    allowed_destination_cidr_ports = [
      {
        cidr       = "10.62.0.0/16" # CIDR of the VPC that hosts your database
        port_start = 5432           # PostgreSQL
      },
      {
        cidr       = "203.0.113.0/24"
        port_start = 9000
        port_end   = 9100
      }
    ]
  }
}

  • cidr: The destination IPv4 CIDR block, for example 10.62.0.0/16.

  • port_start: The start of the TCP/UDP port range, 1-65535.

  • port_end (optional): The end of the TCP/UDP port range. To allow a single port, omit this attribute. When set, the value must be greater than or equal to port_start.

You can add or change the redpanda_connect block on an existing cluster. Terraform updates the cluster in place without recreating it.

The allowlist permits outbound traffic from pipelines, but it does not create a network path to the destination. For private destinations, you must also establish routing, for example by peering the Redpanda data plane VPC with the VPC that hosts your database and adding routes in both directions.

After you apply the configuration, pipelines can connect to the allowed destinations. For example, a pipeline with a sql_insert output can write to a PostgreSQL database at a private address in the peered VPC:

resource "redpanda_pipeline" "postgres_sink" {
  cluster_api_url = redpanda_cluster.example.cluster_api_url
  display_name    = "postgres-sink"
  description     = "Writes topic data to a private PostgreSQL database"
  state           = "running"
  allow_deletion  = true

  config_yaml = <<-YAML
    input:
      redpanda:
        seed_brokers:
          - $${REDPANDA_BROKERS}
        topics: ["orders"]
        consumer_group: postgres-sink
        tls:
          enabled: true
        sasl:
          - mechanism: SCRAM-SHA-256
            username: $${secrets.KAFKA_USER_CONNECT}
            password: $${secrets.KAFKA_PASSWORD_CONNECT}

    output:
      sql_insert:
        driver: postgres
        dsn: postgres://$${secrets.PG_USER}:$${secrets.PG_PASSWORD}@db.private.example.internal:5432/orders?sslmode=require
        table: orders
        columns: ["payload"]
        args_mapping: root = [ content().string() ]
  YAML
}

Manage cluster secrets

The Redpanda Terraform provider (v2.0.0+) supports creating and managing secrets in a cluster’s secret store with the redpanda_secret resource. Cluster secrets hold sensitive values, such as pipeline credentials, that other resources reference by name using ${secrets.<NAME>}.

The secret value (secret_data) is a write-only attribute, so Terraform never stores it in state. To rotate the value, change secret_data and increment secret_data_version, following the same pattern as other write-only attributes.

variable "secret_value" {
  description = "Value to store in the secret"
  sensitive   = true
}

resource "redpanda_secret" "example" {
  name                = "EXAMPLE_SECRET"
  secret_data         = var.secret_value
  secret_data_version = 1
  scopes              = ["SCOPE_REDPANDA_CONNECT"]
  cluster_api_url     = data.redpanda_cluster.byoc.cluster_api_url
  allow_deletion      = true
}

  • name: An uppercase identifier matching ^[A-Z][A-Z0-9_]*$. You cannot rename a secret after you create it.

  • secret_data: The secret value. Requires Terraform 1.11 or later.

  • scopes: The features that can use the secret. For example, use SCOPE_REDPANDA_CONNECT for Redpanda Connect pipelines or SCOPE_REDPANDA_CLUSTER for Shadowing credentials. If a secret doesn’t have a scope that matches its consumer, the consumer fails at runtime.

  • labels (optional): Labels to organize your secrets. You can update labels on an existing secret (provider v2.1.0+).

For the full schema and import syntax, see the redpanda_secret resource documentation. To use secrets in Redpanda Connect pipelines, see Manage Secrets.

Configure cluster properties

To set cluster configuration properties on a BYOC or Dedicated cluster, add a cluster_configuration block to the redpanda_cluster resource and pass the properties as a JSON string with custom_properties_json:

resource "redpanda_cluster" "test" {
  # ... other cluster arguments ...

  cluster_configuration = {
    custom_properties_json = jsonencode({
      "audit_enabled" = true
    })
  }
}

Only the properties listed in Cluster Configuration Properties and Object Storage Properties can be configured through the provider. Setting an unsupported property returns a REASON_INVALID_INPUT error indicating that the property is not allowed. For example, kafka_batch_max_bytes cannot be set through the provider. To control the maximum message size, set the max.message.bytes property on each topic instead. For more information, see Configure Cluster Properties.

Cluster properties are not available on Serverless clusters or on BYOC and Dedicated clusters running on Azure.

By default, AWS PrivateLink only accepts connections from VPCs in the same region as your Redpanda cluster. The Redpanda Terraform provider (v1.7.0+) supports cross-region PrivateLink, which allows clients in other AWS regions to connect to your cluster through PrivateLink.

To enable cross-region PrivateLink, add the supported_regions attribute to the aws_private_link block of a redpanda_cluster resource. The attribute accepts a list of up to 50 unique AWS region identifiers from which PrivateLink endpoints can connect. If you omit it, only same-region connections are allowed.

Requirements:

  • The cluster must be deployed across multiple availability zones (multi-AZ). This is an AWS limitation for cross-region PrivateLink.

  • For BYOC clusters, the Redpanda agent IAM role must have the vpce:AllowMultiRegion and elasticloadbalancing:DescribeListenerAttributes permissions.

resource "redpanda_cluster" "example" {
  # ... other cluster arguments ...

  zones = ["use2-az1", "use2-az2", "use2-az3"]  # Cross-region PrivateLink requires a multi-AZ cluster

  aws_private_link = {
    enabled            = true
    connect_console    = true
    allowed_principals = ["arn:aws:iam::123456789012:root"]
    supported_regions  = ["us-east-1", "us-west-2", "eu-west-1"]
  }
}

output "privatelink_service_name" {
  value = redpanda_cluster.example.aws_private_link.status.service_name
}

  • enabled: Whether the Redpanda PrivateLink endpoint service is enabled.

  • connect_console: Whether Redpanda Console is connected over PrivateLink.

  • allowed_principals: The ARNs of the AWS principals allowed to access the endpoint service. To allow all principals, use an asterisk (*).

  • supported_regions (optional): The AWS regions from which clients can create PrivateLink endpoints that connect to the cluster.

You can add or change the aws_private_link block on an existing cluster. Terraform updates the cluster in place without recreating it.

After you apply the configuration, clients in the listed regions can create VPC endpoints that connect to the cluster’s endpoint service, identified by the service_name output in the example. See Create a cross-region VPC endpoint for the AWS CLI workflow, and Configure AWS PrivateLink with the Cloud API for the full PrivateLink setup, including client VPC configuration and connection testing.

Examples

The following examples use the Redpanda Terraform provider to create and manage clusters. For descriptions of resources and data sources, see the Redpanda Terraform Provider documentation.

For more information on the different cluster types mentioned in these examples, see Redpanda Cloud cluster types.

See the full list of zones and tiers available with each cloud provider in the Control Plane API reference.

Create a BYOC cluster

A BYOC (Bring Your Own Cloud) cluster runs in your own cloud account. This example creates one on AWS with a custom network, resource group, and cluster.

terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = ">= 2.1.1"
    }
  }
}

# Variables to parameterize the configuration
variable "resource_group_name" {
  description = "Name of the Redpanda resource group"
  default     = "testname"
}

variable "network_name" {
  description = "Name of the Redpanda network"
  default     = "testname"
}

variable "cluster_name" {
  description = "Name of the Redpanda BYOC cluster"
  default     = "test-cluster"
}

variable "region" {
  description = "Region for the Redpanda network and cluster"
  default     = "us-east-2"
}

variable "cloud_provider" {
  description = "Cloud provider for the Redpanda network"
  default     = "aws"
}

variable "zones" {
  description = "List of availability zones for the cluster"
  type        = list(string)
  default     = ["use2-az1", "use2-az2", "use2-az3"]
}

variable "cidr_block" {
  description = "CIDR block for the Redpanda network"
  default     = "10.0.0.0/20"
}

variable "throughput_tier" {
  description = "Throughput tier for the cluster"
  default     = "tier-1-aws-v2-x86"
}

# Redpanda provider configuration
provider "redpanda" {}

# Create a Redpanda resource group
resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

# Create a Redpanda network
resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "byoc"  # Specify BYOC cluster type
  cidr_block        = var.cidr_block
}

# Create a Redpanda BYOC cluster
resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "byoc"
  connection_type   = "public"  # Publicly accessible cluster
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true      # Allow the cluster to be deleted
  tags = {                      # Add metadata tags
    "environment" = "dev"
  }
}

Enable Redpanda SQL on a BYOC cluster

Use the rpsql block on a redpanda_cluster resource (requires provider version >= 2.1.0) to enable and manage Redpanda SQL on a BYOC cluster. You can add or change rpsql settings on an existing cluster without recreating it.

Attribute Type Required Description

enabled

Boolean

No

Whether Redpanda SQL is enabled. Default: false.

replicas

Number

No

Number of SQL compute replicas. Scalable in place. Default: 1.

zones

List of String

No

For multi-AZ clusters, the single availability zone to deploy the SQL engine into. Provide one zone from the cluster’s zones list. If omitted, a zone is chosen automatically. Redpanda locks the zone when SQL is first enabled. You cannot change it afterward. See Enable Redpanda SQL on a BYOC cluster.

url

String

No (computed)

SQL engine endpoint URL. Empty while disabled; populated by the server when SQL is enabled.

 
terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = ">= 2.1.0"
    }
  }
}

variable "cluster_name" {
  description = "Name of the Redpanda BYOC cluster"
  default     = "sql-enabled-cluster"
}

variable "resource_group_id" {
  description = "ID of the Redpanda resource group"
}

variable "network_id" {
  description = "ID of the Redpanda network"
}

variable "region" {
  description = "Region for the cluster"
  default     = "us-east-2"
}

variable "throughput_tier" {
  description = "Throughput tier for the cluster"
  default     = "tier-1-aws-v2-x86"
}

variable "zones" {
  description = "List of availability zones for the cluster"
  type        = list(string)
  default     = ["use2-az1", "use2-az2", "use2-az3"]
}

provider "redpanda" {}

resource "redpanda_cluster" "example" {
  name              = var.cluster_name
  resource_group_id = var.resource_group_id
  network_id        = var.network_id
  cloud_provider    = "aws"
  region            = var.region
  cluster_type      = "byoc"
  connection_type   = "public"
  throughput_tier   = var.throughput_tier
  zones             = var.zones

  rpsql = {
    enabled  = true
    replicas = 3
    zones    = ["use2-az1"]
  }
}

output "sql_url" {
  value = redpanda_cluster.example.rpsql.url
}

To scale Redpanda SQL, update replicas and run terraform apply. To disable Redpanda SQL, set enabled = false. Both operations update the cluster in place. Terraform does not recreate the cluster.

For the full enable workflow using the Cloud Console or Cloud API, see Enable Redpanda SQL on a BYOC cluster.

Create a Dedicated cluster

Redpanda fully manages Dedicated clusters for consistent performance. This example creates one on AWS with specific zones and a usage tier.

terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = ">= 2.1.1"
    }
  }
}

# Variables for configuration
variable "resource_group_name" {
  description = "Name of the Redpanda resource group"
  default     = "test-dedicated-group"
}

variable "network_name" {
  description = "Name of the Redpanda network"
  default     = "dedicated-network"
}

variable "cluster_name" {
  description = "Name of the Redpanda dedicated cluster"
  default     = "dedicated-cluster"
}

variable "region" {
  description = "Region for the Redpanda network and cluster"
  default     = "us-west-1"
}

variable "cloud_provider" {
  description = "Cloud provider for the Redpanda network"
  default     = "aws"
}

variable "zones" {
  description = "List of availability zones for the cluster"
  type        = list(string)
  default     = ["usw1-az1", "usw1-az2", "usw1-az3"]
}

variable "cidr_block" {
  description = "CIDR block for the Redpanda network"
  default     = "10.1.0.0/20"
}

variable "throughput_tier" {
  description = "Throughput tier for the dedicated cluster"
  default     = "tier-1-aws-v2-arm"
}

# Redpanda provider configuration
provider "redpanda" {}

# Create a Redpanda resource group
resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

# Create a Redpanda network
resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"  # Specify Dedicated cluster type
  cidr_block        = var.cidr_block
}

# Create a Redpanda dedicated cluster
resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"
  connection_type   = "public"
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true
  aws_private_link = {  # Configure AWS PrivateLink for dedicated clusters
    enabled            = true
    connect_console    = true
    allowed_principals = ["arn:aws:iam::123456789024:root"]
    supported_regions  = ["us-east-1", "us-west-2"]  # Optional: Enable cross-region PrivateLink (requires a multi-AZ cluster)
  }
  tags = {
    "environment" = "dev"
  }
}

For details on the aws_private_link block, including the multi-AZ requirement for cross-region PrivateLink, see Configure cross-region AWS PrivateLink.

Create a Serverless cluster

A Serverless cluster is cost-effective and scales automatically based on usage. This example creates a cluster in the us-east-1 region with minimal configuration.

terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = ">= 2.1.1"
    }
  }
}

# Redpanda provider configuration
provider "redpanda" {}

# Define a resource group for the Serverless cluster
resource "redpanda_resource_group" "test" {
  name = var.resource_group_name  # Name of the resource group
}

# Create a Serverless cluster
resource "redpanda_serverless_cluster" "test" {
  name              = var.cluster_name                  # Name of the Serverless cluster
  resource_group_id = redpanda_resource_group.test.id   # Link to the resource group
  serverless_region = var.region                        # Specify the region for the cluster
}

# Variables for parameterizing the configuration
variable "resource_group_name" {
  description = "Name of the Redpanda resource group"
  default     = "testgroup"  # Default name for the resource group
}

variable "cluster_name" {
  description = "Name of the Redpanda Serverless cluster"
  default     = "testname"   # Default name for the Serverless cluster
}

variable "region" {
  description = "Region for the Serverless cluster"
  default     = "us-east-1"  # Default region for the cluster
}

Manage an existing cluster

To manage resources in an existing cluster, reference it by cluster ID using a data source. The data source exposes a cluster_api_url attribute that other resources (such as redpanda_topic) reference to connect to the cluster.

data "redpanda_cluster" "byoc" {
  id = "byoc-cluster-id"
}

resource "redpanda_topic" "example" {
  name               = "example-topic"
  partition_count    = 3
  replication_factor = 3
  cluster_api_url    = data.redpanda_cluster.byoc.cluster_api_url
}

Manage Schema Registry and Schema Registry ACLs

You can also use Terraform to manage data plane resources, such as schemas and access controls, through the Redpanda Schema Registry.

Use the Redpanda Terraform provider to create, update, and delete schemas and manage fine-grained access control for Schema Registry resources.

You can use the following Terraform resources:

  • redpanda_schema: Defines and manages schemas in the Schema Registry.

  • redpanda_schema_registry_acl: Defines access control policies for Schema Registry subjects or registry-wide operations.

Create a schema

The redpanda_schema resource registers a schema. Each schema belongs to a subject, a logical name used for schema versioning. When you create or update a schema, Redpanda validates its compatibility level.

data "redpanda_cluster" "byoc" {
  id = "byoc-cluster-id"
}

resource "redpanda_user" "schema_user" {
  name                = "schema-user"
  password_wo         = var.schema_password
  password_wo_version = 1
  mechanism           = "scram-sha-256"
  cluster_api_url     = data.redpanda_cluster.byoc.cluster_api_url
  allow_deletion      = true
}

resource "redpanda_schema" "user_events" {
  cluster_id  = data.redpanda_cluster.byoc.id
  subject     = "user_events-value"
  schema_type = "AVRO"

  schema = jsonencode({
    type = "record"
    name = "UserEvent"
    fields = [
      { name = "user_id", type = "string" },
      { name = "event_type", type = "string" },
      { name = "timestamp", type = "long" }
    ]
  })

  username            = redpanda_user.schema_user.name
  password_wo         = var.schema_password
  password_wo_version = 1
}

In this example:

  • cluster_id identifies the Redpanda cluster where the schema is stored.

  • subject defines the logical name under which schema versions are registered.

  • schema_type specifies the serialization type (AVRO, JSON, or PROTOBUF).

  • schema provides the full schema definition, encoded with jsonencode().

  • username identifies the Schema Registry user. Set password_wo to the password value, and increment password_wo_version to trigger updates. For details, see Manage sensitive attributes with write-only fields.

Store credentials securely

Use Terraform 1.11+ write-only attributes (such as password_wo) to keep Schema Registry credentials out of your .tfstate file. For details, see Manage sensitive attributes with write-only fields.

For short-lived credentials or CI/CD usage, you can also export the Schema Registry credentials as provider-level environment variables. The provider reads them automatically:

export REDPANDA_SR_USERNAME=schema-user
export REDPANDA_SR_PASSWORD="schema-registry-password"

If you must use the deprecated plaintext password attribute (for example, on Terraform versions earlier than 1.11), declare a sensitive Terraform variable and inject the value at runtime to avoid committing secrets to source control:

variable "schema_password" {
  description = "Password for the Schema Registry user"
  sensitive   = true
}
export TF_VAR_schema_password="schema-registry-password"

Manage Schema Registry ACLs

The redpanda_schema_registry_acl resource configures fine-grained access control for Schema Registry subjects or registry-wide operations. Each ACL specifies which principal can perform which operations on a subject or the registry.

resource "redpanda_schema_registry_acl" "allow_user_read" {
  cluster_id    = data.redpanda_cluster.byoc.id
  principal     = "User:${redpanda_user.schema_user.name}"
  resource_type = "SUBJECT"        # SUBJECT or REGISTRY
  resource_name = "user_events-value"
  pattern_type  = "LITERAL"        # LITERAL or PREFIXED
  host          = "*"
  operation     = "READ"           # READ, WRITE, DELETE, DESCRIBE, etc.
  permission    = "ALLOW"          # ALLOW or DENY
  username            = redpanda_user.schema_user.name
  password_wo         = var.schema_password
  password_wo_version = 1
}

In this example:

  • cluster_id identifies the cluster that hosts the Schema Registry.

  • principal specifies the user or service account (for example, User:alice).

  • resource_type determines whether the ACL applies to a specific SUBJECT or the entire REGISTRY.

  • resource_name defines the subject name (use * for wildcard).

  • pattern_type controls how the resource name is matched (LITERAL or PREFIXED).

  • operation defines the permitted action (READ, WRITE, DELETE, and others).

  • permission defines whether the operation is allowed or denied.

  • host specifies the host filter (typically "*" for all hosts).

  • username identifies the Schema Registry principal. Set password_wo to the password value, and increment password_wo_version to trigger updates. For details, see Manage sensitive attributes with write-only fields.

To manage Schema Registry ACLs, the user must have cluster-level ALTER permissions. This is typically granted through a Kafka ACL with ALTER on the CLUSTER resource.

Combine schema and ACLs

You can define both the schema and its ACLs in a single configuration to automate schema registration and access setup.

data "redpanda_cluster" "byoc" {
  id = "byoc-cluster-id"
}

resource "redpanda_user" "schema_user" {
  name                = "schema-user"
  password_wo         = var.schema_password
  password_wo_version = 1
  mechanism           = "scram-sha-256"
  cluster_api_url     = data.redpanda_cluster.byoc.cluster_api_url
  allow_deletion      = true
}

resource "redpanda_schema" "user_events" {
  cluster_id  = data.redpanda_cluster.byoc.id
  subject     = "user_events-value"
  schema_type = "AVRO"

  schema = jsonencode({
    type = "record"
    name = "UserEvent"
    fields = [
      { name = "user_id", type = "string" },
      { name = "event_type", type = "string" },
      { name = "timestamp", type = "long" }
    ]
  })

  username            = redpanda_user.schema_user.name
  password_wo         = var.schema_password
  password_wo_version = 1
}

resource "redpanda_schema_registry_acl" "user_events_acl" {
  cluster_id    = data.redpanda_cluster.byoc.id
  principal     = "User:${redpanda_user.schema_user.name}"
  resource_type = "SUBJECT"
  resource_name = redpanda_schema.user_events.subject
  pattern_type  = "LITERAL"
  host          = "*"
  operation     = "READ"
  permission    = "ALLOW"
  username            = redpanda_user.schema_user.name
  password_wo         = var.schema_password
  password_wo_version = 1
}

This configuration registers an Avro schema for the user_events subject and grants a service account permission to read it from the Schema Registry.

Delete resources

The terraform destroy command deletes all resources defined in your configuration.

Terraform deletes dependent resources in the correct order. For example, Terraform removes a cluster that depends on a network before it deletes the network.

Delete all resources

  1. Navigate to the directory containing your Terraform configuration.

  2. Run the following command:

    terraform destroy

  3. Review the destruction plan Terraform generates. It lists all the resources to be deleted.

  4. Confirm by typing yes when prompted.

  5. Wait for the process to complete. Terraform deletes the resources and displays a summary.

Delete specific resources

If you only want to delete a specific resource rather than everything in your configuration, use the -target flag with terraform destroy. For example:

terraform destroy -target=redpanda_network.example

This deletes only the redpanda_network.example resource.

Suggested reading