Streaming

rpk debug bundle

The rpk debug bundle command collects environment data that can help debug and diagnose issues with a Redpanda cluster, a broker, or the machine it’s running on. It then bundles the collected data into a ZIP file, called a diagnostics bundle.

In Kubernetes, you must run the rpk debug bundle command inside a container that’s running a Redpanda broker.

Diagnostic bundle files

The files and directories in the diagnostics bundle differ depending on the environment in which Redpanda is running:

Common files

  • Kafka metadata: Broker configs, topic configs, start/committed/end offsets, groups, group commits.

  • Controller logs: The controller logs directory up to a limit set by --controller-logs-size-limit flag

  • Data directory structure: A file describing the data directory’s contents.

  • redpanda configuration: The redpanda configuration file (redpanda.yaml; SASL credentials are stripped).

  • /proc/cpuinfo: CPU information like make, core count, cache, frequency.

  • /proc/interrupts: IRQ distribution across CPU cores.

  • Resource usage data: CPU usage percentage, free memory available for the redpanda process.

  • Clock drift: The ntp clock delta (using pool.ntp.org as a reference) and round trip time.

  • Admin API calls: Cluster and broker configurations, cluster health data, CPU profiles, and license key information.

  • Broker metrics: The broker’s Prometheus metrics, fetched through its admin API (/metrics and /public_metrics).

Bare-metal

  • Kernel: The kernel logs ring buffer (syslog) and parameters (sysctl).

  • DNS: The DNS info as reported by 'dig', using the hosts in /etc/resolv.conf.

  • Disk usage: The disk usage for the data directory, as output by 'du'.

  • Redpanda logs: The broker’s Redpanda logs written to journald since yesterday (00:00:00 of the previous day based on systemd.time). If --logs-since or --logs-until is passed, only the logs within the resulting time frame are included.

  • Socket info: The active sockets data output by 'ss'.

  • Running process info: As reported by 'top'.

  • Virtual memory stats: As reported by 'vmstat'.

  • Network config: As reported by 'ip addr'.

  • lspci: List the PCI buses and the devices connected to them.

  • dmidecode: The DMI table contents. Only included if this command is run as root.

Extra requests for partitions

You can provide a list of partitions to save additional admin API requests specifically for those partitions.

The partition flag accepts the format [namespace/]topic/partition[,partition…​] where the namespace is optional. If the namespace is not provided, rpk will assume 'kafka'. For example:

Topic 'foo', partitions 1, 2 and 3:

--partition foo/1,2,3

Namespace _redpanda-internal, topic 'bar', partition 2:

--partition _redpanda-internal/bar/2

If you have an upload URL from the Redpanda support team, provide it in the --upload-url flag to upload your diagnostics bundle to Redpanda.

Kubernetes

  • Kubernetes Resources: Kubernetes manifests for all resources in the given Kubernetes namespace using --namespace, or the shorthand version -n.

  • redpanda logs: Logs of each Pod in the given Kubernetes namespace. If --logs-since is passed, only the logs within the given timeframe are included.

Usage

rpk debug bundle [flags]

Flags

Value Type Description

--controller-logs-size-limit

string

The size limit of the controller logs that can be stored in the bundle. For example: 3MB, 1GiB.

--cpu-profiler-wait

duration

How long to collect samples for the CPU profiler. For example: 30s, 1.5m. Must be higher than 15s.

--kafka-connections-limit

int

The maximum number of Kafka connections to store in the bundle.

-l, --label-selector

stringArray

Comma-separated label selectors to filter your resources. For example: <label>=<value>,<label>=<value> (K8s only).

--logs-since

string

Include logs dated from specified date onward; (journalctl date format: YYYY-MM-DD, yesterday, or today). See the journalctl documentation for more options.

--logs-size-limit

string

Read the logs until the given size is reached. For example: 3MB, 1GiB.

--logs-until

string

Include logs older than the specified date; (journalctl date format: YYYY-MM-DD, yesterday, or today). See the journalctl documentation for more options.

--metrics-interval

duration

Interval between metrics snapshots. For example: 30s, 1.5m.

--metrics-samples

int

Number of metrics samples to take (at the interval of --metrics-interval). Must be >= 2.

-n, --namespace

string

The namespace to use to collect the resources from (K8s only).

-o, --output

string

The file path where the debug file will be written (default ./<timestamp>-bundle.zip).

-p, --partition

stringArray

Comma-separated partition IDs. When provided, rpk saves extra Admin API requests for those partitions. See the help for extended usage.

--timeout

duration

How long to wait for child commands to execute. For example: 30s, 1.5m.

--upload-url

string

If provided, where to upload the bundle in addition to creating a copy on disk.

Global flags

Value Type Description

--config

string

Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml.

-X, --config-opt

stringArray

Override rpk configuration settings; -X help for detail or -X list for terser detail.

--ignore-profile

bool

Ignore rpk.yaml and redpanda.yaml; use default settings.

--profile

string

rpk profile to use.

-v, --verbose

bool

Enable verbose logging.

Result

The files and directories in the diagnostics bundle differ depending on the environment in which Redpanda is running.

  • Linux

  • Kubernetes

File or Directory Description

/admin

Cluster and broker configurations, cluster health data, and license key information.

/controller

Binary-encoded replicated logs that contain the history of configuration changes as well as internal settings.
Redpanda can replay the events that took place in the cluster to arrive at a similar state.

data-dir.txt

Metadata for the Redpanda data directory of the broker on which the rpk debug bundle command was executed.

kafka.json

Kafka metadata, such as broker configuration, topic configuration, offsets, groups, and group commits.

redpanda.log

Redpanda logs for the broker.
If --logs-since is passed, only the logs within the given timeframe are included.

/metrics

Prometheus metrics from both the /metrics endpoint and the public_metrics endpoint.

/proc

CPU details of the broker on which the rpk debug bundle command was executed.
The directory includes a cpuinfo file with CPU information such as processor model, core count, cache size, frequency, as well as an interrupts file that contains IRQ distribution across CPU cores.

redpanda.yaml

The Redpanda configuration file of the broker on which the rpk debug bundle command was executed.
Sensitive data is removed and replaced with (REDACTED).

resource-usage.json

Redpanda resource usage data, such as CPU usage and free memory available.

/utils

Data from the node on which the broker is running. This directory includes:

  • du.txt: The disk usage of the data directory of the broker on which the rpk debug bundle command was executed, as output by the du command.

  • ntp.txt: The NTP clock delta (using ntppool as a reference) and round trip time of the broker on which the rpk debug bundle command was executed.

  • uname.txt: System information, such as the kernel version, hostname, and architecture, as output by the uname command.

  • dig.txt: The DNS resolution information for the node, as output by the dig command.

  • dmidecode.txt: System hardware information from the node, as output by the the dmidecode command. Requires root privileges.

  • free.txt: The amount of free and used memory on the node, as output by the free command.

  • ip.txt: Network interface information, including IP addresses and network configuration, as output by the ip command.

  • lspci.txt: Information about PCI devices on the node, as output by the lspci command.

  • ss.txt: Active socket connections, as output by the ss command, showing network connections, listening ports, and more.

  • sysctl.txt: Kernel parameters of the system, as output by the sysctl command.

  • top.txt: The top processes by CPU and memory usage, as output by the top command.

  • vmstat.txt: Virtual memory statistics, including CPU usage, memory, and IO operations, as output by the vmstat command.
File or Directory Description

/admin

Cluster and broker configurations, cluster health data, and license key information.

/controller

Binary-encoded replicated logs that contain the history of configuration changes as well as internal settings.
Redpanda can replay the events that took place in the cluster to arrive at a similar state.

data-dir.txt

Metadata for the Redpanda data directory of the broker on which the rpk debug bundle command was executed.

kafka.json

Kafka metadata, such as broker configuration, topic configuration, offsets, groups, and group commits.

redpanda.log

Redpanda logs for the broker.
If --logs-since is passed, only the logs within the given timeframe are included.

/metrics

Prometheus metrics from both the /metrics endpoint and the public_metrics endpoint.

/proc

CPU details of the broker on which the rpk debug bundle command was executed.
The directory includes a cpuinfo file with CPU information such as processor model, core count, cache size, frequency, as well as an interrupts file that contains IRQ distribution across CPU cores.

redpanda.yaml

The Redpanda configuration file of the broker on which the rpk debug bundle command was executed.
Sensitive data is removed and replaced with (REDACTED).

resource-usage.json

Redpanda resource usage data, such as CPU usage and free memory available.

/utils

Data from the node on which the broker is running. This directory includes:

  • du.txt: The disk usage of the data directory of the broker on which the rpk debug bundle command was executed, as output by the du command.

  • ntp.txt: The NTP clock delta (using ntppool as a reference) and round trip time of the broker on which the rpk debug bundle command was executed.

  • uname.txt: System information, such as the kernel version, hostname, and architecture, as output by the uname command.

  • dig.txt: The DNS resolution information for the node, as output by the dig command.

  • dmidecode.txt: System hardware information from the node, as output by the the dmidecode command. Requires root privileges.

  • free.txt: The amount of free and used memory on the node, as output by the free command.

  • ip.txt: Network interface information, including IP addresses and network configuration, as output by the ip command.

  • lspci.txt: Information about PCI devices on the node, as output by the lspci command.

  • ss.txt: Active socket connections, as output by the ss command, showing network connections, listening ports, and more.

  • sysctl.txt: Kernel parameters of the system, as output by the sysctl command.

  • top.txt: The top processes by CPU and memory usage, as output by the top command.

  • vmstat.txt: Virtual memory statistics, including CPU usage, memory, and IO operations, as output by the vmstat command.

Examples

This section provides examples of how to use rpk debug bundle.

Collect Redpanda logs from a specific timeframe.

rpk debug bundle --logs-since "2022-02-01" --logs-size-limit 3MiB

Use a custom Kubernetes namespace.

rpk debug bundle --namespace <namespace>