Local development brokers
compose.yaml at the repository root starts single-node Kafka, Apache Pulsar,
and RabbitMQ brokers, an ElasticMQ SQS emulator, the Google Cloud Pub/Sub
emulator, and the Azure Service Bus emulator, with web consoles for the
brokers. Use it to run the broker examples, the
ignored live tests, and manual experiments. It requires Docker with Compose v2.
Quick start
docker compose up -d --wait # start the brokers and wait until they are healthy
cargo kafka-produce # publish sample orders to Kafka
cargo kafka-process # run the Kafka pipeline until Ctrl-C
cargo test-live # run every ignored live test
docker compose down # stop the brokers and discard their data
The cargo commands are aliases defined in .cargo/config.toml; they only
run cargo run --example and cargo test with the required features. Cargo
aliases cannot start Docker, so the brokers are always started and stopped with
docker compose directly.
Managing the environment
| Command | Effect |
|---|---|
docker compose up -d --wait |
Starts every service and returns when the brokers pass their health checks |
docker compose up -d --wait kafka pulsar rabbitmq elasticmq pubsub servicebus-health |
Starts only the brokers, without the separate web consoles; servicebus-health brings the Service Bus emulator and its SQL Server |
docker compose ps --all |
Shows the state of each service, including the finished pulsar-init |
docker compose logs -f pulsar |
Follows the logs of one service |
docker compose down |
Stops and removes the containers |
The brokers keep no named volumes, so docker compose down discards every
topic, offset, and subscription; the next up starts from empty brokers.
docker compose stop keeps the containers and their data for a later
docker compose start.
Services
| Service | Address | Purpose |
|---|---|---|
| Kafka | localhost:9092 |
Kafka 4.1 in KRaft mode |
| Kafka UI | http://localhost:8081 | Topics, messages, and consumer groups (kafbat kafka-ui) |
| Pulsar | pulsar://localhost:6650 |
Pulsar 4.0 standalone |
| Pulsar admin API | http://localhost:8080 | pulsar-admin --admin-url and REST |
| Pulsar Manager | http://localhost:9527 | Tenants, topics, and subscriptions; log in as admin / apachepulsar |
| RabbitMQ | amqp://guest:guest@localhost:5672/%2f |
RabbitMQ 4.1 |
| RabbitMQ management | http://localhost:15672 | Queues, exchanges, connections, and the management HTTP API; log in as guest / guest |
| ElasticMQ | http://localhost:9324 | SQS-compatible API; queue URLs look like http://localhost:9324/000000000000/orders |
| ElasticMQ UI | http://localhost:9325 | Queues and their message counts |
| Pub/Sub emulator | localhost:8085 |
Pub/Sub gRPC and REST APIs for the beavers project; set the adapter endpoint to http://localhost:8085 |
| Service Bus emulator | localhost:5673 |
Azure Service Bus over plain AMQP; connection string Endpoint=sb://localhost:5673;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true |
| Service Bus emulator management | http://localhost:5300 | Health endpoint and the queue management API the live tests use |
The examples and live tests use these addresses by default. Set these variables to target other brokers:
| Variable | Default | Used by |
|---|---|---|
KAFKA_BROKERS |
localhost:9092 |
Kafka examples and live tests |
PULSAR_URL |
pulsar://localhost:6650 in examples, pulsar://127.0.0.1:6650 in tests |
Pulsar examples and live tests |
HTTP_SINK_URL |
http://127.0.0.1:8090/orders |
kafka_to_http example |
PULSAR_ADMIN_ADDR |
127.0.0.1:8080 |
Live tests that create Pulsar topics or subscriptions |
RABBITMQ_URL |
amqp://guest:guest@127.0.0.1:5672/%2f |
RabbitMQ live tests |
RABBITMQ_MANAGEMENT_ADDR |
127.0.0.1:15672 |
RabbitMQ live tests that declare queues or close connections |
SQS_ENDPOINT |
http://127.0.0.1:9324 |
SQS live tests |
PUBSUB_EMULATOR_HOST |
127.0.0.1:8085 |
Pub/Sub live tests |
SERVICEBUS_CONNECTION_STRING |
The emulator connection string on 127.0.0.1:5673 |
Service Bus live tests |
SERVICEBUS_ADMIN_ADDR |
127.0.0.1:5300 |
Service Bus live tests that create or delete queues |
Broker configuration
Kafka auto-creates topics with three partitions, so per-partition ordering and
parallelism are visible with a single broker. Replication factors are 1 and the
transaction state log accepts a single replica, which allows transactional
producers against the one broker. Applications on the host use the listener
advertised as localhost:9092; containers use kafka:29092.
Pulsar runs in standalone mode without the functions worker or stream storage.
standalone.conf settings can be added to the pulsar service environment, which apply-config-from-env.py
applies before the broker starts.
The one-shot pulsar-init service runs after the broker and Pulsar Manager
start. It creates the beavers-examples subscription on
persistent://public/default/orders at the earliest position, because a new
Pulsar subscription otherwise starts at the latest message, and it creates the
Pulsar Manager login. Pulsar Manager is preconfigured with a local
environment pointing at the broker. The credentials are for local use only.
RabbitMQ runs with its management plugin and the default guest user, which
the broker accepts from the host through the published port. It starts without
queues; the live tests declare their own through the management API.
ElasticMQ implements the SQS API in memory and accepts any credentials and
region. Point a handle at it with
Sqs::new().region("elasticmq").endpoint_url("http://localhost:9324") and
static credentials such as .credentials(SqsCredentials::new("local", "local")).
It starts without queues; create one with the AWS CLI
(aws --endpoint-url http://localhost:9324 sqs create-queue --queue-name orders)
or the query API, as the live tests do. It does not reproduce every AWS limit
and behavior, so verify production settings against SQS itself.
The Pub/Sub emulator, from the google/cloud-sdk emulators image, keeps topics
and subscriptions in memory and needs no credentials. Connect to it with
PubSub::new().endpoint("http://localhost:8085").anonymous(). It starts without topics; create them through
its REST API, as the live tests do:
curl -X PUT http://localhost:8085/v1/projects/beavers/topics/orders
curl -X PUT http://localhost:8085/v1/projects/beavers/subscriptions/orders \
-H 'Content-Type: application/json' -d '{"topic":"projects/beavers/topics/orders"}'
It does not reproduce every Pub/Sub limit and behavior, so verify production settings against Pub/Sub itself.
The Service Bus emulator stores its entities in the servicebus-sql SQL Server
container and reads its namespace from dev/servicebus/config.json, which
defines the beavers-events topic with the beavers-test subscription.
Connect to it with ServiceBus::from_connection_string(...) and the connection
string listed above; UseDevelopmentEmulator=true selects plain AMQP, which
.emulator() sets on a handle built with ServiceBus::new. Its
management API on port 5300 creates and deletes queues
(PUT /{queue}?api-version=2021-05 with an Atom QueueDescription), as the
live tests do, but cannot create topic subscriptions, so add topics and
subscriptions to the configuration file and recreate the container. Its AMQP
port is published as 5673 because RabbitMQ uses 5672. The emulator image has no
shell, so the servicebus-health container polls its health endpoint for
docker compose up --wait. Both images are amd64 only and need the Microsoft
license accepted, which ACCEPT_EULA does. The emulator serves plain AMQP
without TLS, refuses binary application properties, and applies smaller quotas
than a real namespace, so verify production settings against Azure.
Examples
| Command | Example | Flow |
|---|---|---|
cargo kafka-produce [count] |
kafka_orders |
Publishes orders, 10 by default, to Kafka orders |
cargo kafka-process |
kafka_orders |
Kafka orders → order-events |
cargo pulsar-produce [count] |
pulsar_orders |
Publishes orders, 10 by default, to Pulsar orders |
cargo pulsar-process |
pulsar_orders |
Pulsar orders → order-events with a Key_Shared subscription |
cargo kafka-to-pulsar |
kafka_to_pulsar |
Kafka orders → Pulsar orders-from-kafka |
cargo http-receive |
kafka_to_http |
HTTP server on 127.0.0.1:8090 that prints the orders it receives |
cargo kafka-to-http |
kafka_to_http |
Kafka orders → HTTP POST to HTTP_SINK_URL |
Each alias expands to cargo run --example <name> --features <features> -- <command>, and arguments after the alias are appended, as in
cargo kafka-produce 100.
The produce commands are ordinary beavers applications that publish an
IterSource of JSON orders keyed by customer through a broker sink, and end
when the source is exhausted. The process commands and kafka-to-pulsar run
a subscription until Ctrl-C. process registers a value handler, so each
output inherits the input's key and headers or properties.
kafka_to_http sends each order to HTTP_SINK_URL, by default the
http-receive server, with an idempotency-key header built from the record's
topic, partition, and offset. Stop http-receive while kafka-to-http runs to
see the sink retry, stop after its publish retries, and resume from the
uncommitted offset on the next run.
kafka_to_pulsar maps metadata explicitly with MapMetadata: the Kafka key
becomes the Pulsar key and the tenant and request-id headers with UTF-8
values become Pulsar properties. It skips Kafka tombstones with Tombstones::skip().
The Kafka examples read from the earliest offset when their consumer group has
no committed offset, and the Pulsar example uses the subscription created by
pulsar-init, so orders produced before a first process run are consumed.
Run a process command again to see that committed or acknowledged deliveries
are not processed twice.
GitHub Actions runs each example flow in its own Example job, which starts
the services the flow needs and runs .github/scripts/example.sh. The script
runs the commands above and checks that 10 outputs arrive on the output topic
or at the HTTP receiver. Run the script locally with the same scenario name, such
as .github/scripts/example.sh kafka-to-http, after starting the brokers.
Inspecting broker state
Kafka UI shows topics, partitions, messages, and consumer group offsets. Pulsar Manager shows tenants, namespaces, topics, and subscription backlogs. The same information is available from the command line:
docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:29092 --topic order-events --from-beginning \
--property print.key=true
docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server localhost:29092 --describe --group beavers-examples
docker compose exec pulsar bin/pulsar-admin topics stats \
persistent://public/default/order-events
Live tests
The Kafka, Pulsar, RabbitMQ, SQS, Pub/Sub, and Service Bus adapter tests in
tests/kafka.rs, tests/pulsar.rs, tests/rabbitmq.rs, tests/sqs.rs,
tests/pubsub.rs, and tests/servicebus.rs and the end-to-end pipeline tests in tests/pipelines.rs are ignored by
default. With the brokers running:
cargo test-live # cargo test --features kafka,pulsar,rabbitmq,sqs,pubsub,servicebus -- --ignored
cargo test-live pipeline # only tests whose name contains "pipeline"
The pipeline tests run the same flows as the examples as complete applications: a Kafka pipeline that inherits keys and headers and commits every consumed offset, a Pulsar pipeline that inherits keys and properties and leaves no subscription backlog, and a Kafka-to-Pulsar pipeline with an explicit metadata mapping. They create Pulsar subscriptions through the admin API before producing input.
Every live test uses unique topic, group, subscription, and queue names, so the
tests can run repeatedly without resetting the brokers. GitHub Actions runs them
in the Live broker tests job, which starts the kafka, pulsar,
rabbitmq, elasticmq, and pubsub services and the Service Bus emulator from
compose.yaml and runs cargo test --all-features -- --ignored.
Troubleshooting
- A port is already in use. The environment publishes ports 9092, 6650,
8080, 8081, 9527, 5672, 15672, 9324, 9325, 8085, 5673, and 5300. Stop the conflicting process, or change the host side
of the mapping in
compose.yaml; the Kafka port must stay 9092 unless the advertisedHOSTlistener changes with it. - An example or test waits without output. The brokers are probably not
running. Check
docker compose ps. A Kafka producer keeps retrying an unreachable broker until its delivery timeout expires. - A Pulsar consumer misses messages produced before it started. New
subscriptions start at the latest message. Create the subscription first, as
pulsar-initand the pipeline tests do. - Stale state from an earlier session. Run
docker compose downand start again.