beave-rs

Local development brokers

compose.yaml at the repository root starts single-node Kafka, Apache Pulsar, and RabbitMQ brokers, an ElasticMQ SQS emulator, the Google Cloud Pub/Sub emulator, and the Azure Service Bus emulator, with web consoles for the brokers. Use it to run the broker examples, the ignored live tests, and manual experiments. It requires Docker with Compose v2.

Quick start

docker compose up -d --wait   # start the brokers and wait until they are healthy
cargo kafka-produce           # publish sample orders to Kafka
cargo kafka-process           # run the Kafka pipeline until Ctrl-C
cargo test-live               # run every ignored live test
docker compose down           # stop the brokers and discard their data

The cargo commands are aliases defined in .cargo/config.toml; they only run cargo run --example and cargo test with the required features. Cargo aliases cannot start Docker, so the brokers are always started and stopped with docker compose directly.

Managing the environment

Command Effect
docker compose up -d --wait Starts every service and returns when the brokers pass their health checks
docker compose up -d --wait kafka pulsar rabbitmq elasticmq pubsub servicebus-health Starts only the brokers, without the separate web consoles; servicebus-health brings the Service Bus emulator and its SQL Server
docker compose ps --all Shows the state of each service, including the finished pulsar-init
docker compose logs -f pulsar Follows the logs of one service
docker compose down Stops and removes the containers

The brokers keep no named volumes, so docker compose down discards every topic, offset, and subscription; the next up starts from empty brokers. docker compose stop keeps the containers and their data for a later docker compose start.

Services

Service Address Purpose
Kafka localhost:9092 Kafka 4.1 in KRaft mode
Kafka UI http://localhost:8081 Topics, messages, and consumer groups (kafbat kafka-ui)
Pulsar pulsar://localhost:6650 Pulsar 4.0 standalone
Pulsar admin API http://localhost:8080 pulsar-admin --admin-url and REST
Pulsar Manager http://localhost:9527 Tenants, topics, and subscriptions; log in as admin / apachepulsar
RabbitMQ amqp://guest:guest@localhost:5672/%2f RabbitMQ 4.1
RabbitMQ management http://localhost:15672 Queues, exchanges, connections, and the management HTTP API; log in as guest / guest
ElasticMQ http://localhost:9324 SQS-compatible API; queue URLs look like http://localhost:9324/000000000000/orders
ElasticMQ UI http://localhost:9325 Queues and their message counts
Pub/Sub emulator localhost:8085 Pub/Sub gRPC and REST APIs for the beavers project; set the adapter endpoint to http://localhost:8085
Service Bus emulator localhost:5673 Azure Service Bus over plain AMQP; connection string Endpoint=sb://localhost:5673;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true
Service Bus emulator management http://localhost:5300 Health endpoint and the queue management API the live tests use

The examples and live tests use these addresses by default. Set these variables to target other brokers:

Variable Default Used by
KAFKA_BROKERS localhost:9092 Kafka examples and live tests
PULSAR_URL pulsar://localhost:6650 in examples, pulsar://127.0.0.1:6650 in tests Pulsar examples and live tests
HTTP_SINK_URL http://127.0.0.1:8090/orders kafka_to_http example
PULSAR_ADMIN_ADDR 127.0.0.1:8080 Live tests that create Pulsar topics or subscriptions
RABBITMQ_URL amqp://guest:guest@127.0.0.1:5672/%2f RabbitMQ live tests
RABBITMQ_MANAGEMENT_ADDR 127.0.0.1:15672 RabbitMQ live tests that declare queues or close connections
SQS_ENDPOINT http://127.0.0.1:9324 SQS live tests
PUBSUB_EMULATOR_HOST 127.0.0.1:8085 Pub/Sub live tests
SERVICEBUS_CONNECTION_STRING The emulator connection string on 127.0.0.1:5673 Service Bus live tests
SERVICEBUS_ADMIN_ADDR 127.0.0.1:5300 Service Bus live tests that create or delete queues

Broker configuration

Kafka auto-creates topics with three partitions, so per-partition ordering and parallelism are visible with a single broker. Replication factors are 1 and the transaction state log accepts a single replica, which allows transactional producers against the one broker. Applications on the host use the listener advertised as localhost:9092; containers use kafka:29092.

Pulsar runs in standalone mode without the functions worker or stream storage. standalone.conf settings can be added to the pulsar service environment, which apply-config-from-env.py applies before the broker starts.

The one-shot pulsar-init service runs after the broker and Pulsar Manager start. It creates the beavers-examples subscription on persistent://public/default/orders at the earliest position, because a new Pulsar subscription otherwise starts at the latest message, and it creates the Pulsar Manager login. Pulsar Manager is preconfigured with a local environment pointing at the broker. The credentials are for local use only.

RabbitMQ runs with its management plugin and the default guest user, which the broker accepts from the host through the published port. It starts without queues; the live tests declare their own through the management API.

ElasticMQ implements the SQS API in memory and accepts any credentials and region. Point a handle at it with Sqs::new().region("elasticmq").endpoint_url("http://localhost:9324") and static credentials such as .credentials(SqsCredentials::new("local", "local")). It starts without queues; create one with the AWS CLI (aws --endpoint-url http://localhost:9324 sqs create-queue --queue-name orders) or the query API, as the live tests do. It does not reproduce every AWS limit and behavior, so verify production settings against SQS itself.

The Pub/Sub emulator, from the google/cloud-sdk emulators image, keeps topics and subscriptions in memory and needs no credentials. Connect to it with PubSub::new().endpoint("http://localhost:8085").anonymous(). It starts without topics; create them through its REST API, as the live tests do:

curl -X PUT http://localhost:8085/v1/projects/beavers/topics/orders
curl -X PUT http://localhost:8085/v1/projects/beavers/subscriptions/orders \
  -H 'Content-Type: application/json' -d '{"topic":"projects/beavers/topics/orders"}'

It does not reproduce every Pub/Sub limit and behavior, so verify production settings against Pub/Sub itself.

The Service Bus emulator stores its entities in the servicebus-sql SQL Server container and reads its namespace from dev/servicebus/config.json, which defines the beavers-events topic with the beavers-test subscription. Connect to it with ServiceBus::from_connection_string(...) and the connection string listed above; UseDevelopmentEmulator=true selects plain AMQP, which .emulator() sets on a handle built with ServiceBus::new. Its management API on port 5300 creates and deletes queues (PUT /{queue}?api-version=2021-05 with an Atom QueueDescription), as the live tests do, but cannot create topic subscriptions, so add topics and subscriptions to the configuration file and recreate the container. Its AMQP port is published as 5673 because RabbitMQ uses 5672. The emulator image has no shell, so the servicebus-health container polls its health endpoint for docker compose up --wait. Both images are amd64 only and need the Microsoft license accepted, which ACCEPT_EULA does. The emulator serves plain AMQP without TLS, refuses binary application properties, and applies smaller quotas than a real namespace, so verify production settings against Azure.

Examples

Command Example Flow
cargo kafka-produce [count] kafka_orders Publishes orders, 10 by default, to Kafka orders
cargo kafka-process kafka_orders Kafka orders → order-events
cargo pulsar-produce [count] pulsar_orders Publishes orders, 10 by default, to Pulsar orders
cargo pulsar-process pulsar_orders Pulsar orders → order-events with a Key_Shared subscription
cargo kafka-to-pulsar kafka_to_pulsar Kafka orders → Pulsar orders-from-kafka
cargo http-receive kafka_to_http HTTP server on 127.0.0.1:8090 that prints the orders it receives
cargo kafka-to-http kafka_to_http Kafka orders → HTTP POST to HTTP_SINK_URL

Each alias expands to cargo run --example <name> --features <features> -- <command>, and arguments after the alias are appended, as in cargo kafka-produce 100.

The produce commands are ordinary beavers applications that publish an IterSource of JSON orders keyed by customer through a broker sink, and end when the source is exhausted. The process commands and kafka-to-pulsar run a subscription until Ctrl-C. process registers a value handler, so each output inherits the input's key and headers or properties.

kafka_to_http sends each order to HTTP_SINK_URL, by default the http-receive server, with an idempotency-key header built from the record's topic, partition, and offset. Stop http-receive while kafka-to-http runs to see the sink retry, stop after its publish retries, and resume from the uncommitted offset on the next run.

kafka_to_pulsar maps metadata explicitly with MapMetadata: the Kafka key becomes the Pulsar key and the tenant and request-id headers with UTF-8 values become Pulsar properties. It skips Kafka tombstones with Tombstones::skip().

The Kafka examples read from the earliest offset when their consumer group has no committed offset, and the Pulsar example uses the subscription created by pulsar-init, so orders produced before a first process run are consumed. Run a process command again to see that committed or acknowledged deliveries are not processed twice.

GitHub Actions runs each example flow in its own Example job, which starts the services the flow needs and runs .github/scripts/example.sh. The script runs the commands above and checks that 10 outputs arrive on the output topic or at the HTTP receiver. Run the script locally with the same scenario name, such as .github/scripts/example.sh kafka-to-http, after starting the brokers.

Inspecting broker state

Kafka UI shows topics, partitions, messages, and consumer group offsets. Pulsar Manager shows tenants, namespaces, topics, and subscription backlogs. The same information is available from the command line:

docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:29092 --topic order-events --from-beginning \
  --property print.key=true
docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:29092 --describe --group beavers-examples
docker compose exec pulsar bin/pulsar-admin topics stats \
  persistent://public/default/order-events

Live tests

The Kafka, Pulsar, RabbitMQ, SQS, Pub/Sub, and Service Bus adapter tests in tests/kafka.rs, tests/pulsar.rs, tests/rabbitmq.rs, tests/sqs.rs, tests/pubsub.rs, and tests/servicebus.rs and the end-to-end pipeline tests in tests/pipelines.rs are ignored by default. With the brokers running:

cargo test-live             # cargo test --features kafka,pulsar,rabbitmq,sqs,pubsub,servicebus -- --ignored
cargo test-live pipeline    # only tests whose name contains "pipeline"

The pipeline tests run the same flows as the examples as complete applications: a Kafka pipeline that inherits keys and headers and commits every consumed offset, a Pulsar pipeline that inherits keys and properties and leaves no subscription backlog, and a Kafka-to-Pulsar pipeline with an explicit metadata mapping. They create Pulsar subscriptions through the admin API before producing input.

Every live test uses unique topic, group, subscription, and queue names, so the tests can run repeatedly without resetting the brokers. GitHub Actions runs them in the Live broker tests job, which starts the kafka, pulsar, rabbitmq, elasticmq, and pubsub services and the Service Bus emulator from compose.yaml and runs cargo test --all-features -- --ignored.

Troubleshooting

Markdown · Source on GitHub