# Pulsar adapter

The optional `pulsar` feature provides typed Apache Pulsar source and sink
adapters backed by the `magnetar-driver` 1.7 client.

```toml
beavers = { version = "0.1", features = ["pulsar"] }
```

## Configuration

A `Pulsar` handle holds the service URL, using the `pulsar://` or
`pulsar+ssl://` scheme, and optional static authentication.
`handle.source::<C>(topic, subscription)` builds a source and
`handle.sink::<C>(topic)` a sink; `source_with_codec` and `sink_with_codec`
take a codec value as their first argument instead of using the codec's
default. Optional settings are builder methods on the handle, the source, and
the sink.

```rust
use beavers::Utf8;
use beavers::adapters::pulsar::{Pulsar, PulsarSubscriptionType};

let pulsar = Pulsar::new("pulsar://localhost:6650");
let source = pulsar
    .source::<Utf8>("persistent://public/default/orders", "orders-workers")
    .subscription_type(PulsarSubscriptionType::Failover);
```

All sources and sinks built from one handle share one client. The client
connects on the first `receive` or publication of any of them, which also
validates the settings, and reconnects their producers and consumers after the
broker connection drops. It closes once the handle, every source and sink built
from it, and every delivery that can still acknowledge through it have been
dropped. Closing a source or sink closes only its own consumers or producers.
Each builder method on the handle returns a handle with a client of its own, so
configure the handle before building sources and sinks from it.

The handle, the source, and the sink each have `validate`, which checks the
values without contacting a broker. A partitioned topic is consumed through one
consumer per partition, all in the same subscription; the partition list is
read when the source connects.

`subscription_type` selects a `PulsarSubscriptionType` and defaults to
`Shared`. `buffer_size` is the number of messages each partition's consumer
prefetches, 100 by default. A new subscription starts at the latest message.

`Pulsar::token` supplies a JWT, while `Pulsar::authentication` accepts a
general `PulsarAuthentication` with a Pulsar authentication method name and
credential bytes. Leave both unset for an unauthenticated broker.

Build a sink from the same handle. `prepare` accepts a `PulsarPublish<T>`,
encodes the typed value once, and returns a cloneable `PulsarPrepared` payload.
Set the key, ordering key, event time, and user properties before preparation:

```rust
use beavers::{Sink, Utf8};
use beavers::adapters::pulsar::{Pulsar, PulsarPublish};

let pulsar = Pulsar::new("pulsar://localhost:6650").token("eyJhbGciOi...");
let sink = pulsar.sink::<Utf8>("persistent://public/default/processed-orders");
let mut output = PulsarPublish::new("processed".to_owned());
output.key = Some(b"customer-42".to_vec());
output.properties.insert("kind".into(), "order".into());
let prepared = sink.prepare(output)?;
sink.publish(&prepared).await?;
# Ok::<(), beavers::BoxError>(())
```

`producer_name` names each partition's producer; by default the broker assigns
names.

## Records

The codec applies to the payload. `PulsarMessage::decode` returns a
`PulsarRecord<T>` with top-level `value`, `key`, `properties`, and `event_time`
fields. These application fields map directly to `PulsarPublish<T>`. Its
`metadata` contains read-only delivery facts: topic, message ID, and publish
time. For a partitioned topic, the metadata topic is the partition's topic,
such as `persistent://public/default/orders-partition-2`. The source reads the
key, properties, event time, and null marker of a message inside a producer
batch from that message's own metadata.

The raw form of a `PulsarMessage` is a `PulsarRecord<Vec<u8>>` with the
undecoded payload bytes, or `value: None` for a tombstone. Dead letters carry
it, so the original payload, key, properties, event time, and delivery metadata
survive even when decoding fails.

## Acknowledgements and ordering

`receive` waits for the next message of any partition, starting with the
partition after the one that delivered last so that a busy partition cannot
starve the others. A message leaves its consumer's queue only when a receive
completes, so dropping a pending receive future does not consume a delivery.
Dropping a `PulsarMessage` leaves it unacknowledged. Call `ack` only after the
handler has completed successfully. An individual ACK is sent with the
message's ID and waits for the broker's response; the adapter does not use
cumulative acknowledgements. Broker receive errors are reported as
`ReceiveError::Retry`, while connection, configuration, and malformed message
metadata errors are `ReceiveError::Fatal`. The client reconnects after the
broker connection drops and reattaches its consumers.

A topic unload, as the broker's load balancer performs routinely, or a broker
restart closes the consumers. An acknowledgement in flight at that moment
fails, and one sent before the consumer has reattached times out after the
client's 30-second operation timeout. `ack` therefore retries a failed
acknowledgement under the source's `ack_retry`, five attempts with
backoff from 100 ms to 2 s by default, and an attempt on the reattached
consumer succeeds. Only an exhausted retry fails the acknowledgement and stops
the subscription. After reattaching, the broker redelivers every
unacknowledged message, including messages the consumer had already
prefetched and still returns, so deliveries received around an unload can be
processed twice and out of their original order.

Call `close` during shutdown to close the consumers. A delivery acknowledged
after its source closed fails with an error. The client stays open for the
handle's other sources and sinks.

Each delivery's ordering key follows the order Pulsar guarantees for the
configured subscription type:

| Subscription type | Ordering key |
|---|---|
| `Exclusive`, `Failover` | Topic partition |
| `Key_Shared` | Ordering key, or the message key when no ordering key is set, within the topic partition |
| `Shared` | None |

Under the default `SourceOrder` distribution, deliveries with the same ordering
key are processed one at a time in receive order. `Key_Shared` messages without
a key and all `Shared` deliveries run in parallel up to the subscription's
`concurrency`. `PulsarDistribution` names choices that apply the same way under
every subscription type:

| `PulsarDistribution` | Processed one at a time |
|---|---|
| `Partition` | The deliveries of one topic partition, as `SourceOrder` under `Exclusive` and `Failover` |
| `MessageKey` | The deliveries of one key within a topic partition: the ordering key, or the message key when no ordering key is set; deliveries with neither one at a time per partition |

```rust,ignore
use beavers::adapters::pulsar::PulsarDistribution;

Subscription::new("orders", source, sink, handler)
    .concurrency(16)
    .distribution(PulsarDistribution::MessageKey);
```

Under `Exclusive` or `Failover`, `MessageKey` runs different keys of a busy
partition in parallel while keeping each key in receive order. Under `Shared`,
it orders only the deliveries this consumer receives, since the broker may
deliver one key's messages to several consumers. The Pulsar client does not report consumer failover or key-range
reassignment, so Pulsar deliveries have no revocation token. When the broker
moves a subscription to another consumer, unacknowledged messages are
redelivered there and can be processed twice.

## Publication

The sink connects lazily on the first `publish`, creating one producer per
partition of a partitioned topic. A keyed message is routed like the Java
client's default: the Java `String.hashCode` of its base64 key, modulo the
partition count, so the same key always reaches the same partition. Keyless
messages rotate across partitions. Keys are always sent in Pulsar's base64
representation because they are arbitrary bytes. Each submission clones the
prepared bytes and metadata into a fresh producer message. Retrying a prepared
value does not rerun the codec.

`PulsarSink::submit` returns once the partition's producer has queued the
message, with a completion that resolves on Pulsar's broker receipt. The
runtime frees the job's concurrency slot at submission and acknowledges the
input when the receipt arrives. `publish` queues the message and waits for the
receipt. A producer sends queued messages in order and replays unacknowledged
ones in order after a reconnect, so outputs of one partition keep their
submission order unless the broker rejects a send, as described below.

`max_pending` bounds the messages queued by `submit` on one sink whose receipt
has not arrived; the default is 1000. `submit` waits while that many are
outstanding. A completion dropped before its receipt, as when the input is
abandoned, frees its slot; the message stays queued and may still be written,
but it is not sent again after a rejection. Publish retries apply to queueing
only.

The client replays sends that lose their connection. A send the broker answers
with an error is not replayed by the client, and a broker that is shutting down
or unloading the topic, as during a restart or a routine load-balancer unload,
rejects the sends in flight with a persistence error. The sink therefore sends
a rejected message again under the sink's `send_retry`, ten sends with
backoff from 100 ms to 5 s by default, measured from the previous send. A task
per producer observes the receipts in submission order, so the messages of one
rejected run are sent again in their original order and reach the topic's next
owner. Like the Java client, the sink does not send a message again after a
`NotAllowedError` or `TopicTerminatedError` rejection. A send error that is
not sent again, or a message rejected on every attempt, fails the completion,
which stops the subscription without acknowledging the input.

A message sent again is written after later messages of its partition that the
client replayed or the broker accepted in the meantime, so a rejection can
reorder the outputs of one partition. A message the broker wrote despite
answering with an error is written twice.

`close` stops new submissions, waits for every outstanding receipt, then closes
the sink's producers. The client stays open for the handle's other sources and
sinks.

## Metadata inheritance

Register `PulsarInherit` on a Pulsar-to-Pulsar subscription to forward the
received key, properties, and event time:

```rust,ignore
use beavers::adapters::pulsar::PulsarInherit;

Subscription::new("orders", pulsar_source, pulsar_sink, handler)
    .middleware(PulsarInherit::new())
```

Explicit output fields take precedence. The key and event time are inherited
only when the output leaves them `None`, and a received property is added only
when the output does not already set that name. Dead-letter properties starting
with `beavers-dlq-` are never inherited. `without_key()`,
`without_properties()`, and `without_event_time()` disable the corresponding
field. The source topic, message ID, and publish time are never inherited, and
no ordering key is derived from the input.

A handler that returns a plain value between a Pulsar source and sink gets
`PulsarInherit::new()` automatically. See the
[runtime guide](../runtime/handlers.md#handler-shapes).

Inherited properties include trace-context properties such as `traceparent`.
Register `TraceContext` after `PulsarInherit` to replace them with the
processing span's context; see
[trace-context propagation](../runtime/observability.md#trace-context-propagation).

## Tombstones

A null value is a tombstone. Pulsar topic compaction deletes a key when it sees
a message with that key and an empty payload, and producers can also mark a
value as null explicitly. The source's `empty_payload_is_tombstone`,
enabled by default, follows the compaction rule: an empty payload or a
null-marked message becomes `PulsarRecord { value: None, .. }` and is not passed
to the codec. When disabled, only a null-marked message becomes `None`, and an
empty payload is decoded by the codec. Disable it only for topics that carry
meaningful empty values and are not compacted.

The sink publishes `PulsarPublish { value: None, .. }` as an empty payload with
the null marker set, so readers see a null value with or without
`empty_payload_is_tombstone`. `PulsarPublish::tombstone(key)` builds one, and
`Tombstones::propagate()` supports Pulsar-to-Pulsar forwarding. The sink publishes non-null values as
encoded, even when the encoding is empty; readers that treat empty payloads as
tombstones, including topic compaction, will read such a value as a deletion.
See [tombstones](../runtime/tombstones.md).

## Dead letters

`PulsarPublish::from_dead_letter` forwards a dead letter to a Pulsar topic with
its original message and failure properties, and
`PulsarDeadLetter::from_record` reads them back; see
[forwarding dead letters](../runtime/errors.md#forwarding-dead-letters-to-a-broker-topic).

## Health and tracing

The source reports whether every partition's consumer is connected to
[health checks](../runtime/health.md#connection-state); a consumer
reconnecting after a topic unload or broker restart makes the subscription
unready until it reconnects. The subscription backlog comes from the broker's
`pulsar_subscription_back_log` metric. Each delivery's `message` span carries the configured
topic, partition, subscription, and message ID as
[messaging attributes](../runtime/observability.md#messaging-attributes).

## Delivery guarantees

A source ACK and a sink publication are separate broker operations, so a
process failure between them can produce a duplicate on redelivery. The
adapter provides no Pulsar transactions: `magnetar-driver` 1.7 mishandles
transactional acknowledgements of batched messages and transactions after a
broker restart, so Pulsar pipelines are at-least-once.
