Building a Resilient Event-Driven Data Lakehouse: MinIO Bucket Notifications to Apache Kafka

Eliminate object storage polling and prevent race conditions using MinIO’s native Kafka notification engine and the Sentinel Marker pattern.

Special thanks to mohsen taleb.

In distributed architectures, few patterns cause as much latency and architectural debt as active polling. Having downstream consumers periodically query an object store like AWS S3 or MinIO to check whether a batch calculation or machine learning dataset has finished is inefficient, difficult to scale, and fraught with timing issues.

Conversely, passing entire multi-gigabyte analytical datasets directly through a message broker like Apache Kafka can saturate network interfaces, spike memory usage, and exhaust broker storage.

The optimal pattern is Claim-Check / Event-Driven Metadata Notification: the analytical engine writes the bulk data directly into object storage, while the storage layer itself notifies Apache Kafka that the dataset is ready. Downstream consumers simply receive a lightweight event containing the bucket and object path, then fetch only the files they need via standard S3 API primitives.

However, there is a catch that catches many teams off guard: the partial read problem.

Here is a guide to designing and implementing an enterprise-grade MinIO-to-Kafka event pipeline that solves this problem using the sentinel marker pattern, resilient local queueing, and secure SCRAM authentication.

The Partial Read Challenge and the Sentinel Marker Pattern

Distributed compute engines — such as Apache Spark, Flink, Trino, or containerized data workers — write dataset partitions concurrently across many parallel tasks. A single pipeline run may output dozens of Parquet files alongside a metadata manifest.

If you configure MinIO to publish an event on every generic object creation (s3:ObjectCreated:*), your message broker will be flooded with dozens of notifications while the dataset is still being written. Downstream consumers will inevitably attempt to read partitions before the entire dataset has finished persisting, resulting in corrupted states, missing records, or failed pipeline runs.

To achieve atomic completion semantics without altering consumer architectures, use the Sentinel Marker Pattern:

  1. Write Data Partitions: The pipeline writes all raw partition chunks into the object prefix (for example, v1/domain=telemetry/generation=20260927/).
  2. Write Manifest: The pipeline writes a summarizing manifest.json containing table metadata and file checksums.
  3. Emit Sentinel Object: As its final, atomic action, the pipeline writes a zero-byte marker file named _SUCCESS to the partition root.
  4. Trigger Event Notification: MinIO monitors only put events matching the prefix v1/ and the exact suffix _SUCCESS.
  5. Consume Safely: Kafka delivers exactly one event per generation. Downstream workers ingest the notification, strip the _SUCCESS suffix to find the dataset root, verify the manifest, and process the complete data payload.

Furthermore, internal retention policies and Data Lifecycle Management (ILM) deletions will never trigger spurious consumer actions, because object removal events (s3:ObjectRemoved:*) are omitted from the configuration.

Architecture and End-to-End Workflow

The workflow separates concerns across the pipeline:

  • Analytical Writer: Produces partitioned files and writes _SUCCESS last.
  • MinIO Notification Subsystem: Intercepts the upload, filters by prefix and suffix, stages the notification in an on-disk buffer, and publishes to Kafka.
  • Apache Kafka Broker: Stores the notification event in a secured, durable topic.
  • Consumer Group: Microservices or ingestion workers read the event, fetch the corresponding manifest, and load the dataset.

Step 1: Kafka Broker Setup and Access Control

To secure this integration in production, we configure transport security using SASL_SSL and authenticate via SCRAM-SHA-512.

Topic Provisioning

Because completed generation events have a low-to-moderate throughput profile, configuring a single partition guarantees strict global FIFO ordering across all downstream tasks:

bash
kafka-topics.sh --bootstrap-server kafka1.example.com:9093 \
  --command-config /etc/kafka/admin-client.properties \
  --create \
  --topic lakehouse-ingestion-events \
  --partitions 1 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config cleanup.policy=delete \
  --config retention.ms=604800000

Setting retention.ms to 7 days (604,800,000 ms) provides an operational safety window to resolve consumer outages without message loss.

Credentials and Access Control Lists (ACLs)

MinIO acts exclusively as a Kafka producer. It requires topic write permissions and cluster metadata access, but should never be granted consumer group privileges.

Create the producer principal using kafka-configs.sh:

bash
kafka-configs.sh --bootstrap-server kafka1.example.com:9093 \
  --entity-type users \
  --entity-name minio-lakehouse-producer \
  --alter \
  --add-config 'SCRAM-SHA-512=[password=CHANGE_TO_A_SECURE_TOKEN]' \
  --command-config /etc/kafka/admin-client.properties

Next, grant least-privilege permissions:

bash
kafka-acls.sh --bootstrap-server kafka1.example.com:9093 \
  --command-config /etc/kafka/admin-client.properties \
  --add \
  --allow-principal User:minio-lakehouse-producer \
  --operation Write \
  --operation Describe \
  --topic lakehouse-ingestion-events

Downstream consumers will use a separate identity (for example, User:downstream-consumer) authorized with Read operations on the topic and their respective Consumer Group.

Step 2: Configuring MinIO with Durable Queuing

MinIO supports dynamic Kafka target configuration via its command-line client (mc). The top-level configuration key is notify_kafka, where an appended suffix identifies the unique endpoint target.

1. Distribute Kafka’s Certificate Authority (CA)

When certificate verification is enabled (tls_skip_verify="off"), MinIO verifies the broker's TLS identity. Place the root CA chain into MinIO’s trusted authority directory across every node in the cluster:

bash
sudo mkdir -p /etc/minio/certs/CAs
sudo cp kafka-ca.crt /etc/minio/certs/CAs/kafka-ca.crt
sudo chown -R minio-user:minio-user /etc/minio/certs/CAs
sudo chmod 644 /etc/minio/certs/CAs/kafka-ca.crt

Note: MinIO loads trusted CA certificates into its memory pool during service initialization. Adding a new CA requires a service restart.

2. Allocate an On-Disk Durable Queue

What happens if your Kafka cluster is down for maintenance or undergoing a partition rebalance?

By configuring queue_dir, MinIO serializes unsent events to a local filesystem spool instead of dropping them or keeping them in volatile RAM. Create this directory across all MinIO hosts:

bash
sudo mkdir -p /var/lib/minio/events/analytics
sudo chown -R minio-user:minio-user /var/lib/minio/events/analytics
sudo chmod 700 /var/lib/minio/events/analytics

3. Register the Kafka Target

Run mc admin config set using the brokers parameter to specify the bootstrap endpoints, and configure authentication and disk queuing:

bash
mc admin config set my-minio notify_kafka:analytics \
  brokers="kafka1.example.com:9093,kafka2.example.com:9093,kafka3.example.com:9093" \
  topic="lakehouse-ingestion-events" \
  sasl="on" \
  sasl_username="minio-lakehouse-producer" \
  sasl_password="CHANGE_TO_A_SECURE_TOKEN" \
  sasl_mechanism="SHA512" \
  tls="on" \
  tls_skip_verify="off" \
  queue_dir="/var/lib/minio/events/analytics" \
  queue_limit="100000" \
  comment="Completed pipeline run notifications"

Restart the MinIO service to activate the Kafka worker pool and load the TLS certificates:

bash
mc admin service restart my-minio

Confirm that MinIO has assigned an Amazon Resource Name (ARN) to the endpoint:

bash
mc event target list my-minio

You should see an active ARN:

text
arn:minio:sqs::analytics:kafka

Step 3: Binding the Filtered Notification Rule

Now that the Kafka target is active, bind it to your target bucket (analytics-lakehouse) with the required event filters:

bash
mc event add \
  --event put \
  --prefix "v1/" \
  --suffix "_SUCCESS" \
  my-minio/analytics-lakehouse \
  arn:minio:sqs::analytics:kafka

Verify that the rule is registered:

bash
mc event ls my-minio/analytics-lakehouse

Expected output:

text
arn:minio:sqs::analytics:kafka s3:ObjectCreated:* Filter: prefix="v1/", suffix="_SUCCESS"

What the Downstream Event Looks Like

When MinIO intercepts the upload of v1/domain=telemetry/generation=20260927/_SUCCESS, it constructs an Amazon S3-compliant JSON notification record and dispatches it to Kafka:

json
{
  "EventName": "s3:ObjectCreated:Put",
  "Key": "analytics-lakehouse/v1/domain=telemetry/generation=20260927/_SUCCESS",
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "minio:s3",
      "awsRegion": "",
      "eventTime": "2026-09-27T08:14:02.150Z",
      "eventName": "s3:ObjectCreated:Put",
      "s3": {
        "bucket": {
          "name": "analytics-lakehouse",
          "arn": "arn:aws:s3:::analytics-lakehouse"
        },
        "object": {
          "key": "v1/domain=telemetry/generation=20260927/_SUCCESS",
          "size": 0,
          "eTag": "d41d8cd98f00b204e9800998ecf8427e",
          "sequencer": "182178E8B3728CAC"
        }
      }
    }
  ]
}

Consumer Best Practices: Handling at-Least-Once Delivery

Distributed messaging systems operate under at-least-once delivery guarantees. Network disconnects, consumer crashes, or MinIO queue replays can occasionally deliver duplicate messages.

To keep downstream data pipelines reliable:

  • De-duplicate via the Sequencer: Notice the s3.object.sequencer token inside the payload. MinIO produces this monotonically increasing hexadecimal identifier to record state changes on an object. Store processed sequencer tokens in a quick-lookup store (like Redis or a tracking table). If an event arrives with an already-processed sequencer, acknowledge and skip it.
  • Trim the Sentinel Suffix: Strip the trailing _SUCCESS to infer the generation prefix.
python
object_key = event["Records"][0]["s3"]["object"]["key"]
base_dir = object_key.replace("_SUCCESS", "")
# Results in: "v1/domain=telemetry/generation=20260927/"
  • Read the Manifest First: Query base_dir + "manifest.json". Confirm that all data chunks listed in the manifest exist and match expected checksums before kicking off heavy downstream transformation jobs.

Operational Observability

When running this pattern in production, monitor MinIO’s Prometheus metrics endpoint (/minio/v2/metrics/cluster). Key metrics to track include:

  • minio_notify_target_queue_length: The count of pending messages spooled in queue_dir. Under healthy conditions, this sits near 0. If it climbs for more than a few minutes, investigate Kafka broker connectivity or SASL authorization failures.
  • minio_notify_events_skipped_total: Increments only if queue_limit is saturated and MinIO is forced to discard events to prevent disk exhaustion. This metric should trigger a critical alert.
  • minio_notify_target_failed_events: Increments upon connection failures or TLS handshake rejections.

Summary

Decoupling object storage persistence from message passing creates resilient, scalable data pipelines. By pairing MinIO’s native notify_kafka subsystem with the Sentinel Marker pattern, you gain:

  • Zero Polling Overhead: Compute resources activate only when data is ready.
  • Deterministic Reads: Applications consume only complete datasets.
  • Network Efficiency: Kafka brokers handle lightweight metadata instead of heavy data transfers.
  • Outage Resilience: Persistent on-disk buffers keep notifications safe during broker downtime.

With these configurations in place, your object storage tier becomes an active, event-driven participant in your analytics infrastructure.

Minio Kafka Big Data Data Lake