Building a Complete Kafka + ClickHouse Streaming Stack with Docker Compose

Abstarct

This article presents a comprehensive, reproducible, and fully functional real-time data streaming and analytics development stack combining Apache Kafka for event streaming and ClickHouse for high-performance analytical storage, orchestrated entirely using Docker Compose.

The architecture integrates four core services: Kafka (with dual listeners for container-internal and host-external access), Zookeeper, ClickHouse, and the management console Kafka-UI. The key feature demonstrated is the use of the ClickHouse Kafka Engine, which enables ClickHouse to act as a Kafka Producer, allowing users to stream data from the analytical database directly into a Kafka topic via simple SQL INSERT statements.

The setup is optimized for development and provides a zero-code, SQL-driven method for building lightweight ETL/ELT pipelines. All services are configured with a single docker-compose.yml file, facilitating easy deployment and reproducibility. The article details the configuration steps, including the dual-listener Kafka setup and curl-based querying/ingestion methods, concluding with a discussion of the architecture's benefits (simplicity, high throughput for batch exports, and automation potential) and limitations (batch-oriented nature and lack of built-in real-time guarantees).

In this article, we’ll build a fully functional development stack that includes:

Kafka (with dual listeners for host + container access)
Zookeeper
ClickHouse
Kafka‑UI (a lightweight Kafka management console)
ClickHouse → Kafka streaming using the Kafka Engine
curl‑based querying and ingestion

Everything runs inside Docker, making it reproducible and easy to extend.

1. Architecture Overview

The stack consists of four core services:

Kafka + Zookeeper

Kafka requires Zookeeper for broker coordination. We configure Kafka with two listeners:

INSIDE: for communication between Docker containers
OUTSIDE: for communication from your host machine

This ensures tools like Kafka‑UI and ClickHouse can talk to Kafka internally, while CLI tools on your host can still connect.

ClickHouse

ClickHouse provides fast analytical storage and supports Kafka integration through the Kafka Engine, which allows ClickHouse to act as a Kafka producer or consumer.

Kafka‑UI

Kafka‑UI gives you a clean interface to inspect topics, messages, consumer groups, and cluster metadata.

2. Final Docker Compose File

Below is the complete docker-compose.yml used to run the entire stack:

yaml

services:
  clickhouse:
    image: clickhouse/clickhouse-server:24.8
    container_name: clickhouse
    ports:
      - "9000:9000"
      - "8123:8123"
    environment:
      - CLICKHOUSE_DB=default
      - CLICKHOUSE_USER=default
      - CLICKHOUSE_PASSWORD=pass
      - CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
    volumes:
      - ./data:/var/lib/clickhouse
      - ./logs:/var/log/clickhouse-server
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    restart: unless-stopped
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"
    restart: unless-stopped
  kafka:
    image: confluentinc/cp-kafka:7.5.0
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      # Define both listeners
      KAFKA_LISTENERS: INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      # Advertise correct addresses
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:29092,OUTSIDE://localhost:9092
      # map listener names to protocol
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      # default listener
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    restart: unless-stopped
  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    ports:
      - "8080:8080"
    environment:
      DYNAMIC_CONFIG_ENABLED: "true"
    volumes:
      - ./config/config.yml:/etc/kafkaui/dynamic_config.yaml
    restart: unless-stopped

3. Kafka‑UI Configuration

Create config/config.yml:

text

kafka:  clusters:    - name: local      bootstrapServers: kafka:29092

4. Starting the Stack

Run:

bash

docker compose up -d

Kafka‑UI becomes available at:

text

http://localhost:8080

5. Interacting with ClickHouse via curl

ClickHouse exposes an HTTP interface on port 8123, making it easy to run SQL queries.

Create a table:

bash

curl -u default:pass \
  -X POST "http://localhost:8123" \
  --data-binary
  "CREATE TABLE my_table (    id UInt64,    message String  ) ENGINE = MergeTree ORDER BY id"

Insert data:

bash

curl -u default:pass \
  -X POST "http://localhost:8123" \
  --data-binary "INSERT INTO my_table VALUES (1, 'hello'), (2, 'world')"

Query data:

bash

curl -u default:pass \
  "http://localhost:8123/?query=SELECT+*+FROM+my_table+FORMAT+Pretty"

6. Streaming Data from ClickHouse → Kafka

ClickHouse can act as a Kafka producer using the Kafka Engine.

Step 1 — Create a Kafka sink table

bash

curl -u default:pass \
  -X POST "http://localhost:8123" \
  --data-binary
  "CREATE TABLE kafka_output (    id UInt64,    message String)ENGINE = KafkaSETTINGS    kafka_broker_list = 'kafka:29092',    kafka_topic_list = 'clickhouse_out',    kafka_group_name = 'ch_producer',    kafka_format = 'JSONEachRow'"

Step 2 — Insert data into Kafka via ClickHouse

bash

curl -u default:pass \
  -X POST "http://localhost:8123" \
  --data-binary "INSERT INTO kafka_output VALUES (1, 'hello from clickhouse')"

Step 3 — Verify in Kafka‑UI

Open topic:

text

clickhouse_out

You’ll see messages like:

json

{
  "id": 1,
  "message": "hello from clickhouse"
}

Step 4 — Verify via Kafka CLI

bash

docker exec -it kafka \
  kafka-console-consumer \
  --topic clickhouse_out \
  --bootstrap-server kafka:29092 \
  --from-beginning

7. Why This Architecture Works

This setup gives you:

A clean separation of concerns

Kafka handles streaming.
ClickHouse handles analytics.
Kafka‑UI handles observability.

Dual listener Kafka

Internal traffic stays inside Docker.
External tools can still connect.

ClickHouse Kafka Engine

Acts as a producer or consumer.
Supports JSON, Avro, Protobuf, CSV, TSV, and more.

curl-based SQL

Perfect for automation, scripts, and CI pipelines.

Pros

Simple architecture (no Kafka Connect, no custom producer service)
Zero code required (pure SQL)
High throughput for batch exports
Easy to automate with cron or scheduled jobs
Easy to debug (Kafka‑UI + ClickHouse logs)
Works perfectly inside Docker
Supports SQL transformations before sending to Kafka
Great for ETL/ELT pipelines
Reproducible local development environment
No external dependencies beyond Kafka + ClickHouse

Cons

Not real-time (only produces when you run INSERT)
No exactly-once delivery guarantees
No retry or error handling built in
No partitioning control (no custom keys)
No schema evolution or Schema Registry support
SELECT from Kafka Engine is blocked by default
Not suitable for high-frequency event emission
Harder to monitor compared to Kafka Connect
Batch-oriented, not event-driven

Building a Complete Kafka + ClickHouse Streaming Stack with Docker Compose