{
  "slug": "Building-a-Complete-Kafka---ClickHouse-Streaming-Stack-with-Docker-Compose-98cbfc4bbf0c",
  "title": "Building a Complete Kafka + ClickHouse Streaming Stack with Docker Compose",
  "subtitle": "The article demonstrates building a complete, reproducible real-time data streaming and analytics stack using Kafka and ClickHouse.",
  "excerpt": "The article demonstrates building a complete, reproducible real-time data streaming and analytics stack using Kafka and ClickHouse.",
  "date": "2025-12-07",
  "tags": [
    "Kafka",
    "ClickHouse",
    "Docker"
  ],
  "readingTime": "4 min",
  "url": "https://medium.com/@mobinshaterian/building-a-complete-kafka-clickhouse-streaming-stack-with-docker-compose-98cbfc4bbf0c",
  "hero": "https://cdn-images-1.medium.com/max/800/1*e-ZI0ccrs-sMoO1yqgI3qA.png",
  "content": [
    {
      "type": "heading",
      "level": 2,
      "text": "Building a Complete Kafka + ClickHouse Streaming Stack with Docker Compose"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Abstarct"
    },
    {
      "type": "paragraph",
      "html": "This article presents a comprehensive, reproducible, and fully functional <strong>real-time data streaming and analytics development stack</strong> combining <strong>Apache Kafka</strong> for event streaming and <strong>ClickHouse</strong> for high-performance analytical storage, orchestrated entirely using <strong>Docker Compose</strong>."
    },
    {
      "type": "paragraph",
      "html": "The architecture integrates four core services: <strong>Kafka</strong> (with dual listeners for container-internal and host-external access), <strong>Zookeeper</strong>, <strong>ClickHouse</strong>, and the management console <strong>Kafka-UI</strong>. The key feature demonstrated is the use of the <strong>ClickHouse Kafka Engine</strong>, which enables ClickHouse to act as a <strong>Kafka Producer</strong>, allowing users to stream data from the analytical database directly into a Kafka topic via simple SQL <code>INSERT</code> statements."
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*e-ZI0ccrs-sMoO1yqgI3qA.png",
      "alt": "Building a Complete Kafka + ClickHouse Streaming Stack with Docker Compose",
      "caption": "",
      "width": 1024,
      "height": 1024
    },
    {
      "type": "paragraph",
      "html": "The setup is optimized for development and provides a <strong>zero-code, SQL-driven</strong> method for building lightweight ETL/ELT pipelines. All services are configured with a single <code>docker-compose.yml</code> file, facilitating easy deployment and reproducibility. The article details the configuration steps, including the dual-listener Kafka setup and curl-based querying/ingestion methods, concluding with a discussion of the architecture's benefits (simplicity, high throughput for batch exports, and automation potential) and limitations (batch-oriented nature and lack of built-in real-time guarantees)."
    },
    {
      "type": "paragraph",
      "html": "In this article, we’ll build a fully functional development stack that includes:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Kafka</strong> (with dual listeners for host + container access)",
        "<strong>Zookeeper</strong>",
        "<strong>ClickHouse</strong>",
        "<strong>Kafka‑UI</strong> (a lightweight Kafka management console)",
        "<strong>ClickHouse → Kafka streaming</strong> using the Kafka Engine",
        "<strong>curl‑based querying and ingestion</strong>"
      ]
    },
    {
      "type": "paragraph",
      "html": "Everything runs inside Docker, making it reproducible and easy to extend."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1. Architecture Overview"
    },
    {
      "type": "paragraph",
      "html": "The stack consists of four core services:"
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Kafka + Zookeeper"
    },
    {
      "type": "paragraph",
      "html": "Kafka requires Zookeeper for broker coordination. We configure Kafka with two listeners:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>INSIDE</strong>: for communication between Docker containers",
        "<strong>OUTSIDE</strong>: for communication from your host machine"
      ]
    },
    {
      "type": "paragraph",
      "html": "This ensures tools like Kafka‑UI and ClickHouse can talk to Kafka internally, while CLI tools on your host can still connect."
    },
    {
      "type": "heading",
      "level": 3,
      "text": "ClickHouse"
    },
    {
      "type": "paragraph",
      "html": "ClickHouse provides fast analytical storage and supports Kafka integration through the <strong>Kafka Engine</strong>, which allows ClickHouse to act as a Kafka producer or consumer."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Kafka‑UI"
    },
    {
      "type": "paragraph",
      "html": "Kafka‑UI gives you a clean interface to inspect topics, messages, consumer groups, and cluster metadata."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2. Final Docker Compose File"
    },
    {
      "type": "paragraph",
      "html": "Below is the complete <code>docker-compose.yml</code> used to run the entire stack:"
    },
    {
      "type": "code",
      "lang": "yaml",
      "code": "services:\n  clickhouse:\n    image: clickhouse/clickhouse-server:24.8\n    container_name: clickhouse\n    ports:\n      - \"9000:9000\"\n      - \"8123:8123\"\n    environment:\n      - CLICKHOUSE_DB=default\n      - CLICKHOUSE_USER=default\n      - CLICKHOUSE_PASSWORD=pass\n      - CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1\n    volumes:\n      - ./data:/var/lib/clickhouse\n      - ./logs:/var/log/clickhouse-server\n    ulimits:\n      nofile:\n        soft: 262144\n        hard: 262144\n    restart: unless-stopped\n  zookeeper:\n    image: confluentinc/cp-zookeeper:7.5.0\n    container_name: zookeeper\n    environment:\n      ZOOKEEPER_CLIENT_PORT: 2181\n      ZOOKEEPER_TICK_TIME: 2000\n    ports:\n      - \"2181:2181\"\n    restart: unless-stopped\n  kafka:\n    image: confluentinc/cp-kafka:7.5.0\n    container_name: kafka\n    depends_on:\n      - zookeeper\n    ports:\n      - \"9092:9092\"\n    environment:\n      KAFKA_BROKER_ID: 1\n      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n      # Define both listeners\n      KAFKA_LISTENERS: INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092\n      # Advertise correct addresses\n      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:29092,OUTSIDE://localhost:9092\n      # map listener names to protocol\n      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT\n      # default listener\n      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE\n      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n    restart: unless-stopped\n  kafka-ui:\n    container_name: kafka-ui\n    image: provectuslabs/kafka-ui:latest\n    ports:\n      - \"8080:8080\"\n    environment:\n      DYNAMIC_CONFIG_ENABLED: \"true\"\n    volumes:\n      - ./config/config.yml:/etc/kafkaui/dynamic_config.yaml\n    restart: unless-stopped"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3. Kafka‑UI Configuration"
    },
    {
      "type": "paragraph",
      "html": "Create <code>config/config.yml</code>:"
    },
    {
      "type": "code",
      "lang": "text",
      "code": "kafka:  clusters:    - name: local      bootstrapServers: kafka:29092"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "4. Starting the Stack"
    },
    {
      "type": "paragraph",
      "html": "Run:"
    },
    {
      "type": "code",
      "lang": "bash",
      "code": "docker compose up -d"
    },
    {
      "type": "paragraph",
      "html": "Kafka‑UI becomes available at:"
    },
    {
      "type": "code",
      "lang": "text",
      "code": "http://localhost:8080"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "5. Interacting with ClickHouse via curl"
    },
    {
      "type": "paragraph",
      "html": "ClickHouse exposes an HTTP interface on port <strong>8123</strong>, making it easy to run SQL queries."
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Create a table:"
    },
    {
      "type": "code",
      "lang": "bash",
      "code": "curl -u default:pass \\\n  -X POST \"http://localhost:8123\" \\\n  --data-binary\n  \"CREATE TABLE my_table (    id UInt64,    message String  ) ENGINE = MergeTree ORDER BY id\""
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Insert data:"
    },
    {
      "type": "code",
      "lang": "bash",
      "code": "curl -u default:pass \\\n  -X POST \"http://localhost:8123\" \\\n  --data-binary \"INSERT INTO my_table VALUES (1, 'hello'), (2, 'world')\""
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Query data:"
    },
    {
      "type": "code",
      "lang": "bash",
      "code": "curl -u default:pass \\\n  \"http://localhost:8123/?query=SELECT+*+FROM+my_table+FORMAT+Pretty\""
    },
    {
      "type": "heading",
      "level": 2,
      "text": "6. Streaming Data from ClickHouse → Kafka"
    },
    {
      "type": "paragraph",
      "html": "ClickHouse can act as a Kafka <strong>producer</strong> using the Kafka Engine."
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Step 1 — Create a Kafka sink table"
    },
    {
      "type": "code",
      "lang": "bash",
      "code": "curl -u default:pass \\\n  -X POST \"http://localhost:8123\" \\\n  --data-binary\n  \"CREATE TABLE kafka_output (    id UInt64,    message String)ENGINE = KafkaSETTINGS    kafka_broker_list = 'kafka:29092',    kafka_topic_list = 'clickhouse_out',    kafka_group_name = 'ch_producer',    kafka_format = 'JSONEachRow'\""
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Step 2 — Insert data into Kafka via ClickHouse"
    },
    {
      "type": "code",
      "lang": "bash",
      "code": "curl -u default:pass \\\n  -X POST \"http://localhost:8123\" \\\n  --data-binary \"INSERT INTO kafka_output VALUES (1, 'hello from clickhouse')\""
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Step 3 — Verify in Kafka‑UI"
    },
    {
      "type": "paragraph",
      "html": "Open topic:"
    },
    {
      "type": "code",
      "lang": "text",
      "code": "clickhouse_out"
    },
    {
      "type": "paragraph",
      "html": "You’ll see messages like:"
    },
    {
      "type": "code",
      "lang": "json",
      "code": "{\n  \"id\": 1,\n  \"message\": \"hello from clickhouse\"\n}"
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Step 4 — Verify via Kafka CLI"
    },
    {
      "type": "code",
      "lang": "bash",
      "code": "docker exec -it kafka \\\n  kafka-console-consumer \\\n  --topic clickhouse_out \\\n  --bootstrap-server kafka:29092 \\\n  --from-beginning"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "7. Why This Architecture Works"
    },
    {
      "type": "paragraph",
      "html": "This setup gives you:"
    },
    {
      "type": "heading",
      "level": 3,
      "text": "A clean separation of concerns"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Kafka handles streaming.",
        "ClickHouse handles analytics.",
        "Kafka‑UI handles observability."
      ]
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Dual listener Kafka"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Internal traffic stays inside Docker.",
        "External tools can still connect."
      ]
    },
    {
      "type": "heading",
      "level": 3,
      "text": "ClickHouse Kafka Engine"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Acts as a producer or consumer.",
        "Supports JSON, Avro, Protobuf, CSV, TSV, and more."
      ]
    },
    {
      "type": "heading",
      "level": 3,
      "text": "curl-based SQL"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Perfect for automation, scripts, and CI pipelines."
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Pros"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Simple architecture (no Kafka Connect, no custom producer service)",
        "Zero code required (pure SQL)",
        "High throughput for batch exports",
        "Easy to automate with cron or scheduled jobs",
        "Easy to debug (Kafka‑UI + ClickHouse logs)",
        "Works perfectly inside Docker",
        "Supports SQL transformations before sending to Kafka",
        "Great for ETL/ELT pipelines",
        "Reproducible local development environment",
        "No external dependencies beyond Kafka + ClickHouse"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Cons"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Not real-time (only produces when you run INSERT)",
        "No exactly-once delivery guarantees",
        "No retry or error handling built in",
        "No partitioning control (no custom keys)",
        "No schema evolution or Schema Registry support",
        "SELECT from Kafka Engine is blocked by default",
        "Not suitable for high-frequency event emission",
        "Harder to monitor compared to Kafka Connect",
        "Batch-oriented, not event-driven"
      ]
    }
  ]
}