{ "slug": "Real-Time-Data-Ingestion--Connecting-Distributed-Kafka-to-ClickHouse-in-Docker-1e29e0b0f665", "title": "Real-Time Data Ingestion: Connecting Distributed Kafka to ClickHouse in Docker", "subtitle": "Building a high-throughput analytics pipeline requires a database that can ingest millions of rows per second and a message broker that can…", "excerpt": "Building a high-throughput analytics pipeline requires a database that can ingest millions of rows per second and a message broker that can…", "date": "2026-06-01", "tags": [ "Kafka", "ClickHouse", "Docker", "Performance" ], "readingTime": "3 min", "url": "https://medium.com/@mobinshaterian/real-time-data-ingestion-connecting-distributed-kafka-to-clickhouse-in-docker-1e29e0b0f665", "hero": "https://cdn-images-1.medium.com/max/800/1*VjPc3qTw5yfppjX1o0wsrg.png", "content": [ { "type": "heading", "level": 2, "text": "Real-Time Data Ingestion: Connecting Distributed Kafka to ClickHouse in Docker" }, { "type": "paragraph", "html": "Building a high-throughput analytics pipeline requires a database that can ingest millions of rows per second and a message broker that can handle massive streams. ClickHouse and Kafka are the industry standard for this pair." }, { "type": "paragraph", "html": "When running ClickHouse in a Docker environment while your Kafka cluster lives on a separate service (multi-node), there are specific architectural patterns and networking configurations you must follow." }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*VjPc3qTw5yfppjX1o0wsrg.png", "alt": "Real-Time Data Ingestion: Connecting Distributed Kafka to ClickHouse in Docker", "caption": "", "width": 2752, "height": 1536 }, { "type": "quote", "html": "🚨 HIRING: Tech Talent
💰 $50–$120/hr | 🔥 Multiple Roles" }, { "type": "quote", "html": "Frontend • Backend • Full Stack • Mobile • AI/ML • DevOps
👉 Apply Here" }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*pkQFf0NUnzoFbVEVQCF5WQ.png", "alt": "Real-Time Data Ingestion: Connecting Distributed Kafka to ClickHouse in Docker", "caption": "", "width": 1380, "height": 226 }, { "type": "heading", "level": 2, "text": "1. The Three-Tier Architecture" }, { "type": "paragraph", "html": "In ClickHouse, the `Kafka` engine does not store data on disk. It acts as a stateful consumer. To move data from Kafka to a permanent table, we use a three-part “Pipe” pattern:" }, { "type": "paragraph", "html": "1. The Source (Kafka Engine): A virtual table that pulls data from the broker." }, { "type": "paragraph", "html": "2. The Storage (MergeTree Engine): The physical table where data is indexed and saved." }, { "type": "paragraph", "html": "3. The Pipe (Materialized View): The background trigger that connects the two." }, { "type": "heading", "level": 2, "text": "2. Infrastructure Setup (Docker Compose)" }, { "type": "paragraph", "html": "Even though Kafka is hosted elsewhere, your ClickHouse instance needs a stable Docker environment. Here is the configuration to run ClickHouse with optimized limits for high-volume ingestion." }, { "type": "code", "lang": "yaml", "code": "services:\n clickhouse:\n image: clickhouse/clickhouse-server:24.8\n container_name: clickhouse\n ports:\n - \"9000:9000\"\n # Native protocol\n - \"8123:8123\"\n # HTTP protocol\n environment:\n - CLICKHOUSE_DB=default\n - CLICKHOUSE_USER=default\n - CLICKHOUSE_PASSWORD=pass\n - CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1\n volumes:\n - ./ch_data:/var/lib/clickhouse\n networks:\n - analytics_net\n ulimits:\n nofile:\n soft: 262144\n hard: 262144networks:\n analytics_net:\n driver: bridge" }, { "type": "heading", "level": 2, "text": "3. Implementing the Pipeline (SQL)" }, { "type": "paragraph", "html": "Step 1: Create the Storage Table" }, { "type": "paragraph", "html": "This is your final destination. We use the MergeTree engine, which is optimized for time-series and analytical queries." }, { "type": "code", "lang": "sql", "code": "CREATE TABLE events_destination (\n event_id UInt64,\n event_name String,\n event_time DateTime,\n payload String,\n ingested_at DateTime DEFAULT now()\n)\nENGINE = MergeTree()\nORDER BY (event_time, event_id);" }, { "type": "paragraph", "html": "Step 2: Create the Kafka Engine" }, { "type": "paragraph", "html": "This table connects to your external brokers. Note the kafka_broker_list containing multiple IPs for high availability." }, { "type": "code", "lang": "sql", "code": "CREATE TABLE kafka_queue_stream (\n event_id UInt64,\n event_name String,\n event_time DateTime,\n payload String\n)\nENGINE = Kafka\nSETTINGS\n kafka_broker_list = '1.1.1.1:9092,1.1.1.2:9092',\n kafka_topic_list = 'your_topic_name',\n kafka_group_name = 'clickhouse_consumer_v1',\n kafka_format = 'JSONEachRow',\n kafka_skip_broken_messages = 1;" }, { "type": "paragraph", "html": "Step 3: Create the Materialized View" }, { "type": "paragraph", "html": "The Materialized View acts as the “glue.” As soon as this is created, ClickHouse begins consuming messages from Kafka." }, { "type": "code", "lang": "sql", "code": "CREATE MATERIALIZED VIEW kafka_ingest_mv TO events_destination ASSELECT event_id,\nevent_name, event_time, payloadFROM kafka_queue_stream;" }, { "type": "heading", "level": 2, "text": "4. Connectivity & Performance Tips" }, { "type": "heading", "level": 3, "text": "External Networking" }, { "type": "paragraph", "html": "Since Kafka is on a separate service (IPs 10.x.x.x), ensure your Docker host can route to those IPs. You can verify the connection from inside the ClickHouse container:" }, { "type": "code", "lang": "bash", "code": "docker exec -it clickhouse\ncurl -v 1.1.1.1:9092" }, { "type": "heading", "level": 3, "text": "Handling Broken Messages" }, { "type": "paragraph", "html": "In a production JSON stream, malformed data is inevitable. The setting kafka_skip_broken_messages = 1 prevents the entire pipeline from stopping if a producer sends invalid JSON." }, { "type": "heading", "level": 3, "text": "Performance Tuning" }, { "type": "paragraph", "html": "For high-volume streams, consider these settings in your Kafka Table:" }, { "type": "list", "ordered": false, "items": [ "kafka_num_consumers: Increase this to match the number of partitions in your Kafka topic.", "kafka_max_block_size: Increase this (e.g., to 1,000,000) to batch more messages before writing to disk, reducing IO overhead." ] }, { "type": "heading", "level": 2, "text": "Conclusion" }, { "type": "paragraph", "html": "By decoupling the consumer (Kafka Engine) from the storage (MergeTree), you create a resilient architecture. If your database needs to go down for maintenance, Kafka will simply buffer the data until the Materialized View starts pulling again, ensuring zero data loss." }, { "type": "heading", "level": 2, "text": "Thank you for being a part of the community" }, { "type": "paragraph", "html": "Before you go:" }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*d9QTaaaxboQP_gKSLedW_w.png", "alt": "Real-Time Data Ingestion: Connecting Distributed Kafka to ClickHouse in Docker", "caption": "", "width": 1024, "height": 1024 }, { "type": "paragraph", "html": "👉 Be sure to clap and follow the writer ️👏️️" }, { "type": "paragraph", "html": "👉 Follow us: Linkedin| Medium" }, { "type": "paragraph", "html": "👉 CodeToDeploy Tech Community is live on Discord — Join now!" }, { "type": "paragraph", "html": "Disclosure: This post includes affiliate and partnership links." } ] }