{
  "slug": "Connecting-Kafka-to-ClickHouse-with-SSL--A-Complete-Integration-Guide-e5a0a5957de3",
  "title": "Connecting Kafka to ClickHouse with SSL: A Complete Integration Guide",
  "subtitle": "Setting up a production-ready data pipeline requires balancing high performance with rigorous security. When integrating ClickHouse with a…",
  "excerpt": "Setting up a production-ready data pipeline requires balancing high performance with rigorous security. When integrating ClickHouse with a…",
  "date": "2026-04-23",
  "tags": [
    "Kafka",
    "ClickHouse",
    "Performance",
    "Security"
  ],
  "readingTime": "2 min",
  "url": "https://medium.com/@mobinshaterian/connecting-kafka-to-clickhouse-with-ssl-a-complete-integration-guide-e5a0a5957de3",
  "hero": "https://cdn-images-1.medium.com/max/800/1*SdrefYZIq_r_PPp2MejyRQ.png",
  "content": [
    {
      "type": "paragraph",
      "html": "Setting up a production-ready data pipeline requires balancing high performance with rigorous security. When integrating <strong>ClickHouse</strong> with a <strong>Kafka</strong> cluster secured by <strong>SASL_SSL</strong>, engineers often hit a “wall” trying to pass SSL certificates directly through SQL commands."
    },
    {
      "type": "paragraph",
      "html": "This guide provides a definitive walkthrough for configuring server-side security, building the ingestion pipeline, and tuning the engine for performance."
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*SdrefYZIq_r_PPp2MejyRQ.png",
      "alt": "Connecting Kafka to ClickHouse with SSL: A Complete Integration Guide",
      "caption": "",
      "width": 768,
      "height": 1376
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Architecture Overview"
    },
    {
      "type": "paragraph",
      "html": "The ClickHouse Kafka engine acts as a “buffer.” To move data into permanent storage, you need three distinct objects working in tandem:"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "<strong>Kafka Engine Table:</strong> A virtual table that acts as a live consumer/pointer to the Kafka stream.",
        "<strong>Materialized View:</strong> The “glue” that triggers consumption and handles data transformation.",
        "<strong>MergeTree Table:</strong> The final destination where data is physically stored and indexed for analytics."
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Part 1: Infrastructure & Security Configuration"
    },
    {
      "type": "paragraph",
      "html": "The most common error when setting up this integration is <strong>Code: 115 (UNKNOWN_SETTING)</strong>. This happens because ClickHouse (as of v25.8) does not allow library-level SSL configurations — like <code>kafka_ssl_ca_location</code>—to be \"inlined\" in a <code>CREATE TABLE</code> statement."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Step 1: Deploy the CA Certificate"
    },
    {
      "type": "paragraph",
      "html": "Place your Kafka CA certificate on the ClickHouse server. The <code>clickhouse</code> user must have explicit read access."
    },
    {
      "type": "code",
      "lang": "bash",
      "code": "sudo\nmkdir -p /etc/clickhouse-server/certs\nsudo cp your-ca.pem /etc/clickhouse-server/certs/kafka-ca.pem\nsudo\nchown clickhouse:clickhouse /etc/clickhouse-server/certs/kafka-ca.pem\nsudo\nchmod 440 /etc/clickhouse-server/certs/kafka-ca.pem"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Step 2: Configure Server-Side Security"
    },
    {
      "type": "paragraph",
      "html": "By placing credentials in a configuration file rather than SQL, you keep sensitive data out of query logs and system metadata. Create a new file in the <code>config.d</code> directory:"
    },
    {
      "type": "paragraph",
      "html": "<code>sudo nano /etc/clickhouse-server/config.d/kafka_ssl.xml</code>"
    },
    {
      "type": "code",
      "lang": "xml",
      "code": "<clickhouse>\n<kafka>\n<ssl_ca_location>/etc/clickhouse-server/certs/kafka-ca.pem</ssl_ca_location>\n<security_protocol>sasl_ssl</security_protocol>\n<sasl_mechanism>scram-sha-512</sasl_mechanism>\n<sasl_username>your_kafka_user</sasl_username>\n<sasl_password>your_kafka_password</sasl_password>\n</kafka>\n</clickhouse>"
    },
    {
      "type": "paragraph",
      "html": "<em>Note: Ensure the XML tags match your broker’s required mechanism (e.g., PLAIN, SCRAM-SHA-256).</em>"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Step 3: Verify Connectivity"
    },
    {
      "type": "paragraph",
      "html": "Restart the server to apply changes and use <code>openssl</code> to ensure the ClickHouse host can reach the broker over port 9093:"
    },
    {
      "type": "code",
      "lang": "bash",
      "code": "sudo systemctl restart clickhouse-serveropenssl s_client -connect <broker-ip>:9093 -CAfile\n/etc/clickhouse-server/certs/kafka-ca.pem"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Part 2: Building the Ingestion Pipeline"
    },
    {
      "type": "paragraph",
      "html": "With security handled at the server level, your SQL code remains clean and focused on the data schema."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1. The Destination Table"
    },
    {
      "type": "paragraph",
      "html": "This is your “Source of Truth,” optimized for analytical queries."
    },
    {
      "type": "code",
      "lang": "sql",
      "code": "CREATE TABLE events_destination (\n    event_id UInt64,\n    event_name String,\n    event_time DateTime,\n    payload String,\n    ingested_at DateTime DEFAULT now()\n)\nENGINE = MergeTree()\nORDER BY (event_time, event_id);"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2. The Kafka Engine Table"
    },
    {
      "type": "paragraph",
      "html": "This table defines the connection to the topic. Because of our XML config, we don’t need to specify SSL details here."
    },
    {
      "type": "code",
      "lang": "sql",
      "code": "CREATE TABLE kafka_queue_stream (\n    event_id UInt64,\n    event_name String,\n    event_time DateTime,\n    payload String\n)\nENGINE = Kafka\nSETTINGS\n    kafka_broker_list = 'ip:9093,ip:9093',\n    kafka_topic_list = 'your_topic_name',\n    kafka_group_name = 'clickhouse_consumer_v1',\n    kafka_format = 'JSONEachRow',\n    kafka_num_consumers = 4, -- Matches Kafka partitions for parallel reading\n    kafka_skip_broken_messages = 1;"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3. The Materialized View"
    },
    {
      "type": "paragraph",
      "html": "Once created, this object starts the background consumption process."
    },
    {
      "type": "code",
      "lang": "sql",
      "code": "CREATE MATERIALIZED VIEW kafka_ingest_mv TO events_destination ASSELECT event_id, event_name,\nevent_time, payloadFROM kafka_queue_stream;"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Performance Tuning Parameters"
    },
    {
      "type": "paragraph",
      "html": "To scale your pipeline, adjust these settings within the Kafka Engine <code>SETTINGS</code> block:"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*oyF_BMMv01BIuL_hiGF6uA.png",
      "alt": "Connecting Kafka to ClickHouse with SSL: A Complete Integration Guide",
      "caption": "",
      "width": 718,
      "height": 371
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*kEEDmga0o3N40LdASzgWmA.png",
      "alt": "Connecting Kafka to ClickHouse with SSL: A Complete Integration Guide",
      "caption": "",
      "width": 719,
      "height": 408
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Summary"
    },
    {
      "type": "paragraph",
      "html": "By separating <strong>infrastructure concerns</strong> (SSL/SASL in XML) from <strong>application concerns</strong> (schemas and topics in SQL), you create a robust, secure, and maintainable pipeline. This architecture ensures that sensitive credentials stay out of logs while maintaining the high-performance “at-least-once” delivery guarantees ClickHouse is known for."
    }
  ]
}