{
  "slug": "Integrating-Kafka-with-ClickHouse-using-SSL-SASL--A-Technical-Retrospective-be9d75864340",
  "title": "Integrating Kafka with ClickHouse using SSL/SASL: A Technical Retrospective",
  "subtitle": "Integrating ClickHouse with a secured Kafka cluster often presents a challenge because of the strict separation between SQL-level settings…",
  "excerpt": "Integrating ClickHouse with a secured Kafka cluster often presents a challenge because of the strict separation between SQL-level settings…",
  "date": "2026-05-30",
  "tags": [
    "Kafka",
    "ClickHouse"
  ],
  "readingTime": "2 min",
  "url": "https://medium.com/@mobinshaterian/integrating-kafka-with-clickhouse-using-ssl-sasl-a-technical-retrospective-be9d75864340",
  "hero": "https://cdn-images-1.medium.com/max/800/1*VqZuRPmlC5E5MgWfgYN28A.png",
  "content": [
    {
      "type": "paragraph",
      "html": "Integrating ClickHouse with a secured Kafka cluster often presents a challenge because of the strict separation between SQL-level settings and server-level security configurations. This article summarizes the journey from initial configuration to a stable, production-ready solution."
    },
    {
      "type": "paragraph",
      "html": "<strong>The Challenge: SQL vs. Security</strong>"
    },
    {
      "type": "paragraph",
      "html": "In a typical DevOps environment, Kafka is secured with <strong>SASL_SSL</strong> (using mechanisms like SCRAM-SHA-512). While ClickHouse provides a Kafka storage engine, engineers often encounter a “wall” when trying to pass SSL certificates directly through a CREATE TABLE statement."
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*VqZuRPmlC5E5MgWfgYN28A.png",
      "alt": "Integrating Kafka with ClickHouse using SSL/SASL: A Technical Retrospective",
      "caption": "",
      "width": 2752,
      "height": 1536
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Initial Attempt"
    },
    {
      "type": "paragraph",
      "html": "The initial strategy involved passing the CA certificate path directly into the SETTINGS block:"
    },
    {
      "type": "code",
      "lang": "text",
      "code": "-- This results in a DB::Exception: Unknown settingSETTINGS    kafka_security_protocol = 'SASL_SSL',\nkafka_ssl_ca_cert = '/etc/ssl/certs/ca-cert.pem',    ..."
    },
    {
      "type": "paragraph",
      "html": "<strong>The Problem:</strong> ClickHouse (as of v25.8) does not recognize kafka_ssl_ca_cert as a valid SQL setting. Furthermore, for security reasons, ClickHouse restricts the use of local file paths within SQL commands to prevent unauthorized file system probing."
    },
    {
      "type": "paragraph",
      "html": "<strong>Understanding the Engine Parameters</strong>"
    },
    {
      "type": "paragraph",
      "html": "To optimize performance, four key parameters are used to throttle the data flow. Using the defaults (in parentheses) is a good starting point, but they usually require tuning for high-throughput environments:"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "<strong>kafka_num_consumers (1):</strong> Should be increased to match the number of Kafka partitions to enable parallel reading.",
        "<strong>kafka_max_batch_size (65,536):</strong> The target number of rows to collect before performing a single “insert” into ClickHouse.",
        "<strong>kafka_poll_max_batch_size (65,536):</strong> The maximum number of messages fetched in a single request to the broker.",
        "<strong>kafka_poll_timeout_ms (500ms):</strong> How long the consumer waits for data to arrive before timing out."
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>The Error: Code 115 (UNKNOWN_SETTING)</strong>"
    },
    {
      "type": "paragraph",
      "html": "When attempting to use kafka_ssl_ca_location or kafka_ssl_ca_pem directly in the SQL console, ClickHouse returns:"
    },
    {
      "type": "paragraph",
      "html": "Code: 115. DB::Exception: Unknown setting ‘kafka_ssl_ca_location’: for storage Kafka."
    },
    {
      "type": "paragraph",
      "html": "This confirms that even in modern versions, library-level SSL configurations cannot be “inlined” in the table definition."
    },
    {
      "type": "paragraph",
      "html": "<strong>The Final Solution: Server-Side Configuration</strong>"
    },
    {
      "type": "paragraph",
      "html": "The most robust and secure way to resolve this is to move the security configuration to the ClickHouse server files. This allows the database to handle the handshake at the system level."
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Step 1: Deploy the Certificate"
    },
    {
      "type": "paragraph",
      "html": "The CA certificate (.pem) must be placed in a directory accessible to the ClickHouse service user (typically clickhouse)."
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Recommended Path:</strong> /etc/clickhouse-server/certs/kafka-ca.pem"
      ]
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Step 2: Create a Configuration XML"
    },
    {
      "type": "paragraph",
      "html": "A new configuration file should be added to the config.d directory. This tells the underlying Kafka driver exactly how to handle SSL."
    },
    {
      "type": "paragraph",
      "html": "<strong>File:</strong> /etc/clickhouse-server/config.d/kafka_ssl.xml"
    },
    {
      "type": "code",
      "lang": "xml",
      "code": "<clickhouse>\n<kafka>\n<ssl_ca_location>/etc/clickhouse-server/certs/kafka-ca.pem</ssl_ca_location>\n<security_protocol>SASL_SSL</security_protocol>\n<sasl_mechanism>SCRAM-SHA-512</sasl_mechanism>\n</kafka>\n</clickhouse>"
    },
    {
      "type": "paragraph",
      "html": "<strong>Step 3: Simplified Table Creation</strong>"
    },
    {
      "type": "paragraph",
      "html": "Once the server configuration is active, the SQL statement becomes much cleaner and no longer causes “Unknown Setting” errors:"
    },
    {
      "type": "code",
      "lang": "sql",
      "code": "CREATE TABLE kafka_raw (\n    timestamp DateTime,\n    server String,\n    metric String,\n    value Float64\n)\nENGINE = Kafka\nSETTINGS\n    kafka_brokers = 'ip:9093',\n    kafka_topic_list = 'measurements-test-1',\n    kafka_group_name = 'clickhouse-consumer',\n    kafka_format = 'JSONEachRow',\n    kafka_num_consumers = 4; -- Adjusted for performance"
    },
    {
      "type": "paragraph",
      "html": "<strong>Summary</strong>"
    },
    {
      "type": "paragraph",
      "html": "By separating <strong>Infrastructure concerns</strong> (SSL certificates, SASL mechanisms) from <strong>Application concerns</strong> (topics, formats, consumers), we achieve a secure and maintainable architecture. This approach ensures that sensitive paths are kept out of the SQL logs and that the database maintains high performance with “at-least-once” delivery guarantees."
    }
  ]
}