{
  "slug": "Solving-the--NaN--Deadlock--Streaming-Kafka-to-ClickHouse-with-Data-Sanitization-530ff573762f",
  "title": "Solving the “NaN” Deadlock: Streaming Kafka to ClickHouse with Data Sanitization",
  "subtitle": "Integrating high-velocity streaming data into ClickHouse often feels like a “set it and forget it” task — until it isn’t. Recently, we…",
  "excerpt": "Integrating high-velocity streaming data into ClickHouse often feels like a “set it and forget it” task — until it isn’t. Recently, we…",
  "date": "2026-05-19",
  "tags": [
    "Kafka",
    "ClickHouse"
  ],
  "readingTime": "2 min",
  "url": "https://medium.com/@mobinshaterian/solving-the-nan-deadlock-streaming-kafka-to-clickhouse-with-data-sanitization-530ff573762f",
  "hero": "https://cdn-images-1.medium.com/max/800/1*TKElNPazRoBASd_W3gia9g.png",
  "content": [
    {
      "type": "paragraph",
      "html": "Integrating high-velocity streaming data into ClickHouse often feels like a “set it and forget it” task — until it isn’t. Recently, we faced a production roadblock while migrating a telemetry pipeline from a raw string format to JSONEachRow. The system was crashing with a cryptic CANNOT_PARSE_INPUT_ASSERTION_FAILED error."
    },
    {
      "type": "paragraph",
      "html": "Here is the breakdown of how we optimized our Kafka engine, resolved SASL/SSL authorization issues, and implemented a robust “on-the-fly” data sanitization layer."
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*TKElNPazRoBASd_W3gia9g.png",
      "alt": "Solving the “NaN” Deadlock: Streaming Kafka to ClickHouse with Data Sanitization",
      "caption": "",
      "width": 2752,
      "height": 1536
    },
    {
      "type": "paragraph",
      "html": "<strong>1. The Challenge: Toxic Data in the Stream</strong>"
    },
    {
      "type": "paragraph",
      "html": "Our pipeline consumes performance metrics (PM) from Kafka. The source data occasionally contains “NaN” (Not a Number) values for empty metrics."
    },
    {
      "type": "paragraph",
      "html": "While ClickHouse is incredibly fast, it is also strict. When a downstream Materialized View tried to CAST a JSON map containing “NaN” into a Float64, the entire ingestion thread would crash. Because Materialized Views in ClickHouse operate as a “push” mechanism, a failure in a secondary view can stop the primary ingestion dead in its tracks."
    },
    {
      "type": "paragraph",
      "html": "<strong>2. The Solution: A Three-Tiered Architecture</strong>"
    },
    {
      "type": "paragraph",
      "html": "We moved away from expensive JSONExtract functions and adopted a more performant, schema-aware approach."
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Phase I: Optimized Kafka Engine"
    },
    {
      "type": "paragraph",
      "html": "By switching to JSONEachRow, we allow ClickHouse to use its native C++ parser. This reduces CPU overhead significantly compared to parsing raw strings manually."
    },
    {
      "type": "code",
      "lang": "sql",
      "code": "CREATE MATERIALIZED VIEW mv_metrics_consumer TO dist_metrics_final(    `meta_begin_time` DateTime,\n`measurements` String,    `created_at` DateTime    -- ... other fields)AS\nSELECT    -- Best effort\nparsing handles various ISO date formats automatically    parseDateTimeBestEffort(meta_begin_time)\nAS meta_begin_time,        -- CRITICAL FIX: Sanitize \"NaN\"\nvalues before they hit downstream CASTs\nreplace(measurements, 'NaN', '0') AS measurements,        parseDateTimeBestEffort(created_at) AS\ncreated_at,    id,    cell_id,    technologyFROM kafka_metrics_source;"
    },
    {
      "type": "heading",
      "level": 3,
      "text": "Phase II: The “Sanitizing” Materialized View"
    },
    {
      "type": "paragraph",
      "html": "The “Magic” happens here. Instead of just passing data through, we use the replace() function. By intercepting the measurements string and swapping “NaN” for “0”, we ensure that any downstream view performing mathematical operations or type casting receives valid numeric data."
    },
    {
      "type": "code",
      "lang": "sql",
      "code": "CREATE MATERIALIZED VIEW mv_metrics_consumer TO dist_metrics_final(    `meta_begin_time` DateTime,\n`measurements` String,    `created_at` DateTime    -- ... other fields)AS\nSELECT    -- Best effort\nparsing handles various ISO date formats automatically    parseDateTimeBestEffort(meta_begin_time)\nAS meta_begin_time,        -- CRITICAL FIX: Sanitize \"NaN\"\nvalues before they hit downstream CASTs\nreplace(measurements, 'NaN', '0') AS measurements,        parseDateTimeBestEffort(created_at) AS\ncreated_at,    id,    cell_id,    technologyFROM kafka_metrics_source;"
    },
    {
      "type": "paragraph",
      "html": "<strong>3. Key Takeaways &amp; Lessons Learned</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Atomic Failures:</strong> Remember that Materialized Views are part of the insert transaction. If a downstream view fails to parse a row, the Kafka offset isn’t committed (or the message is lost depending on settings). Always sanitize data as early as possible in the chain.",
        "<strong>Performance:</strong> JSONEachRow is the gold standard for Kafka ingestion. It turns a “blob” of data into typed columns before the Materialized View even touches it.",
        "<strong>Authorization:</strong> Group authorization failed is a common Kafka ACL issue. Ensure your ClickHouse user has Read access not just to the <strong>Topic</strong>, but specifically to the <strong>Consumer Group ID</strong>.",
        "<strong>Flexible Schema:</strong> Using parseDateTimeBestEffort() is a lifesaver when dealing with multiple data sources that might send slightly different timestamp formats (e.g., YYYY-MM-DD vs ISO8601)."
      ]
    },
    {
      "type": "paragraph",
      "html": "By moving the sanitization logic into the primary Materialized View, we transformed a brittle pipeline into a resilient one that handles “toxic” telemetry data without human intervention."
    }
  ]
}