{
"slug": "Building-a-High-Throughput-KPI-Pipeline-on-ClickHouse--Partitioning--Idempotent-Recalculation--and--3302a12ddf83",
"title": "Building a High-Throughput KPI Pipeline on ClickHouse: Partitioning, Idempotent Recalculation, and…",
"subtitle": "Every hour, 15 million rows of raw network measurement data land in our ClickHouse cluster. From that data we calculate per-cell…",
"excerpt": "Every hour, 15 million rows of raw network measurement data land in our ClickHouse cluster. From that data we calculate per-cell…",
"date": "2026-05-25",
"tags": [
"Star",
"My Experience",
"ClickHouse",
"Performance",
"Data Engineering"
],
"readingTime": "8 min",
"url": "https://medium.com/@mobinshaterian/building-a-high-throughput-kpi-pipeline-on-clickhouse-partitioning-idempotent-recalculation-and-3302a12ddf83",
"hero": "https://cdn-images-1.medium.com/max/800/1*w4lrvaiZs_-0Y5WR_YzYQw.png",
"content": [
{
"type": "heading",
"level": 2,
"text": "Building a High-Throughput KPI Pipeline on ClickHouse: Partitioning, Idempotent Recalculation, and Flat Column Storage"
},
{
"type": "paragraph",
"html": "Every hour, 15 million rows of raw network measurement data land in our ClickHouse cluster. From that data we calculate per-cell, per-technology KPIs — hundreds of them — and store the results for dashboards, alerting, and trend analysis. Sounds straightforward. But there is a catch that makes the design genuinely interesting: some measurement sources go silent for hours, then come back. When they do, we need to recalculate past windows as if the gap never happened, without leaving stale results behind."
},
{
"type": "paragraph",
"html": "This article focuses on the three ClickHouse-specific design decisions that make this work cleanly:"
},
{
"type": "list",
"ordered": true,
"items": [
"Partition by date and use ALTER TABLE … DELETE to replace a time window atomically.",
"Flatten all KPI values into columns on a single wide table instead of normalizing into rows.",
"Build an INSERT … SELECT with CTEs and FULL OUTER JOIN so every KPI ends up in the right row even when sources are sparse."
]
},
{
"type": "paragraph",
"html": "Special thanks to mohsen taleb And Taha Amanzadi"
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*w4lrvaiZs_-0Y5WR_YzYQw.png",
"alt": "Building a High-Throughput KPI Pipeline on ClickHouse: Partitioning, Idempotent Recalculation, and…",
"caption": "",
"width": 2752,
"height": 1536
},
{
"type": "heading",
"level": 2,
"text": "The Problem in Concrete Terms"
},
{
"type": "paragraph",
"html": "Our data comes from telecom network nodes. Each node sends counters every 15 minutes. An Airflow DAG aggregates those counters into hourly KPIs and writes results to ClickHouse."
},
{
"type": "paragraph",
"html": "Here is what goes wrong in practice:"
},
{
"type": "list",
"ordered": false,
"items": [
"A vendor feed goes down at 14:00 and recovers at 17:00. The 14:00–15:00, 15:00–16:00, and 16:00–17:00 windows were computed with incomplete data.",
"When the feed recovers we need to re-run those three windows and replace what we wrote.",
"Rows cannot simply be updated. ClickHouse’s MergeTree family is optimised for inserts and range scans; in-place updates are expensive mutations that run asynchronously."
]
},
{
"type": "paragraph",
"html": "The naive fix — just insert again and deduplicate at query time with argMax or ReplacingMergeTree — works but it forces every downstream query to carry dedup logic, and old rows pile up indefinitely."
},
{
"type": "paragraph",
"html": "We chose a cleaner path: delete the window, then re-insert."
},
{
"type": "heading",
"level": 2,
"text": "Design Decision 1: Partition by Date, Delete by Partition"
},
{
"type": "heading",
"level": 2,
"text": "Table Definition"
},
{
"type": "code",
"lang": "sql",
"code": "CREATE TABLE IF NOT EXISTS kpi_flat_hourly (\n time DateTime('UTC'),\n cell_id String,\n vendor String,\n -- ... more dimension columns ...\n updated_at DateTime('UTC') DEFAULT now('UTC'),\n kpi_id_1001 Nullable(Float64),\n kpi_id_1002 Nullable(Float64)\n -- columns are added dynamically as new KPIs are discovered\n)\nENGINE = MergeTree\nPARTITION BY toDate(time)\nORDER BY (time, cell_id)\nSETTINGS index_granularity = 8192;"
},
{
"type": "paragraph",
"html": "The PARTITION BY toDate(time) clause is the key. Each calendar day becomes its own partition — a separate set of files on disk. ClickHouse can drop or replace an entire partition in milliseconds because it is purely a filesystem rename."
},
{
"type": "heading",
"level": 2,
"text": "Replacing a Time Window"
},
{
"type": "paragraph",
"html": "When Airflow re-runs an hourly window, the pipeline issues:"
},
{
"type": "code",
"lang": "sql",
"code": "ALTER TABLE kpi_flat_hourly\nDELETE\nWHERE meta_begin_time >= toDateTime('2024-11-01 14:00:00', 'UTC')\n AND meta_begin_time < toDateTime('2024-11-01 15:00:00', 'UTC');"
},
{
"type": "paragraph",
"html": "followed immediately by the fresh INSERT … SELECT."
},
{
"type": "paragraph",
"html": "This is a lightweight delete (ALTER TABLE … DELETE), which in recent ClickHouse versions marks rows as deleted in metadata and removes them during the next merge, without rewriting entire parts immediately. For our hourly cadence — where a window is at most a few million rows inside a single daily partition — this is fast enough and keeps the pipeline idempotent."
},
{
"type": "quote",
"html": "Why not DROP PARTITION and re-insert? We run 24 hourly windows per day. Dropping the entire daily partition to replace one hour would destroy the other 23 hours of valid data. ALTER TABLE … DELETE with a time range filter is the right tool here."
},
{
"type": "heading",
"level": 2,
"text": "Performance Characteristics"
},
{
"type": "list",
"ordered": false,
"items": [
"The delete touches only the parts whose time range overlaps the window. Thanks to the ORDER BY (time, cell_id), ClickHouse primary key pruning limits which granules are read.",
"The delete itself is logged as a mutation and executes asynchronously on the storage layer. The subsequent insert does not wait for the mutation to complete — data is visible to new queries before old rows fully vanish, but since the insert writes new parts, SELECT queries see the new values from the fresh parts first (ClickHouse reads the latest parts first during merges).",
"Old KPI calculations from months ago accumulate in historical partitions and are never touched unless a reprocessing job explicitly targets them. Storage grows predictably at one daily partition per table per day."
]
},
{
"type": "heading",
"level": 2,
"text": "Design Decision 2: Flat KPI Columns Instead of a Key-Value Model"
},
{
"type": "paragraph",
"html": "The alternative — a normalised (cell_id, timestamp, kpi_id, value) schema — is tempting because it does not require schema changes when new KPIs are added. We rejected it for two reasons."
},
{
"type": "heading",
"level": 2,
"text": "Query Ergonomics"
},
{
"type": "paragraph",
"html": "A flat schema makes per-cell, multi-KPI queries trivial:"
},
{
"type": "code",
"lang": "sql",
"code": "SELECT\n time,\n cell_id,\n kpi_id_1001,\n kpi_id_1002,\n kpi_id_1001 / nullIf(kpi_id_1002, 0) AS utilisation_ratio\nFROM kpi_flat_hourly\nWHERE time >= '2024-11-01 00:00:00'\n AND time < '2024-11-02 00:00:00'\n AND region = 'north';"
},
{
"type": "paragraph",
"html": "The same query in a key-value layout requires two self-joins or a groupBy with conditional aggregation. At tens of millions of rows that extra join cost is real."
},
{
"type": "heading",
"level": 2,
"text": "Column-Store Compression"
},
{
"type": "paragraph",
"html": "ClickHouse compresses each column independently using LZ4 or ZSTD. A column of Nullable(Float64) values for a single KPI compresses far better than a mixed value Float64 column that interleaves values from hundreds of different KPIs. Benchmarks in our cluster showed roughly 3× better compression on the flat layout for a 30-day window."
},
{
"type": "heading",
"level": 2,
"text": "Schema Evolution Without Downtime"
},
{
"type": "paragraph",
"html": "New KPIs are added by the pipeline itself:"
},
{
"type": "code",
"lang": "python",
"code": "def ensure_kpi_columns(client, table_name: str, kpi_columns: list[str]):\n existing = get_existing_columns(client, table_name)\n for col in kpi_columns: if col not in existing: client.command(\n f\"ALTER TABLE {table_name} \" f\"ADD COLUMN IF NOT EXISTS {col} Nullable(Float64)\"\n )"
},
{
"type": "paragraph",
"html": "ALTER TABLE … ADD COLUMN in ClickHouse is a metadata-only operation — it does not rewrite existing data parts. The new column returns NULL for all historical rows, which is the correct semantics: we did not have that KPI before, and NULL is distinguishable from a legitimate zero."
},
{
"type": "heading",
"level": 2,
"text": "Design Decision 3: Merging Sparse KPI Sources with CTEs and FULL OUTER JOIN"
},
{
"type": "paragraph",
"html": "Not every KPI SQL query selects the same set of dimension columns. Some are cell-level, some are site-level. Some include region; others omit it. When we need to combine multiple KPIs into a single wide row we cannot assume the schemas align."
},
{
"type": "heading",
"level": 2,
"text": "The SQL Builder"
},
{
"type": "paragraph",
"html": "The pipeline groups query files by (vendor, technology, entity, granularity) and builds a single INSERT … SELECT using CTEs:"
},
{
"type": "code",
"lang": "sql",
"code": "WITH kpi_1 AS (\nSELECT time, cell_id, vendor, ..., sum(counter_a) AS kpi_id_1001\nFROM\nflat_lte_vendor_a\nWHERE time>= toDateTime('2024-11-01 14:00:00', 'UTC') AND time <\ntoDateTime('2024-11-01 15:00:00', 'UTC')\nGROUP BY time, cell_id, vendor, ... ), kpi_2 AS (\nSELECT meta_begin_time, cell_id, vendor, ..., avg(counter_b) AS kpi_id_1002\nFROM\nflat_lte_vendor_a\nWHERE ...\nGROUP BY ... )INSERT INTO kpi_flat_lte_vendor_a_cell_hourlySELECT\ncoalesce(kpi_1.time, kpi_2.time) AS meta_begin_time, coalesce(kpi_1.cell_id,\nkpi_2.cell_id) AS cell_id, -- ... coalesce for every dimension column ... now('UTC')\nAS updated_at, kpi_1.kpi_id_1001, kpi_2.kpi_id_1002FROM kpi_1FULL\nOUTER\nJOIN kpi_2 ON\nkpi_1.meta_begin_time = kpi_2.meta_begin_time AND kpi_1.cell_id = kpi_2.cell_id AND\nkpi_1.vendor = kpi_2.vendor;"
},
{
"type": "paragraph",
"html": "FULL OUTER JOIN ensures that a cell which has data for KPI 1001 but not KPI 1002 in a given hour still gets a row, with kpi_id_1002 = NULL. This is exactly what we want: the row represents what we know about that cell in that hour, and absent measurements are explicitly null rather than silently missing."
},
{
"type": "heading",
"level": 2,
"text": "Putting It All Together: The Hourly Run"
},
{
"type": "paragraph",
"html": "Each Airflow DAG run follows this sequence for a given (vendor, technology, entity, granularity) group:"
},
{
"type": "code",
"lang": "text",
"code": "1. Discover SQL files matching the group.2. ensure_target_table — CREATE TABLE IF NOT EXISTS (no-op\nafter first run).3. ensure_kpi_columns — ADD COLUMN IF NOT EXISTS for any new KPIs.4. ALTER TABLE\n… DELETE WHERE meta_begin_time in [start, end).5. WITH … INSERT INTO … SELECT (the merged CTE\nquery)."
},
{
"type": "paragraph",
"html": "Steps 2 and 3 are metadata operations and complete in under a millisecond. Step 4 schedules a mutation. Step 5 writes new parts immediately. From the application’s point of view the window has been replaced atomically — no query will see a half-written state because ClickHouse part visibility is atomic at the part level."
},
{
"type": "heading",
"level": 2,
"text": "Trade-offs and Lessons Learned"
},
{
"type": "paragraph",
"html": "Mutations are asynchronous. ALTER TABLE … DELETE returns before the rows are physically removed. If you query immediately after, you may briefly see both old and new rows from the same window. In practice our downstream dashboards tolerate this — they aggregate over the whole window and the fresh parts dominate — but it is worth knowing if you have strict consistency requirements."
},
{
"type": "paragraph",
"html": "Wide tables and column count. We have tables with several hundred KPI columns. ClickHouse handles wide tables well because each column is a separate set of files, but DESCRIBE TABLE output becomes unwieldy and INSERT queries grow long. We generate them programmatically and never write them by hand."
},
{
"type": "paragraph",
"html": "Nullable(Float64) has a cost. ClickHouse stores a separate 1-bit null map alongside each nullable column. For columns that are almost always non-null this is pure overhead. Once a KPI source is stable we could migrate the column to Float64 DEFAULT 0 or Float64 DEFAULT nan, but for operational simplicity we leave everything nullable."
},
{
"type": "paragraph",
"html": "One CTE per KPI file vs. grouping in SQL. We could push multiple KPI calculations into a single SQL file and avoid the FULL OUTER JOIN complexity entirely. We kept the one-file-per-KPI convention because it makes it trivial for analysts to add or remove individual metrics without touching a monolithic query, and the join overhead in ClickHouse is negligible compared to the I/O cost of the source scans."
},
{
"type": "heading",
"level": 2,
"text": "Summary"
},
{
"type": "paragraph",
"html": "Requirement ClickHouse mechanism Replace a past hour’s results ALTER TABLE … DELETE scoped to time window Avoid stale rows accumulating Daily PARTITION BY toDate(time) keeps deletes fast Store many KPIs without row duplication Flat wide table, one Nullable(Float64) column per KPI Add new KPIs without downtime ALTER TABLE … ADD COLUMN IF NOT EXISTS (metadata-only) Merge sparse per-KPI results into one row FULL OUTER JOIN with dynamic join key detection + coalesce on dims"
},
{
"type": "paragraph",
"html": "The design is not exotic — it leans on features ClickHouse already does well. The insight is knowing which features to reach for: partitioned deletes instead of updates, column-per-metric instead of EAV, and set-based SQL generation instead of row-by-row Python loops. Together they give us a pipeline that is fast, idempotent, and easy to extend as the network and its KPIs keep growing."
}
]
}