{
  "slug": "A-Microservice-That-Teaches-Everything-Not-to-Do-e055f7480424",
  "title": "A Microservice That Teaches Everything Not to Do",
  "subtitle": "Every engineer eventually encounters a system that becomes a living catalog of anti-patterns. Recently, I reviewed a microservice…",
  "excerpt": "Every engineer eventually encounters a system that becomes a living catalog of anti-patterns. Recently, I reviewed a microservice…",
  "date": "2026-05-03",
  "tags": [
    "Microservices",
    "Career"
  ],
  "readingTime": "6 min",
  "url": "https://medium.com/@mobinshaterian/a-microservice-that-teaches-everything-not-to-do-e055f7480424",
  "hero": "https://cdn-images-1.medium.com/max/800/1*f55lHpCxu3TeUajOIQf4kg.png",
  "content": [
    {
      "type": "heading",
      "level": 2,
      "text": "A Microservice That Teaches Everything Not to Do"
    },
    {
      "type": "paragraph",
      "html": "Every engineer eventually encounters a system that becomes a living catalog of anti-patterns. Recently, I reviewed a microservice responsible for parsing large measurement files and storing the extracted results in a database. The problem itself is straightforward: read files, extract measurements, enrich the data, and persist it. Unfortunately, the implementation demonstrates how a relatively simple pipeline can become fragile, inefficient, and operationally dangerous when basic distributed-systems principles are ignored."
    },
    {
      "type": "paragraph",
      "html": "This article walks through several architectural mistakes found in this service and explains what better alternatives would look like."
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*f55lHpCxu3TeUajOIQf4kg.png",
      "alt": "A Microservice That Teaches Everything Not to Do",
      "caption": "",
      "width": 1536,
      "height": 1024
    },
    {
      "type": "heading",
      "level": 2,
      "text": "The Problem the Service Was Supposed to Solve"
    },
    {
      "type": "paragraph",
      "html": "The pipeline’s intended workflow is simple:"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "Large measurement files are uploaded.",
        "A microservice reads the files.",
        "The service parses measurements.",
        "Data is enriched with reference tables.",
        "Results are stored in ClickHouse for analytics."
      ]
    },
    {
      "type": "paragraph",
      "html": "The service runs as multiple pods in Kubernetes to scale horizontally. In theory, this architecture should allow parallel processing of files and high ingestion throughput."
    },
    {
      "type": "paragraph",
      "html": "In practice, the system fights its own design."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1. Random Sleep Instead of Concurrency Control"
    },
    {
      "type": "paragraph",
      "html": "The first red flag appears immediately when the service starts processing files. Each worker begins with a random delay measured in seconds. The intention was to reduce the chance that multiple pods would pick the same file simultaneously."
    },
    {
      "type": "paragraph",
      "html": "This approach reveals a misunderstanding of concurrency in distributed systems. Random delays are not synchronization mechanisms. They only reduce the probability of collisions; they never eliminate them."
    },
    {
      "type": "paragraph",
      "html": "In distributed systems, coordination must be explicit. Common solutions include:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Distributed locks (Redis, etcd, ZooKeeper)",
        "Atomic file claiming via metadata storage",
        "Message queues with consumer groups",
        "Database row locking or job tables"
      ]
    },
    {
      "type": "paragraph",
      "html": "For example, a simple and reliable pattern is to maintain a job table:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Each file is inserted as a job.",
        "Workers atomically claim jobs using a state transition (<code>pending → processing</code>).",
        "A unique constraint or transactional update ensures only one worker processes each file."
      ]
    },
    {
      "type": "paragraph",
      "html": "Without deterministic coordination, race conditions are guaranteed eventually."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2. Using NFS as a Coordination Mechanism"
    },
    {
      "type": "paragraph",
      "html": "Instead of using object storage, the system relies on a shared NFS mount for file management."
    },
    {
      "type": "paragraph",
      "html": "NFS can work for simple shared storage, but it is poorly suited for distributed event pipelines. It provides weak guarantees around concurrent file operations and becomes a bottleneck under heavy parallel workloads."
    },
    {
      "type": "paragraph",
      "html": "Modern systems typically use object storage, such as:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Amazon S3",
        "MinIO",
        "Google Cloud Storage",
        "Azure Blob Storage"
      ]
    },
    {
      "type": "paragraph",
      "html": "Object storage provides durability, versioning, event triggers, and scalability that NFS simply cannot match. It also integrates naturally with event-driven architectures."
    },
    {
      "type": "paragraph",
      "html": "Using NFS for a distributed ingestion pipeline is a common source of race conditions, performance degradation, and operational complexity."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3. File Renaming as a Locking Strategy"
    },
    {
      "type": "paragraph",
      "html": "The service tries to prevent multiple workers from processing the same file by renaming it. When a pod starts processing, it appends a suffix like&nbsp;<code>.zumbolize</code> to the filename. Once processing finishes, the file is deleted."
    },
    {
      "type": "paragraph",
      "html": "This creates several problems:"
    },
    {
      "type": "paragraph",
      "html": "First, the rename operation itself is not a reliable distributed lock. Multiple pods can still read the file list simultaneously and race to rename it."
    },
    {
      "type": "paragraph",
      "html": "Second, deleting the file after processing removes any ability to audit or reprocess data."
    },
    {
      "type": "paragraph",
      "html": "Third, there is no traceability. If processing fails halfway through, there is no reliable record of what happened."
    },
    {
      "type": "paragraph",
      "html": "A proper ingestion pipeline maintains clear state transitions, such as:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<code>uploaded</code>",
        "<code>queued</code>",
        "<code>processing</code>",
        "<code>completed</code>",
        "<code>failed</code>"
      ]
    },
    {
      "type": "paragraph",
      "html": "These states are persisted in a durable store. Files remain immutable artifacts rather than temporary coordination tools."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "4. Data Modeling Contradictions"
    },
    {
      "type": "paragraph",
      "html": "Another problematic area is the database schema."
    },
    {
      "type": "paragraph",
      "html": "The system stores measurements in a massive table that approaches 1 terabyte. However, enrichment is performed by joining with normalized reference tables designed using the third normal form (3NF)."
    },
    {
      "type": "paragraph",
      "html": "Normalization is useful in transactional systems to eliminate redundancy. But analytical databases like ClickHouse are optimized for denormalized, columnar data."
    },
    {
      "type": "paragraph",
      "html": "Mixing OLTP normalization concepts with OLAP storage leads to two major issues:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Expensive joins on very large tables",
        "Repeated enrichment operations during ingestion"
      ]
    },
    {
      "type": "paragraph",
      "html": "In analytical pipelines, denormalization is often intentional. It improves query performance and simplifies downstream processing."
    },
    {
      "type": "paragraph",
      "html": "ClickHouse is especially optimized for wide tables with many columns. Trying to force a highly normalized schema into a columnar analytics database defeats its design advantages."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "5. Storing Structured Data as Strings"
    },
    {
      "type": "paragraph",
      "html": "The most painful design decision appears in how measurement metadata is stored."
    },
    {
      "type": "paragraph",
      "html": "The system builds a dictionary of key-value pairs, serializes it as a string, and then stores it in ClickHouse. Every downstream query must:"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "Convert the string into JSON",
        "Query the fields",
        "Convert the result back to a string",
        "Store it again"
      ]
    },
    {
      "type": "paragraph",
      "html": "This is computationally wasteful and unnecessary."
    },
    {
      "type": "paragraph",
      "html": "ClickHouse already supports multiple structured data types:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "JSON / Object types",
        "Map types",
        "Nested columns",
        "Explicit flattened columns"
      ]
    },
    {
      "type": "paragraph",
      "html": "Serializing structured data into opaque strings eliminates:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Query optimization",
        "Column compression",
        "Indexing capabilities",
        "Type safety"
      ]
    },
    {
      "type": "paragraph",
      "html": "It also dramatically increases CPU overhead during query execution."
    },
    {
      "type": "paragraph",
      "html": "In a columnar database designed for analytical workloads, flattening structured data into proper columns usually produces the best performance."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "6. Database Access Anti-Pattern: Per-Row Queries and Connection Exhaustion"
    },
    {
      "type": "paragraph",
      "html": "One of the most critical failures in the system is not architectural at a high level, but operational at the code level — and it directly impacts reliability."
    },
    {
      "type": "paragraph",
      "html": "A production incident exposed a severe flaw: the service opens a new ClickHouse connection and executes a full table scan <strong>for every single cell processed</strong>."
    },
    {
      "type": "paragraph",
      "html": "The failure manifests as:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<code>Code: 210 — Connection reset by peer</code>",
        "Parse jobs failing under moderate load",
        "ClickHouse is actively terminating connections"
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>What’s happening under the hood?</strong>"
    },
    {
      "type": "paragraph",
      "html": "Inside the parsing pipeline:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Each measurement object triggers a <code>cell_id</code> lookup",
        "Which opens a <strong>new database connection and runs a full SELECT query</strong>"
      ]
    },
    {
      "type": "paragraph",
      "html": "This happens <strong>inside a loop over all cells</strong>."
    },
    {
      "type": "paragraph",
      "html": "So for a file with 1,000 cells:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "1,000 database connections are opened",
        "1,000 full table scans are executed",
        "Each lookup performs an O(n) linear scan"
      ]
    },
    {
      "type": "paragraph",
      "html": "This is not just inefficient — it is catastrophic under load."
    },
    {
      "type": "paragraph",
      "html": "<strong>Why does this fail in practice?</strong>"
    },
    {
      "type": "paragraph",
      "html": "Databases are not designed for this access pattern. Specifically:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Connection pools get exhausted",
        "TCP connections are reset under pressure",
        "Query latency compounds multiplicatively",
        "The system becomes I/O bound instead of CPU-bound"
      ]
    },
    {
      "type": "paragraph",
      "html": "This is a textbook violation of a core principle:"
    },
    {
      "type": "quote",
      "html": "<em>Never perform external I/O inside a tight processing loop.</em>"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "7. The Missing Abstraction: In-Memory Lookup"
    },
    {
      "type": "paragraph",
      "html": "The fix is trivial, which makes the mistake more costly."
    },
    {
      "type": "paragraph",
      "html": "Instead of querying the database per cell, the system should:"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "Fetch the reference data <strong>once per job</strong>",
        "Transform it into an in-memory structure",
        "Perform constant-time lookups during parsing"
      ]
    },
    {
      "type": "paragraph",
      "html": "Concretely:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Load the cell map once",
        "Build a dictionary keyed",
        "Pass that dictionary through the call chain"
      ]
    },
    {
      "type": "paragraph",
      "html": "This transforms:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Before:</strong> O(n²) behavior + N database calls",
        "<strong>After:</strong> O(n) behavior + 1 database call"
      ]
    },
    {
      "type": "paragraph",
      "html": "It also eliminates connection exhaustion."
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*geqCpcFwzaGD_MWIRI5AGg.png",
      "alt": "A Microservice That Teaches Everything Not to Do",
      "caption": "",
      "width": 1408,
      "height": 768
    },
    {
      "type": "heading",
      "level": 2,
      "text": "8. Secondary Issues Amplifying the Problem"
    },
    {
      "type": "paragraph",
      "html": "Several additional flaws made the situation worse:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "No connection timeouts → workers hang indefinitely",
        "No caching layer → repeated identical queries",
        "Linear scans instead of indexed lookups",
        "Weak exception handling → loss of stack traces"
      ]
    },
    {
      "type": "paragraph",
      "html": "These are not independent issues — they compound each other."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "9. Lack of Observability and Idempotency"
    },
    {
      "type": "paragraph",
      "html": "The architecture also lacks two key properties required for reliable pipelines:"
    },
    {
      "type": "paragraph",
      "html": "<strong>Idempotency:</strong><br>If a file is processed twice, the system should produce the same result without duplicating data."
    },
    {
      "type": "paragraph",
      "html": "<strong>Observability:</strong><br>Operators should be able to answer simple questions such as:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Which files were processed?",
        "Which ones failed?",
        "How long did processing take?",
        "Can we replay a file?"
      ]
    },
    {
      "type": "paragraph",
      "html": "Deleting files and avoiding job tracking make these questions impossible to answer reliably."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Final Thoughts"
    },
    {
      "type": "paragraph",
      "html": "None of these mistakes individually would necessarily break a system. But together they create a pipeline that is fragile, inefficient, and difficult to operate."
    },
    {
      "type": "paragraph",
      "html": "The most striking part is that the system does not fail because the problem is complex. It fails because fundamental engineering principles are ignored:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Deterministic coordination instead of randomness",
        "Appropriate storage for the workload",
        "Respect for database access patterns",
        "Separation of I/O from computation"
      ]
    },
    {
      "type": "paragraph",
      "html": "When those fundamentals are violated, engineers compensate with random sleeps, filename tricks, and excessive database calls — until the system collapses under its own weight."
    },
    {
      "type": "paragraph",
      "html": "And that is often the most expensive mistake of all."
    }
  ]
}