{ "slug": "The-Best-Design-Pattern-for-Query-Based-Change-Data-Capture--Hybrid-Stateful-Polling-with-Offset--95856947ef99", "title": "The Best Design Pattern for Query-Based Change Data Capture: Hybrid Stateful Polling with Offset…", "subtitle": "In today’s data-driven architectures, keeping downstream systems synchronized with source databases in near real time is no longer optional…", "excerpt": "In today’s data-driven architectures, keeping downstream systems synchronized with source databases in near real time is no longer optional…", "date": "2025-10-16", "tags": [ "Architecture", "Machine Learning" ], "readingTime": "21 min", "url": "https://medium.com/@mobinshaterian/the-best-design-pattern-for-query-based-change-data-capture-hybrid-stateful-polling-with-offset-95856947ef99", "hero": "https://cdn-images-1.medium.com/max/800/1*n84qM5RROI2V5Li9AXOcFA.png", "content": [ { "type": "heading", "level": 2, "text": "The Best Design Pattern for Query-Based Change Data Capture: Hybrid Stateful Polling with Offset Tracking + Adaptive Backoff" }, { "type": "paragraph", "html": "In today’s data-driven architectures, keeping downstream systems synchronized with source databases in near real time is no longer optional — it’s essential. Yet for many engineering teams, the ideal solution for Change Data Capture (CDC) — log-based streaming via transaction logs — is out of reach due to constraints like managed cloud databases, read-only access, or legacy infrastructure. This reality forces a reliance on query-based CDC, where data changes are detected by periodically polling the source. But not all polling is created equal. Naive, full-table scans or rigid fixed-interval checks lead to high database load, missed updates, or unnecessary delays. The true path to reliability and efficiency lies in a smarter approach: Hybrid Stateful Polling with Offset Tracking + Adaptive Backoff. This design pattern blends predictable scheduling, intelligent retry logic, and durable state management to deliver a production-grade CDC pipeline that’s both responsive and resilient — even when you can’t access the database logs." }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*n84qM5RROI2V5Li9AXOcFA.png", "alt": "The Best Design Pattern for Query-Based Change Data Capture: Hybrid Stateful Polling with Offset…", "caption": "", "width": 1024, "height": 1024 }, { "type": "heading", "level": 2, "text": "Polling with Stateful Offset Tracking" }, { "type": "paragraph", "html": "Polling with stateful offset tracking is a software design pattern where a system periodically checks a data source for new information, but instead of scanning everything each time, it uses a stored marker (the offset) to resume from where it last left off. This approach is ideal for processing event streams and ensuring no data is missed, even after a system crash." }, { "type": "heading", "level": 2, "text": "Polling Design Pattern in iWF" }, { "type": "heading", "level": 2, "text": "What is the Polling Design Pattern?" }, { "type": "paragraph", "html": "The Polling Design Pattern is a workflow architecture approach where your workflow periodically checks an external system or condition until a specific criterion is met. Once the condition is satisfied, the workflow can proceed to its next steps." }, { "type": "paragraph", "html": "Simple Polling: Using a fixed interval timer to check conditions regularly
Backoff Polling: Using an exponential backoff retry mechanism that adapts the polling frequency" }, { "type": "heading", "level": 2, "text": "Cron Schedule Workflow" }, { "type": "paragraph", "html": "A pattern for replacing traditional Orc jobs with workflow-based cron scheduling. This pattern demonstrates how to use scheduled workflows to execute tasks at specified intervals using CRON expressions. Key features include automatic initialization, configurable scheduling, and management through Temporal Cloud UI." }, { "type": "paragraph", "html": "Use Cases: Periodic data processing, cleanup tasks, automated reports, and scheduled maintenance. Endpoints: N/A (automatically scheduled)" }, { "type": "heading", "level": 2, "text": "When to Use Polling" }, { "type": "paragraph", "html": "Polling is invaluable in scenarios like:" }, { "type": "list", "ordered": false, "items": [ "Waiting for External Systems or checking in a loop: When your workflow needs to wait for a database, API, or another service to become available", "Resource Readiness: In cloud environments, waiting for infrastructure components to be ready", "Status Verification: Checking payment gateway statuses or job processing completions", "Data Ingestion: Periodically checking for new data from external providers" ] }, { "type": "heading", "level": 2, "text": "Simple Polling Explained" }, { "type": "paragraph", "html": "This code shows a polling loop that repeatedly checks if a system is ready:" }, { "type": "paragraph", "html": "How it works:" }, { "type": "list", "ordered": true, "items": [ "waitUntil() — Sets a timer for 10 seconds. The workflow pauses and waits this long before doing anything.", "execute() — After 10 seconds pass, it checks if the system is ready by calling checkSystemReadiness().", "The decision:" ] }, { "type": "list", "ordered": false, "items": [ "If the system is ready → move forward to the next step (CompletionState)", "If NOT ready → go back to the polling state and wait another 10 seconds" ] }, { "type": "paragraph", "html": "In plain English: “Every 10 seconds, ask ‘Is the system ready yet?’ If yes, continue. If no, ask again in 10 more seconds.”" }, { "type": "paragraph", "html": "Real-world analogy: It’s like checking on a pizza in the oven every 10 minutes — you wait 10 minutes, open the oven and look, and either take it out (it’s done) or close the door and check again later (it needs more time)." }, { "type": "paragraph", "html": "The downside of this approach is that it’s wasteful if the system becomes ready after just 2 seconds — you’d still wait the full 10 seconds before noticing." }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*lCG2Jk7TKlmiz6U9Fm0jAA.png", "alt": "The Best Design Pattern for Query-Based Change Data Capture: Hybrid Stateful Polling with Offset…", "caption": "", "width": 953, "height": 390 }, { "type": "heading", "level": 2, "text": "Backoff Polling Explained" }, { "type": "paragraph", "html": "This code demonstrates exponential backoff, a more intelligent retry strategy where the wait time increases each time a failure occurs." }, { "type": "paragraph", "html": "How it works:" }, { "type": "list", "ordered": true, "items": [ "initialIntervalSeconds(3) — First retry happens after 3 seconds", "backoffCoefficient(2f) — Each retry doubles the wait time:" ] }, { "type": "list", "ordered": false, "items": [ "1st attempt: immediate", "2nd attempt: wait 3 seconds", "3rd attempt: wait 6 seconds (3 × 2)", "4th attempt: wait 12 seconds (6 × 2)", "5th attempt: wait 24 seconds (12 × 2)" ] }, { "type": "list", "ordered": true, "items": [ "maximumIntervalSeconds(60) — Never wait longer than 60 seconds (caps the growth)", "maximumAttempts(5) — Give up after 5 tries total", "maximumAttemptsDurationSeconds(3600) — Also give up if 1 hour has passed" ] }, { "type": "paragraph", "html": "In plain English: “Try to call the external service. If it fails, wait a bit and try again. But wait a little longer each time you retry, up to a maximum of 1 minute between tries. Stop trying after 5 attempts or 1 hour, whichever comes first.”" }, { "type": "paragraph", "html": "Real-world analogy: It’s like calling customer service — if the line is busy, you hang up and call back. But you get smarter: call back in 3 seconds, then 6 seconds, then 12 seconds, etc., instead of hammering them every second." }, { "type": "paragraph", "html": "Why this is better than simple polling: It’s gentler on the external system (fewer requests when things are struggling) and adapts automatically — the framework handles all the retries for you." }, { "type": "heading", "level": 2, "text": "Choosing Between Simple and Backoff Polling" }, { "type": "heading", "level": 3, "text": "When to Use Simple Polling" }, { "type": "paragraph", "html": "Simple polling works best when you want a steady, predictable rhythm. Use it when checking every 10 seconds (or whatever interval you set) is what you actually need. It’s good for time-sensitive situations where delays compound, like monitoring a server that should be up within seconds. It’s also the choice if you need custom logic during each check, since you control exactly what happens." }, { "type": "heading", "level": 3, "text": "When to Use Backoff Polling" }, { "type": "paragraph", "html": "Backoff polling is smarter when failures are expected and you want to be gentle on both your system and the external service. Use it when you’re retrying something that might be temporarily unavailable (like a network call that’s failing). It automatically backs off, so you’re not hammering a struggling service. The framework handles all the retry logic for you, which is less code to maintain." }, { "type": "heading", "level": 3, "text": "Quick comparison:" }, { "type": "paragraph", "html": "Simple polling = “Check every X seconds, no matter what” (predictable, rigid)" }, { "type": "paragraph", "html": "Backoff polling = “Try, wait a bit, try again, wait longer, repeat” (adaptive, gentle)" }, { "type": "paragraph", "html": "In practice: If you’re checking something that should work quickly and predictably, go simple. If you’re dealing with external services that might be flaky or overwhelmed, backoff is usually the better choice because it automatically adapts to the situation." }, { "type": "heading", "level": 2, "text": "Backoff Polling is Better for Your Case" }, { "type": "heading", "level": 3, "text": "Why backoff polling fits better:" }, { "type": "paragraph", "html": "You don’t know exactly when the data will be ready, so a fixed 10-second check (simple polling) is a gamble. You might check too early and waste resources, or check too late and delay unnecessarily. Backoff polling handles this uncertainty gracefully — it starts checking quickly (3 seconds), then gradually spaces out the checks if the data isn’t there yet." }, { "type": "heading", "level": 3, "text": "How it works for you:" }, { "type": "paragraph", "html": "Since you only have read access and no exact timestamp, your checks might fail initially. Backoff polling is designed exactly for this — it expects failures and retries with increasing patience. After a few quick attempts, if the data still isn’t ready, it waits longer between checks instead of constantly hammering your database." }, { "type": "heading", "level": 3, "text": "Practical example:" }, { "type": "list", "ordered": false, "items": [ "1st check: immediate (data not ready yet)", "2nd check: 3 seconds later (still not ready)", "3rd check: 6 seconds later (still not ready)", "4th check: 12 seconds later (data is finally there!)" ] }, { "type": "paragraph", "html": "You end up with fewer total checks while still catching the data reasonably quickly. Plus, you set a maximum wait time (60 seconds max per attempt) and total attempt limit (5 tries), so you don’t wait forever if something goes wrong." }, { "type": "paragraph", "html": "Bottom line: Backoff polling is more forgiving of uncertainty and wastes fewer resources when you don’t know the exact timing of your data." }, { "type": "heading", "level": 2, "text": "When to Use Backoff Polling" }, { "type": "paragraph", "html": "Rate-Limited APIs — If an external API throttles you for too many requests, backoff polling automatically slows down. Instead of hammering it every 10 seconds, you’re waiting 3, then 6, then 12 seconds. The API stays happy, you don’t get blocked." }, { "type": "paragraph", "html": "Unpredictable Availability — When you have no idea when a resource will be ready (like your database scenario), backoff adapts. It doesn’t waste time with rigid intervals — it naturally spaces out checks as failures continue." }, { "type": "paragraph", "html": "Reduce System Load — The longer intervals mean fewer requests overall. If something takes 5 minutes to be ready, simple polling might check 30 times, but backoff polling checks maybe 10 times. Your system and the external service both benefit." }, { "type": "paragraph", "html": "Simpler Code — You don’t write retry logic. The framework handles everything — the timing, the attempts, and when to give up. You just write the logic for one attempt." }, { "type": "paragraph", "html": "Cost Optimization — Fewer operations mean lower costs if you’re paying per API call or per workflow operation. This is especially valuable at scale." }, { "type": "paragraph", "html": "For your database situation specifically, you get all these benefits. You’re not hammering the database with constant reads; the framework handles the retry logic automatically, and you save resources by spacing out checks intelligently." }, { "type": "heading", "level": 2, "text": "How Stateful Offset Tracking Works" }, { "type": "paragraph", "html": "The Core Idea: The system remembers where it left off, so it never processes the same data twice or loses track of what’s been done." }, { "type": "paragraph", "html": "The Three Key Steps:" }, { "type": "paragraph", "html": "1. Offset Storage — Think of an offset as a bookmark. After processing record #1000, the system saves “1000” in a durable place (database, file, etc.). This bookmark survives crashes and restarts." }, { "type": "paragraph", "html": "2. The Polling Loop:" }, { "type": "list", "ordered": false, "items": [ "Start: When the system boots up, it reads the bookmark (“We last processed record 1000”)", "Fetch: Poll for new data starting from record 1001 onward", "Process: Handle the new batch of records", "Commit: Save the new bookmark (“Now we’ve processed up to record 1050”)", "Repeat: Next poll starts from 1051" ] }, { "type": "paragraph", "html": "3. Failure Recovery — If everything crashes halfway through processing record 1025, the system restarts, reads the bookmark (which still says 1020 because that was the last committed offset), and resumes from there. No data lost, nothing processed twice." }, { "type": "paragraph", "html": "Real-world analogy: It’s like reading a book — you bookmark your current page. If you fall asleep mid-chapter, you wake up, check your bookmark, and continue from exactly where you left off. You don’t restart the book, and you don’t skip ahead either." }, { "type": "paragraph", "html": "Why this matters: With offsets, your system is reliable and predictable, even when things fail unexpectedly." }, { "type": "heading", "level": 2, "text": "Exponential Backoff Polling Service with Optimization Tracking" }, { "type": "paragraph", "html": "This Go application implements a sophisticated polling service using Uber Fx dependency injection that continuously checks for data availability with exponential backoff strategy. The service operates perpetually in cycles, where each cycle has a maximum duration of 30 minutes. It starts with an initial 1-minute interval between checks and exponentially increases the wait time (doubling each attempt) up to a maximum of 10 minutes, capped at 5 attempts per cycle. After each cycle completes, the service waits 5 minutes before starting the next cycle, allowing for continuous monitoring of external systems or databases with read-only access." }, { "type": "paragraph", "html": "The system intelligently tracks the performance metrics of each polling cycle, automatically calculating the average optimal check interval based on historical data. Thread-safe operations ensure reliable concurrent access to shared state, while graceful shutdown mechanisms properly close channels and prevent resource leaks. The service integrates with Uber Fx’s lifecycle management to handle startup and cleanup operations, and provides detailed logging at each step showing elapsed time, attempt counts, and computed optimal intervals. This design is particularly useful for monitoring systems where the exact timing of data availability is unpredictable, combining resource efficiency with robustness through exponential backoff while maintaining statistical insights into optimal polling behavior." }, { "type": "code", "lang": "go", "code": "package main\nimport (\n\"context\"\n\"fmt\"\n\"sync\"\n\"time\"\n\"go.uber.org/fx\")// BackoffConfig holds the configuration for backoff polling\ntype BackoffConfig struct {\n InitialIntervalMinutes int\n MaximumIntervalMinutes int\n BackoffCoefficient float64\n MaximumAttempts int\n MaximumAttemptsDurationMinutes int\n}\n// PollingResult holds the result of a polling attempt\ntype PollingResult struct {\n Success bool Data\n interface{\n }\n Error error Attempt int\n}\n// BackoffPoller handles polling with exponential backoff\ntype BackoffPoller struct {\n config BackoffConfig\n}\n// NewBackoffPoller creates a new BackoffPoller with given config\nfunc NewBackoffPoller(config BackoffConfig) *BackoffPoller {\n return &BackoffPoller{\n config: config\n }\n}\n// Poll executes the polling with exponential backoff\nfunc (bp *BackoffPoller) Poll(ctx context.Context, attemptFunc func() (bool, interface{\n}, error)) (*PollingResult, error) {\n startTime := time.Now() attempt := 0 currentInterval :=\n time.Duration(bp.config.InitialIntervalMinutes) * time.Minute maxInterval :=\n time.Duration(bp.config.MaximumIntervalMinutes) * time.Minute for {\n attempt++ // Check if\n we've exceeded max attempts if attempt > bp.config.MaximumAttempts { return &PollingResult{ Success: false, Attempt: attempt, Error: fmt.Errorf(\"max attempts (%d) exceeded\", bp.config.MaximumAttempts), }, nil } // Check if we've exceeded max duration elapsed := time.Since(startTime) maxDuration := time.Duration(bp.config.MaximumAttemptsDurationMinutes) * time.Minute if elapsed > maxDuration {\n return &PollingResult{\n Success: false,\n Attempt: attempt,\n Error: fmt.Errorf(\"max duration (%s) exceeded\", maxDuration),\n }, nil\n }\n // Check context cancellation select {\n case <-ctx.Done(): return &PollingResult{\n Success: false,\n Attempt: attempt,\n Error: ctx.Err(),\n }, nil default:\n }\n // Execute the attempt fmt.Printf(\"Attempt %d: Checking... (Elapsed: %.2fs)\\n\", attempt,\n elapsed.Seconds()) success, data, err := attemptFunc() if success {\n fmt.Printf(\"Attempt %d: Success! (Elapsed: %.2fs)\\n\", attempt,\n time.Since(startTime).Seconds()) return &PollingResult{\n Success: true,\n Data:\n data,\n Attempt: attempt,\n }, nil\n }\n // If this is the last attempt,\n return failure if attempt >= bp.config.MaximumAttempts {\n return &PollingResult{\n Success: false,\n Data:\n data,\n Error: err,\n Attempt: attempt,\n }, nil\n }\n // Calculate next backoff interval\n fmt.Printf(\"Attempt %d: Failed (%v). Waiting %v before retry...\\n\", attempt, err,\n currentInterval) // Wait before next attempt select {\n case <-time.After(currentInterval): case <-ctx.Done(): return &PollingResult{\n Success: false,\n Attempt: attempt,\n Error: ctx.Err(),\n }, nil\n }\n // Increase interval for next attempt (exponential backoff) nextInterval :=\n time.Duration(float64(currentInterval) * bp.config.BackoffCoefficient) if nextInterval >\n maxInterval {\n currentInterval = maxInterval\n }\n else {\n currentInterval = nextInterval\n }\n }\n}\n// OptimalCheckResult contains statistics about polling performance\ntype OptimalCheckResult struct {\n Success bool Data interface{\n }\n TotalDuration time.Duration TotalAttempts int AverageCheckInterval time.Duration OptimalInterval\n time.Duration Error error\n}\n// PollingService handles the polling operations with optimization tracking\ntype PollingService struct {\n poller *BackoffPoller stopCh chan struct{\n }\n doneCh chan struct{\n }\n maxGlobalDuration time.Duration checkIntervals []time.Duration mu sync.Mutex\n}\n// NewPollingService creates a new PollingService\nfunc NewPollingService(poller *BackoffPoller, maxGlobalDuration time.Duration) *PollingService {\n return &PollingService{\n poller: poller, stopCh: make(chan struct{\n }), doneCh: make(chan struct{\n }), maxGlobalDuration: maxGlobalDuration, checkIntervals:\n make([]time.Duration, 0),\n }\n}\n// recordInterval records the interval between checks for optimization\nfunc (ps *PollingService) recordInterval(interval time.Duration) {\n ps.mu.Lock() defer ps.mu.Unlock() ps.checkIntervals = append(ps.checkIntervals, interval)\n}\n// calculateOptimalInterval calculates the average interval for future cycles\nfunc (ps *PollingService) calculateOptimalInterval() time.Duration {\n ps.mu.Lock() defer ps.mu.Unlock() if len(ps.checkIntervals) == 0 {\n return 0\n }\n var total time.Duration for _, interval := range ps.checkIntervals {\n total += interval\n }\n return total / time.Duration(len(ps.checkIntervals))\n}\n// Start begins the perpetual polling loop with maximum global duration per cycle\nfunc (ps *PollingService) Start(ctx context.Context) error {\n cycleCount := 0 for {\n select {\n case <-ctx.Done(): fmt.Println(\"Polling service context cancelled\") close(ps.doneCh)\n return ctx.Err() case <-ps.stopCh: fmt.Println(\"Polling service stop signal received\")\n close(ps.doneCh) return nil default:\n }\n cycleCount++ fmt.Printf(\"\\n=== Polling Cycle #%d ===\\n\", cycleCount) // Create a context\n with maximum duration for this polling cycle cycleCtx, cancel := context.WithTimeout(ctx,\n ps.maxGlobalDuration) // Example: Polling a database or external service counter := 0\n attemptFunc :=\n func() (bool, interface{\n }, error) {\n counter++ // Simulate: data becomes available after 3 attempts if counter >= 3 {\n return true, \"Data is ready!\", nil\n }\n return false, nil, fmt.Errorf(\"data not ready yet\")\n }\n startCycle := time.Now() result, err := ps.poller.Poll(cycleCtx, attemptFunc) cycleDuration\n := time.Since(startCycle) cancel() if err != nil {\n fmt.Printf(\"Polling error: %v\\n\", err)\n }\n else {\n ps.recordInterval(cycleDuration) fmt.Printf(\"\\n--- Polling Cycle Result ---\\n\")\n fmt.Printf(\"Success: %v\\n\", result.Success) fmt.Printf(\"Attempts: %d\\n\", result.Attempt)\n fmt.Printf(\"Cycle Duration: %.2fs\\n\", cycleDuration.Seconds()) fmt.Printf(\"Data: %v\\n\",\n result.Data) optimalInterval := ps.calculateOptimalInterval()\n fmt.Printf(\"Average Check Interval: %.2fs\\n\", optimalInterval.Seconds()) if result.Error\n != nil {\n fmt.Printf(\"Error: %v\\n\", result.Error)\n }\n }\n // Wait before starting next cycle (unless stopped)\n fmt.Println(\"\\nWaiting 5 minutes before next cycle...\") select {\n case <-time.After(5 * time.Minute): case <-ps.stopCh:\n fmt.Println(\"Polling service stop signal received during wait\") close(ps.doneCh) return\n nil case <-ctx.Done(): fmt.Println(\"Polling service context cancelled during wait\")\n close(ps.doneCh) return ctx.Err()\n }\n }\n}\n// Stop gracefully stops the polling service\nfunc (ps *PollingService) Stop() {\n close(ps.stopCh) <-ps.doneCh optimalInterval := ps.calculateOptimalInterval()\n fmt.Printf(\"=== Polling Service Stopped ===\\n\") fmt.Printf(\"Total Check Cycles: %d\\n\",\n len(ps.checkIntervals)) if len(ps.checkIntervals) > 0 {\n fmt.Printf(\"Average Optimal Interval: %.2fs\\n\", optimalInterval.Seconds())\n }\n fmt.Println(\"Polling service stopped successfully\")\n}\n// Providers for dependency injection\nfunc provideBackoffConfig() BackoffConfig {\n return BackoffConfig{\n InitialIntervalMinutes: 1, MaximumIntervalMinutes: 10, BackoffCoefficient: 2.0,\n MaximumAttempts: 5, MaximumAttemptsDurationMinutes: 30,\n }\n}\nfunc provideBackoffPoller(config BackoffConfig) *BackoffPoller {\n return NewBackoffPoller(config)\n}\n// Maximum time to check data in each cycle (e.g., 5 minutes\n)\nfunc providePollingService(poller *BackoffPoller) *PollingService {\n maxGlobalDuration := 5 * time.Minute // Maximum time to check data per cycle\n return NewPollingService(poller, maxGlobalDuration)\n}\n// Lifecycle hook to start polling service\nfunc startPollingService(lc fx.Lifecycle, ps *PollingService) {\n lc.Append(fx.Hook{\n OnStart: func(ctx context.Context) error {\n fmt.Println(\"Starting polling service...\") go ps.Start(ctx) return nil\n }, OnStop: func(ctx context.Context) error {\n fmt.Println(\"Stopping polling service...\") ps.Stop()\n fmt.Println(\"Polling service cleanup completed\") return nil\n },\n })\n}\n// Example usage\nfunc main() {\n app := fx.New(fx.Provide(provideBackoffConfig, provideBackoffPoller, providePollingService,),\n fx.Invoke(startPollingService),) app.Run()\n}" }, { "type": "heading", "level": 2, "text": "Simple Polling with Cron Jobs in Go" }, { "type": "paragraph", "html": "This Go application implements a straightforward polling mechanism using Uber Fx dependency injection and cron-based scheduling that executes a single data fetch attempt at fixed, predictable intervals. Unlike backoff polling, this service makes only one attempt per scheduled execution without any retry logic or exponential delays. The polling is triggered by cron expressions (e.g., every 5 minutes, daily at 9 AM), making it ideal for systems where you need consistent, reliable checks at specific times. Each polling cycle immediately returns success or failure based on that single attempt, and the service operates indefinitely until gracefully stopped, with each execution separated by the cron schedule rather than being continuous." }, { "type": "paragraph", "html": "The service includes built-in statistics tracking that monitors total polls executed, successful attempts, failed attempts, and calculates the success rate in real-time using thread-safe operations. The cron scheduler ensures precise timing control, eliminating the overhead of continuous loops and waiting intervals. Uber Fx manages the service lifecycle with proper startup and shutdown hooks that close channels safely and display final statistics upon termination. This design is particularly useful for periodic data validation, scheduled health checks, or regular data synchronization tasks where you need simple, predictable behavior without the complexity of exponential backoff or adaptive retry mechanisms." }, { "type": "code", "lang": "go", "code": "package main\nimport (\n\"context\"\n\"fmt\"\n\"sync\"\n\"time\"\n\"github.com/robfig/cron/v3\"\n\"go.uber.org/fx\")// SimplePollingConfig holds the configuration for simple polling\ntype SimplePollingConfig struct {\n CheckIntervalMinutes int\n}\n// PollingResult holds the result of a polling attempt\ntype PollingResult struct {\n Success bool Data interface{\n }\n Error error Timestamp time.Time\n}\n// SimplePoller handles simple polling with fixed intervals\ntype SimplePoller struct {\n config SimplePollingConfig\n}\n// NewSimplePoller creates a new SimplePoller with given config\nfunc NewSimplePoller(config SimplePollingConfig) *SimplePoller {\n return &SimplePoller{\n config: config\n }\n}\n// Poll executes a single polling check\nfunc (sp *SimplePoller) Poll(ctx context.Context, attemptFunc func() (bool, interface{\n}, error)) *PollingResult {\n fmt.Println(\"Executing poll check...\") // Check context cancellation select {\n case <-ctx.Done(): return &PollingResult{\n Success: false, Error: ctx.Err(), Timestamp: time.Now(),\n }\n default:\n }\n // Execute the attempt success, data, err := attemptFunc() result := &PollingResult{\n Success: success, Data: data, Error: err, Timestamp: time.Now(),\n }\n if success {\n fmt.Printf(\"Poll Success! Data: %v\\n\", data)\n }\n else {\n fmt.Printf(\"Poll Failed: %v\\n\", err)\n }\n return result\n}\n// PollingStatistics tracks polling performance\ntype PollingStatistics struct {\n TotalPolls int\n SuccessfulPolls int\n FailedPolls int\n LastPollResult *PollingResult\n mu sync.Mutex\n}\n// recordResult records a polling result\nfunc (ps *PollingStatistics) recordResult(result *PollingResult) {\n ps.mu.Lock() defer ps.mu.Unlock() ps.TotalPolls++ ps.LastPollResult = result if result.Success {\n ps.SuccessfulPolls++\n }\n else {\n ps.FailedPolls++\n }\n}\n// getStatistics returns current statistics\nfunc (ps *PollingStatistics) getStatistics() (total int, successful int, failed int) {\n ps.mu.Lock() defer ps.mu.Unlock() return ps.TotalPolls, ps.SuccessfulPolls, ps.FailedPolls\n}\n// SimplePollingService handles cron-scheduled simple polling\ntype SimplePollingService struct {\n poller *SimplePoller stopCh chan struct{\n }\n doneCh chan struct{\n }\n cronJob\n cron.Cron cronSchedule string statistics *PollingStatistics\n}\n// NewSimplePollingService creates a new SimplePollingService\nfunc NewSimplePollingService(poller *SimplePoller, cronSchedule string) *SimplePollingService {\n return &SimplePollingService{\n poller: poller, stopCh: make(chan struct{\n }), doneCh: make(chan struct{\n }), cronJob: *cron.New(), cronSchedule: cronSchedule, statistics: &PollingStatistics{\n },\n }\n}\n// Start begins the cron-scheduled simple polling\nfunc (sps *SimplePollingService) Start(ctx context.Context) error {\n // Add cron job _, err := sps.cronJob.AddFunc(sps.cronSchedule,\n func() {\n select {\n case <-sps.stopCh: return default:\n }\n fmt.Printf(\"\\n=== Polling at %s ===\\n\", time.Now().Format(\"2006-01-02 15:04:05\")) // Single\n attempt to fetch data (no retries) attemptFunc :=\n func() (bool, interface{\n }, error) {\n // Simulate: try to fetch data once from database or external service // Return success\n or failure on first attempt\n return true, \"Data fetched successfully!\", nil\n }\n // Execute the poll (single attempt, no retries) result := sps.poller.Poll(ctx, attemptFunc)\n // Record the result sps.statistics.recordResult(result) // Display statistics total,\n successful, failed := sps.statistics.getStatistics() fmt.Printf(\"\\n--- Statistics ---\\n\")\n fmt.Printf(\"Total Polls: %d\\n\", total) fmt.Printf(\"Successful: %d\\n\", successful)\n fmt.Printf(\"Failed: %d\\n\", failed) fmt.Printf(\"Success Rate: %.2f%%\\n\",\n float64(successful)/float64(total)*100)\n }) if err != nil {\n fmt.Printf(\"Error adding cron job: %v\\n\", err) close(sps.doneCh) return err\n }\n fmt.Printf(\"Simple polling started with cron schedule: %s\\n\", sps.cronSchedule)\n fmt.Println(\"Poll execution times:\") fmt.Println(\" */1 * * * * - Every minute\")\n fmt.Println(\" */5 * * * * - Every 5 minutes\") fmt.Println(\" 0 * * * * - Every hour\")\n fmt.Println(\" 0 9 * * * - Every day at 9:00 AM\")\n fmt.Println(\" 0 0 * * 0 - Every Sunday at midnight\") sps.cronJob.Start() // Wait for stop\n signal <-sps.stopCh close(sps.doneCh)\n return nil\n}\n// Stop gracefully stops the polling service\nfunc (sps *SimplePollingService) Stop() {\n sps.cronJob.Stop() close(sps.stopCh) <-sps.doneCh total, successful, failed :=\n sps.statistics.getStatistics() fmt.Printf(\"\\n=== Polling Service Stopped ===\\n\")\n fmt.Printf(\"Total Polls Executed: %d\\n\", total) fmt.Printf(\"Successful Polls: %d\\n\", successful)\n fmt.Printf(\"Failed Polls: %d\\n\", failed) if total > 0 {\n fmt.Printf(\"Success Rate: %.2f%%\\n\", float64(successful)/float64(total)*100)\n }\n fmt.Println(\"Polling service stopped successfully\")\n}\n// Providers for dependency injection\nfunc provideSimplePollingConfig() SimplePollingConfig {\n return SimplePollingConfig{\n CheckIntervalMinutes: 5,\n }\n}\nfunc provideSimplePoller(config SimplePollingConfig) *SimplePoller {\n return NewSimplePoller(config)\n}\n// CronSchedule examples:// \"*/1 * * * *\" - Every 1 minute// \"*/5 * * * *\" - Every 5 minutes//\n\"0 * * * *\" - Every hour at minute 0// \"0 9 * * *\" - Every day at 9:00 AM// \"0 0 * * 0\" - Every\nSunday at midnight\nfunc provideSimplePollingService(poller *SimplePoller) *SimplePollingService {\n cronSchedule := \"*/5 * * * *\" // Run every 5 minutes\n return NewSimplePollingService(poller, cronSchedule)\n}\n// Lifecycle hook to start polling service\nfunc startSimplePollingService(lc fx.Lifecycle, sps *SimplePollingService) {\n lc.Append(fx.Hook{\n OnStart: func(ctx context.Context) error {\n fmt.Println(\"Starting simple polling service with cron scheduling...\") go sps.Start(ctx)\n return nil\n }, OnStop: func(ctx context.Context) error {\n fmt.Println(\"Stopping simple polling service...\") sps.Stop()\n fmt.Println(\"Cleanup completed\") return nil\n },\n })\n}\n// Example usage\nfunc main() {\n app := fx.New(fx.Provide(provideSimplePollingConfig, provideSimplePoller,\n provideSimplePollingService,), fx.Invoke(startSimplePollingService),) app.Run()\n}" }, { "type": "heading", "level": 2, "text": "The Best Design Pattern for Query-Based Change Data Capture:" }, { "type": "heading", "level": 3, "text": "Hybrid Stateful Polling with Offset Tracking + Adaptive Backoff" }, { "type": "paragraph", "html": "In modern data architectures, Change Data Capture (CDC) is essential for keeping downstream systems — like data warehouses, analytics platforms, and microservices — synchronized with source databases in near real time. While log-based CDC (e.g., Debezium, AWS DMS) offers the gold standard in performance and fidelity, it’s often unavailable in managed cloud environments, legacy systems, or read-only database setups." }, { "type": "paragraph", "html": "This is where query-based CDC becomes indispensable. But naive polling — scanning entire tables every few minutes — is inefficient, error-prone, and doesn’t scale. So what’s the best way to build a reliable, efficient, and production-grade query-based CDC pipeline?" }, { "type": "paragraph", "html": "The answer lies in a powerful yet practical design pattern:
Hybrid Stateful Polling with Offset Tracking + Adaptive Backoff." }, { "type": "heading", "level": 2, "text": "Why Traditional Polling Falls Short" }, { "type": "paragraph", "html": "Before diving into the solution, let’s examine the limitations of common approaches:" }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*_l4t7yOR99hOnuL5ORR29g.png", "alt": "The Best Design Pattern for Query-Based Change Data Capture: Hybrid Stateful Polling with Offset…", "caption": "", "width": 945, "height": 368 }, { "type": "paragraph", "html": "None of these alone provides the reliability, efficiency, and observability needed for production systems." }, { "type": "heading", "level": 2, "text": "Introducing the Hybrid Pattern" }, { "type": "paragraph", "html": "The Hybrid Stateful Polling with Offset Tracking + Adaptive Backoff pattern combines three proven techniques into a cohesive, resilient architecture:" }, { "type": "list", "ordered": true, "items": [ "Cron-driven scheduling for predictable, bounded execution windows", "Exponential backoff polling within each window for responsiveness", "Stateful offset tracking for exactly-once semantics and crash recovery" ] }, { "type": "paragraph", "html": "This hybrid approach delivers the best of all worlds: low latency when data is ready early, resource efficiency when it’s delayed, and guaranteed progress even after failures." }, { "type": "heading", "level": 2, "text": "How It Works: Step by Step" }, { "type": "heading", "level": 2, "text": "1. Schedule Polling Cycles with Cron" }, { "type": "paragraph", "html": "A scheduler (e.g., Airflow, Temporal, cron, or cloud scheduler) triggers a polling cycle at regular intervals, say, every 5 minutes." }, { "type": "quote", "html": "🔁 This ensures bounded maximum latency: you’ll never wait more than 5 minutes to start looking for new data." }, { "type": "heading", "level": 2, "text": "2. Load the Last Processed Offset" }, { "type": "paragraph", "html": "At the start of each cycle, the system reads a persistent offset from a metadata store (e.g., a cdc_offsets table, Redis, or DynamoDB):" }, { "type": "code", "lang": "json", "code": "{\n \"table\": \"orders\",\n \"last_id\": 10542,\n \"last_updated_at\": \"2025-10-16T12:34:56Z\"\n}" }, { "type": "paragraph", "html": "This offset acts as a bookmark, telling the system where to resume." }, { "type": "heading", "level": 2, "text": "3. Run Adaptive Backoff Polling Within the Cycle" }, { "type": "paragraph", "html": "Instead of checking just once, the system enters a time-bounded backoff loop (e.g., max 4.5 minutes per 5-minute cycle):" }, { "type": "list", "ordered": false, "items": [ "Attempt 1: Wait 3 seconds → query for changes", "Attempt 2: Wait 6 seconds → query again", "Attempt 3: Wait 12 seconds → query again", "… up to a maximum interval (e.g., 60 seconds) and max attempts (e.g., 5)" ] }, { "type": "paragraph", "html": "Each query uses the offset with a small time buffer to handle clock skew:" }, { "type": "code", "lang": "sql", "code": "SELECT\n id,\n customer_id,\n amount,\n updated_at\nFROM orders\nWHERE updated_at > '2025-10-16T12:34:46Z' -- offset minus 10s buffer\n AND id > 10542\nORDER BY updated_at ASC, id ASC\nLIMIT 1000;" }, { "type": "paragraph", "html": "✅ If new records are found:" }, { "type": "paragraph", "html": "Process and send to sink (e.g., Kafka, Snowflake, database)
Only then update the offset to the latest id/updated_at" }, { "type": "paragraph", "html": "✅ If no records:" }, { "type": "paragraph", "html": "Continue backoff until cycle timeout" }, { "type": "heading", "level": 2, "text": "4. Ensure Idempotent Sink Writes" }, { "type": "paragraph", "html": "Downstream systems must handle potential duplicates (e.g., due to retries). Use upserts or deduplication keys:" }, { "type": "code", "lang": "sql", "code": "MERGE INTO analytics.orders AS targetUSING (VALUES ...) AS sourceON target.id = source.idWHEN\nMATCHED AND source.updated_at > target.updated_at\nTHEN UPDATE ...WHEN NOT MATCHED THEN INSERT ..." }, { "type": "heading", "level": 2, "text": "5. Graceful Failure Handling" }, { "type": "list", "ordered": false, "items": [ "If processing fails → do not update the offset", "On restart, the next cycle resumes from the last committed offset", "Failed batches can be routed to a dead-letter queue for inspection" ] }, { "type": "heading", "level": 2, "text": "Real-World Benefits" }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*s2DlYHalaXCZotG6w6hewQ.png", "alt": "The Best Design Pattern for Query-Based Change Data Capture: Hybrid Stateful Polling with Offset…", "caption": "", "width": 949, "height": 480 }, { "type": "heading", "level": 2, "text": "When to Use This Pattern" }, { "type": "paragraph", "html": "✅ Ideal for:" }, { "type": "list", "ordered": false, "items": [ "Cloud databases without binlog access (e.g., AWS RDS, Google Cloud SQL)", "Read-only database connections", "Tables with soft deletes and indexed updated_at columns", "Near-real-time ETL where sub-5-minute latency is acceptable" ] }, { "type": "paragraph", "html": "❌ Not ideal for:" }, { "type": "list", "ordered": false, "items": [ "Sub-second latency requirements → use log-based CDC", "Tables without monotonic update columns → consider triggers or shadow tables", "Extremely high-volume change streams → query-based CDC may not scale" ] }, { "type": "heading", "level": 2, "text": "Conclusion" }, { "type": "paragraph", "html": "In the era of data-driven systems, near-real-time synchronization between source databases and downstream systems is critical, yet log-based Change Data Capture (CDC) remains inaccessible for many due to constraints like managed cloud environments, read-only access, or legacy infrastructure. This article underscores the necessity of query-based CDC and introduces the Hybrid Stateful Polling with Offset Tracking + Adaptive Backoff design pattern as the optimal solution for such scenarios." }, { "type": "paragraph", "html": "By integrating three core components — cron-driven scheduling for predictable cycles, exponential backoff polling for adaptive responsiveness, and stateful offset tracking for reliability — the hybrid pattern addresses the inefficiencies and risks of naive query-based CDC. Cron scheduling ensures bounded maximum latency, preventing delays beyond predefined intervals, while adaptive backoff within each cycle minimizes redundant checks and reduces load on the source database by spacing out retries as failures persist. Stateful offset tracking, with durable storage and failure recovery mechanisms, guarantees exactly-once processing semantics, safeguarding against data loss or duplication even during crashes." }, { "type": "paragraph", "html": "This pattern excels in environments where sub-second latency is not required but consistent, reliable data capture is paramount. It is particularly well-suited for cloud databases with limited log access, read-only connections, tables with indexed update timestamps, and ETL pipelines tolerating sub-5-minute latency. However, it is not ideal for ultra-low-latency needs or high-volume change streams where log-based CDC or specialized tools would be more effective." }, { "type": "paragraph", "html": "The provided Go implementations of adaptive backoff and simple polling services further illustrate its practicality, demonstrating features like thread-safe statistics tracking, graceful shutdowns, and integration with dependency injection frameworks (Uber Fx). By balancing responsiveness, resource efficiency, and robustness, the Hybrid Stateful Polling pattern empowers engineering teams to build production-grade CDC pipelines that meet real-world demands without compromising on reliability." }, { "type": "paragraph", "html": "In summary, when log-based CDC is out of reach, the Hybrid Stateful Polling with Offset Tracking + Adaptive Backoff pattern emerges as the gold standard for query-based CDC, offering a structured, efficient, and resilient approach to keep data ecosystems synchronized." }, { "type": "paragraph", "html": "Your Business — On AutoPilot with DDImedia AI Assistant
(Join Our Waitlist)" }, { "type": "paragraph", "html": "Visit us at DataDrivenInvestor.com" }, { "type": "paragraph", "html": "Join our creator ecosystem here." }, { "type": "paragraph", "html": "DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1" }, { "type": "paragraph", "html": "Follow us on LinkedIn, Twitter, YouTube, and Facebook." } ] }