{
  "slug": "Designing-a-Scalable-Poller-Dispatcher-Worker-Architecture-in-Go-eb5576116190",
  "title": "Designing a Scalable Poller–Dispatcher–Worker Architecture in Go",
  "subtitle": "This architecture establishes a production-grade job processing system in Go by decoupling logic into a Poller (decision-maker), Dispatcher",
  "excerpt": "This architecture establishes a production-grade job processing system in Go by decoupling logic into a Poller (decision-maker), Dispatcher",
  "date": "2025-12-30",
  "tags": [
    "Go",
    "Architecture",
    "Machine Learning"
  ],
  "readingTime": "5 min",
  "url": "https://medium.com/@mobinshaterian/designing-a-scalable-poller-dispatcher-worker-architecture-in-go-eb5576116190",
  "hero": "https://cdn-images-1.medium.com/max/800/1*ad8CdGbXbmt5SSS7LdEKzg.png",
  "content": [
    {
      "type": "heading",
      "level": 2,
      "text": "Designing a Scalable Poller–Dispatcher–Worker Architecture in Go"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Introduction"
    },
    {
      "type": "paragraph",
      "html": "Background job processing is a foundational building block in modern backend systems. Whether handling email delivery, data synchronization, cache warming, or event-driven workflows, a well-designed job execution pipeline must be <strong>scalable, testable, and operationally safe</strong>."
    },
    {
      "type": "paragraph",
      "html": "This article presents a production-ready design for a <strong>Poller–Dispatcher–Worker–Job</strong> architecture in Go, optimized for:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Long-running services",
        "Deterministic behavior",
        "Clean lifecycle management",
        "High testability"
      ]
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*ad8CdGbXbmt5SSS7LdEKzg.png",
      "alt": "Designing a Scalable Poller–Dispatcher–Worker Architecture in Go",
      "caption": "",
      "width": 1024,
      "height": 1024
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Go-Worker"
    },
    {
      "type": "paragraph",
      "html": "Go-worker is an open source project and has a simple impediment of a <strong>Poller–Dispatcher–Worker–Job</strong> architecture in Go."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Core Design Goals"
    },
    {
      "type": "paragraph",
      "html": "The system is designed around the following principles:"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "<strong>Separation of responsibilities</strong>",
        "<strong>Explicit control flow</strong>",
        "<strong>Safe concurrency</strong>",
        "<strong>Graceful startup and shutdown</strong>",
        "<strong>Test-first architecture</strong>"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "High-Level Architecture"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*0AFouIjv8-J84RNjbtNktg.png",
      "alt": "Designing a Scalable Poller–Dispatcher–Worker Architecture in Go",
      "caption": "",
      "width": 680,
      "height": 733
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Component Responsibilities"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1. Job: The Atomic Unit of Work"
    },
    {
      "type": "paragraph",
      "html": "A <code>Job</code> represents a single business action."
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type Job interface {\n    ID() string Service() string Execute(ctx context.Context) error\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Key characteristics:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Stateless or minimally stateful",
        "Self-contained",
        "Retry-safe",
        "Executable independently"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2. Worker: Concurrent Job Executor"
    },
    {
      "type": "paragraph",
      "html": "Workers are long-lived goroutines that process jobs from a channel."
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type Worker interface {\n    Start(ctx context.Context) Stop()\n}"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type SimpleWorker struct {\n    id int jobQueue <-chan job.Job cancel context.CancelFunc\n}\nfunc NewSimpleWorker(id int, jobQueue <-chan job.Job,) *SimpleWorker {\n    return &SimpleWorker{\n        id: id, jobQueue: jobQueue,\n    }\n}\nfunc (w *SimpleWorker) Start(ctx context.Context) {\n    ctx, cancel := context.WithCancel(ctx) w.cancel = cancel go func() {\n        log.Printf(\"[worker-%d] started\\n\", w.id) for {\n            //blocker select {\n                case <-ctx.Done():\n                log.Printf(\"[worker-%d] stopped\\n\", w.id)\n                return case j := <-w.jobQueue:\n                w.handleJob(ctx, j)\n            }\n        }\n    }\n    ()\n}\nfunc (w *SimpleWorker) Stop() {\n    if w.cancel != nil {\n        w.cancel()\n    }\n}\nfunc (w *SimpleWorker) handleJob(ctx context.Context, j job.Job) {\n    log.Printf(\"[worker-%d] executing job id=%s service=%s\\n\", w.id, j.ID(), j.Service(),) if err :=\n    j.Execute(ctx);\n    err != nil {\n        log.Printf(\"[worker-%d] job failed id=%s error=%v\\n\", w.id, j.ID(), err,)\n    }\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Workers</strong>:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Block on job queues",
        "Execute jobs synchronously",
        "Report success or failure",
        "Never make scheduling decisions"
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>Design rule:</strong>"
    },
    {
      "type": "quote",
      "html": "<em>Workers execute work — they never decide </em>what<em> work to run.</em>"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3. Dispatcher: Routing and Backpressure Control"
    },
    {
      "type": "paragraph",
      "html": "The dispatcher is responsible for:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Managing job queues",
        "Enforcing concurrency limits",
        "Routing jobs to the correct worker pool"
      ]
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type Dispatcher interface {\n    Start() Stop() Dispatch(j job.Job) error\n}\nvar ErrServiceNotRegistered = errors.New(\"service not registered\"\n)\ntype Service struct {\n    ctx\n    context.Context cancel context.CancelFunc mu sync.RWMutex queues map[string]chan job.Job workers\n    map[string][]worker.Worker\n}\nfunc New() *Service {\n    ctx, cancel := context.WithCancel(context.Background()) return &Service{\n        ctx: ctx, cancel: cancel, queues: make(map[string]chan job.Job), workers:\n        make(map[string][]worker.Worker),\n    }\n}\nfunc (d *Service) Register(service string, workerCount int, queueSize int,) {\n    d.mu.Lock() defer d.mu.Unlock() if _, exists := d.queues[service];\n    exists {\n        fmt.Println(\"service already registered\") return\n    }\n    queue := make(chan job.Job, queueSize) d.queues[service] = queue var workers []worker.Worker for\n    i := 0;\n    i < workerCount;\n    i++ {\n        w := worker.NewSimpleWorker(i+1, queue) workers = append(workers, w)\n    }\n    d.workers[service] = workers\n}\nfunc (d *Service) Start() {\n    d.mu.RLock() defer d.mu.RUnlock() for _, ws := range d.workers {\n        for _, w := range ws {\n            w.Start(d.ctx)\n        }\n    }\n}\nfunc (d *Service) Dispatch(j job.Job) error {\n    // Prevent dispatch after stop select {\n        case <-d.ctx.Done(): return context.Canceled default:\n    }\n    d.mu.RLock() queue, ok := d.queues[j.Service()] d.mu.RUnlock() if !ok {\n        return ErrServiceNotRegistered\n    }\n    // Safe send select {\n        case queue <- j: return nil case <-d.ctx.Done(): return context.Canceled\n    }\n}\nfunc (d *Service) Stop() {\n    d.cancel() d.mu.RLock() defer d.mu.RUnlock() for _, q := range d.queues {\n        close(q)\n    }\n}\nfunc RegisterLifecycle(lc fx.Lifecycle, d *Service) {\n    lc.Append(fx.Hook{\n        OnStart: func(ctx context.Context) error {\n            fmt.Println(\"starting dispatcher...\") d.Start() return nil\n        }, OnStop: func(ctx context.Context) error {\n            fmt.Println(\"stopping dispatcher...\") d.Stop() return nil\n        },\n    })\n}\nfunc RegisterServices(d *Service) {\n    // Example services d.Register(\"email\", 5, 100)\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Each service has:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Its own buffered queue",
        "A dedicated worker pool",
        "Independent scaling characteristics"
      ]
    },
    {
      "type": "paragraph",
      "html": "This design allows:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Different SLAs per service",
        "Backpressure isolation",
        "Clear failure domains"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "4. Poller: Time-Based Decision Engine"
    },
    {
      "type": "paragraph",
      "html": "The poller is the <strong>control plane</strong> of the system."
    },
    {
      "type": "code",
      "lang": "go",
      "code": "/*Fx App ├── Dispatcher (long-lived) │\n└── Worker pools ├── Poller (long-lived background process) │\n└── Infinite loop │ ├── Check condition (DB / cache / API) │ └── Dispatch job(s) └── Servers*/type\nPoller struct {\n    Dispatcher *dispatcher.Service Interval time.Duration ctx context.Context cancel\n    context.CancelFunc OnTick func(context.Context)\n}\nfunc New(dispatcher *dispatcher.Service,) *Poller {\n    p := &Poller{\n        Dispatcher: dispatcher, Interval: 500 * time.Microsecond,\n    }\n    p.OnTick = p.DefaultTick // default behavior\n    return p\n}\nfunc (p *Poller) Run(ctx context.Context) {\n    ticker := time.NewTicker(p.Interval) defer ticker.Stop() for {\n        fmt.Println(\"poller For loop ...\") select {\n            case <-ctx.Done(): return case <-ticker.C: p.OnTick(ctx)\n        }\n    }\n}\nfunc (p *Poller) DefaultTick(ctx context.Context) {\n    // Example condition (replace with DB/cache logic) log.Println(\"DefaultTick is running ...\") if\n    !p.conditionMet(ctx) {\n        return\n    }\n    j := example.NewExampleJob(\"job-123\", \"email\") _ = p.Dispatcher.Dispatch(j)\n}\nfunc (p *Poller) conditionMet(ctx context.Context) bool {\n    // DB / Redis / API / feature flag / backlog size\n    return true\n}\nfunc RegisterLifecycle(lc fx.Lifecycle, p *Poller) {\n    lc.Append(fx.Hook{\n        OnStart: func(ctx context.Context) error {\n            fmt.Println(\"starting dispatcher...\") p.ctx, p.cancel =\n            context.WithCancel(context.Background()) go p.Run(p.ctx) return nil\n        }, OnStop: func(ctx context.Context) error {\n            fmt.Println(\"stopping poller...\") if p.cancel != nil {\n                p.cancel()\n            }\n            return nil\n        },\n    })\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Responsibilities</strong>:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Runs on a fixed interval",
        "Checks conditions (DB, Redis, API, flags)",
        "Decides <em>whether</em> to create jobs",
        "Dispatches jobs via the dispatcher"
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>Key rule:</strong>"
    },
    {
      "type": "quote",
      "html": "<em>The poller decides </em>when<em> and </em>what<em> to run — never </em>how<em>.</em>"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Poller Execution Model"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func (p *Poller) Run(ctx context.Context) {\n    ticker := time.NewTicker(p.Interval) defer ticker.Stop() for {\n        select {\n            case <-ctx.Done(): return case <-ticker.C: p.OnTick(ctx)\n        }\n    }\n}"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Why This Matters"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Single execution path",
        "No hidden logic",
        "Fully overridable for tests",
        "Deterministic timing"
      ]
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/KZps3z5Dyns?feature=oembed"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Lifecycle Management (Uber Fx Friendly)"
    },
    {
      "type": "paragraph",
      "html": "Each long-lived component follows the same pattern:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func Register(lc fx.Lifecycle, svc Service) {\n    lc.Append(fx.Hook{\n        OnStart: func(ctx context.Context) error {\n            svc.Start() return nil\n        }, OnStop: func(ctx context.Context) error {\n            svc.Stop() return nil\n        },\n    })\n}"
    },
    {
      "type": "paragraph",
      "html": "This ensures:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Clean startup ordering",
        "Graceful shutdown",
        "No orphan goroutines",
        "Controlled resource release"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Backpressure and Safety"
    },
    {
      "type": "paragraph",
      "html": "The dispatcher enforces safety through:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Bounded queues",
        "Non-blocking dispatch",
        "Service-level isolation"
      ]
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func (d *serviceDispatcher) Dispatch(job Job) error {\n    select {\n        case d.queues[job.Service()] <- job: return nil default: return ErrQueueFull\n    }\n}"
    },
    {
      "type": "paragraph",
      "html": "This prevents:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Memory leaks",
        "Goroutine explosion",
        "Cascading failures"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Testability by Design"
    },
    {
      "type": "paragraph",
      "html": "The system is designed for <strong>deterministic testing</strong>."
    },
    {
      "type": "paragraph",
      "html": "<strong>Poller Test Example</strong>"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "p.OnTick = func(ctx context.Context) { tickCount.Add(1)}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Dispatcher Test Example</strong>"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "err := dispatcher.Dispatch(job)require.NoError(t, err)"
    },
    {
      "type": "paragraph",
      "html": "<strong>Worker Test Example</strong>"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "job.Execute = func(ctx context.Context) error { executed.Store(true) return nil}"
    },
    {
      "type": "paragraph",
      "html": "No mocks of time.<br>No sleeps in production logic.<br>No hidden side effects."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Tests"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Operational Benefits"
    },
    {
      "type": "paragraph",
      "html": "This architecture enables:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Horizontal scalability",
        "Clear observability boundaries",
        "Per-service tuning",
        "Graceful degradation",
        "Predictable behavior under load"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Common Anti-Patterns Avoided"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*wW6QZjDZeWRamdcnN6NGeA.png",
      "alt": "Designing a Scalable Poller–Dispatcher–Worker Architecture in Go",
      "caption": "",
      "width": 557,
      "height": 255
    },
    {
      "type": "heading",
      "level": 2,
      "text": "When to Use This Architecture"
    },
    {
      "type": "paragraph",
      "html": "Ideal for:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Background processing systems",
        "Scheduled workflows",
        "Event polling services",
        "ETL task orchestration",
        "Microservice sidecars"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Conclusion"
    },
    {
      "type": "paragraph",
      "html": "A Poller–Dispatcher–Worker architecture succeeds or fails based on <strong>clarity of responsibility</strong> and <strong>explicit control flow</strong>."
    },
    {
      "type": "paragraph",
      "html": "By separating:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Decision making (Poller)</strong>",
        "<strong>Routing and scaling (Dispatcher)</strong>",
        "<strong>Execution (Worker)</strong>",
        "<strong>Business logic (Job)</strong>"
      ]
    },
    {
      "type": "paragraph",
      "html": "This design is simple enough to start small, yet robust enough to scale into production-grade workloads."
    },
    {
      "type": "paragraph",
      "html": "Your Business — On AutoPilot with <em>DDImedia AI Assistant</em><br>(<a href=\"https://waitlist.ddimedia.ai/join-the-waitlist-aie\" target=\"_blank\" rel=\"noreferrer noopener\">Join Our Waitlist</a>)"
    },
    {
      "type": "paragraph",
      "html": "Visit us at <a href=\"https://www.datadriveninvestor.com/\" target=\"_blank\" rel=\"noreferrer noopener\"><em>DataDrivenInvestor.com</em></a>"
    },
    {
      "type": "paragraph",
      "html": "Join our creator ecosystem <a href=\"https://join.datadriveninvestor.com/\" target=\"_blank\" rel=\"noreferrer noopener\"><em>here</em></a>."
    },
    {
      "type": "paragraph",
      "html": "DDI Official Telegram Channel: <a href=\"https://t.me/+tafUp6ecEys4YjQ1\" target=\"_blank\" rel=\"noreferrer noopener\">https://t.me/+tafUp6ecEys4YjQ1</a>"
    },
    {
      "type": "paragraph",
      "html": "Follow us on <a href=\"https://www.linkedin.com/company/data-driven-investor\" target=\"_blank\" rel=\"noreferrer noopener\"><em>LinkedIn</em></a>, <a href=\"https://twitter.com/@DDInvestorHQ\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Twitter</em></a>, <a href=\"https://www.youtube.com/c/datadriveninvestor\" target=\"_blank\" rel=\"noreferrer noopener\"><em>YouTube</em></a>, and <a href=\"https://www.facebook.com/datadriveninvestor\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Facebook</em></a>."
    }
  ]
}