{
  "slug": "Go-WordPress--A-Robust-Web-Scraping-Architecture-for-Product-Aggregation-e99de3f91bd5",
  "title": "Go-WordPress: A Robust Web Scraping Architecture for Product Aggregation",
  "subtitle": "Introduction",
  "excerpt": "Introduction",
  "date": "2026-02-14",
  "tags": [
    "Go",
    "Architecture"
  ],
  "readingTime": "7 min",
  "url": "https://medium.com/@mobinshaterian/go-wordpress-a-robust-web-scraping-architecture-for-product-aggregation-e99de3f91bd5",
  "hero": "https://cdn-images-1.medium.com/max/800/1*v6KTWyj7hindSiOLF_NqaQ.png",
  "content": [
    {
      "type": "heading",
      "level": 2,
      "text": "Introduction"
    },
    {
      "type": "paragraph",
      "html": "The go-wordpress project represents a sophisticated approach to building a production-grade web scraping system in Go. Designed to extract, process, and manage product data from WordPress-based e-commerce websites, this project demonstrates best practices in concurrent scraping, data transformation, and enterprise application architecture. Whether you’re aggregating products from multiple sources or building a distributed crawling system, go-wordpress offers valuable insights into handling complex web extraction challenges at scale."
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*v6KTWyj7hindSiOLF_NqaQ.png",
      "alt": "Go-WordPress: A Robust Web Scraping Architecture for Product Aggregation",
      "caption": "",
      "width": 2752,
      "height": 1536
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Project Overview"
    },
    {
      "type": "paragraph",
      "html": "Go-WordPress is a comprehensive product scraping and management system built with Go, featuring concurrent web crawling, data normalization, caching layers, and both gRPC and HTTP APIs. The system is designed to handle Persian language content (with full Unicode support) and implements sophisticated error handling and batch processing strategies."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Git Commit"
    },
    {
      "type": "paragraph",
      "html": "<strong>Key Components:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Web scraping with dynamic category discovery",
        "Concurrent product listing and description extraction",
        "Data cleaning and normalization (especially for Persian prices)",
        "Multiple storage backends (SQL database and Redis cache)",
        "gRPC and HTTP REST APIs",
        "Dependency injection with Uber’s <code>fx</code> framework",
        "Comprehensive test coverage"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Architecture Overview"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Modular Design with Dependency Injection"
    },
    {
      "type": "paragraph",
      "html": "The project leverages Uber’s <code>fx</code> framework to create a clean, modular architecture. The <code>NewApp()</code> function demonstrates this elegantly:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func NewApp() *fx.App {\n    return fx.New(fx.Provide(logger.NewLogger, config.NewConfig, sql.InitialDB, //...services and\n    controllers), fx.Invoke(// Initialization and lifecycle management),\n    )\n}"
    },
    {
      "type": "paragraph",
      "html": "This approach provides several advantages:"
    },
    {
      "type": "paragraph",
      "html": "<strong>Loose Coupling</strong>: Each component is defined independently and dependencies are automatically resolved by the container. Services don’t need to know how their dependencies are constructed."
    },
    {
      "type": "paragraph",
      "html": "<strong>Testability</strong>: Components can be easily mocked or substituted for testing. The dependency graph ensures that all dependencies are satisfied before initialization."
    },
    {
      "type": "paragraph",
      "html": "<strong>Lifecycle Management</strong>: The framework handles initialization order and graceful shutdown sequences automatically."
    },
    {
      "type": "paragraph",
      "html": "<strong>Configuration Centralization</strong>: All providers are declared in one place, making it easy to understand the system’s composition."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "The Scraping Pipeline"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1. Category Discovery"
    },
    {
      "type": "paragraph",
      "html": "The journey begins with automatic category discovery. The <code>category</code> package implements an intelligent crawler that finds all product categories from a website's navigation:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func (cd *Service) Discover() error {\n    c := colly.NewCollector(colly.AllowedDomains(cd.domain),\n    ) // Find categories in navigation\n    c.OnHTML(\"nav a[href]\",\n    func(e *colly.HTMLElement) {\n        // Extract and validate category links\n    })\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Smart Link Validation</strong>: The <code>isValidLink()</code> function filters out irrelevant URLs:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Blocks different domains (prevents external crawling)",
        "Excludes common non-product pages (login, registration, admin)",
        "Handles Persian URL encoding (specifically filters form registration links)",
        "Validates URL schemes (removes javascript:, mailto:, tel:)"
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>Persistent Configuration</strong>: Discovered categories are saved to JSON files for offline processing:"
    },
    {
      "type": "code",
      "lang": "text",
      "code": "categories.json\n\n├── name: \"آجیل شب یلدا\"\n\n├── link: \"https://hosseinibrothers.ir/آجیل/...\"\n\n└── domain:\n\"hosseinibrothers.ir\""
    },
    {
      "type": "paragraph",
      "html": "This separation allows the discovery phase to run once, and subsequent runs can use cached categories, significantly improving performance."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2. Product Listing with Concurrent Scraping"
    },
    {
      "type": "paragraph",
      "html": "The <code>product</code> package implements sophisticated concurrent scraping using the Colly framework:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func (s *Service) List() error {\n    c := colly.NewCollector(colly.AllowedDomains(s.web.Domain), colly.Async(true), // Enable native\n    parallelism\n    ) c.Limit(&colly.LimitRule{\n        DomainGlob: \"*\" + s.web.Domain + \"*\", Parallelism: 4, Delay: 500 * time.Millisecond,\n    })\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Concurrency Control</strong>: Rather than serial requests, the system makes 4 concurrent requests with 500ms delays between them. This balances efficiency with server courtesy."
    },
    {
      "type": "paragraph",
      "html": "<strong>Configurable Selectors</strong>: The system uses configuration-driven CSS selectors, allowing it to adapt to different WordPress theme structures:"
    },
    {
      "type": "code",
      "lang": "json",
      "code": "{\n  \"product_list\": {\n    \"item\": \"div.product-item\",\n    \"fields\": {\n      \"title\": {\n        \"selector\": \".product-name\",\n        \"attr\": \"text\"\n      },\n      \"price\": {\n        \"selector\": \".price\",\n        \"attr\": \"text\"\n      },\n      \"link\": {\n        \"selector\": \"a.product-link\",\n        \"attr\": \"href\"\n      },\n      \"image\": {\n        \"selector\": \"img.product-image\",\n        \"attr\": \"src\"\n      }\n    }\n  }\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Pagination Handling</strong>: The crawler automatically follows pagination links:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "c.OnHTML(s.web.Pagination.Next, func(e *colly.HTMLElement) {\n    nextPage := e.Attr(\"href\")\n    e.Request.Visit(nextPage)\n})"
    },
    {
      "type": "paragraph",
      "html": "<strong>Thread-Safe Data Collection</strong>: Products are collected using a mutex to prevent race conditions:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "s.mu.Lock()\nproducts = append(products, p)\ns.mu.Unlock()"
    },
    {
      "type": "paragraph",
      "html": "<strong>Graceful Error Handling</strong>: Instead of failing entirely, the system logs errors and continues:‍"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "if err != nil {\n    s.logger.Error(\"Visit failed, skipping category\", ...)\n    continue\n}\n\nc.Wait()"
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/P84b10Gucwg?feature=oembed"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3. Description: Fetching with Batch Processing"
    },
    {
      "type": "paragraph",
      "html": "The <code>description</code> package implements intelligent batch processing for fetching product descriptions:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func (s *Service) processBatches(web *website.Service, categoryIndex int, validProducts []int) error\n{\n    for batchStart := 0;\n    batchStart < len(validProducts);\n    batchStart += s.batchSize {\n        // Process 4 products at a time semaphore := make(chan struct{\n        }, 4) for _, productIdx := range validProducts[batchStart:batchEnd] {\n            go func(productIndex int, categoryIdx int) {\n                semaphore <- struct{\n                }\n                {\n                }\n                defer func() {\n                    <-semaphore\n                }\n                () // Fetch description\n            }\n            (productIdx, categoryIndex)\n        }\n        wg.Wait() time.Sleep(s.sleepBatch) // Pause between batches\n    }\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Batch Processing Benefits</strong>:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Server-Friendly</strong>: Pauses between batches prevent overwhelming target servers",
        "<strong>Efficient</strong>: Processes multiple products concurrently while maintaining control",
        "<strong>Reliable</strong>: Reduces the risk of being blocked by rate-limit detection",
        "<strong>Observable</strong>: Logs batch progress for monitoring large crawls"
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>Semaphore Pattern</strong>: Uses Go channels as semaphores to limit concurrent goroutines while allowing flexibility in request management."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Data Cleaning and Normalization"
    },
    {
      "type": "paragraph",
      "html": "One of the project’s standout features is its comprehensive data cleaning pipeline, essential for handling Persian e-commerce sites with varied formatting."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "The Cleanup Service"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func Clean(cat *domain.Category) []domain.ProductClean {\n    for _, p := range cat.Products {\n        // Strip HTML tags title := reTags.ReplaceAllString(p.Title, \"\") // Clean price:\n        \"749,000 تومان\" → 749000 priceStr := strings.ReplaceAll(p.Price, \"تومان\", \"\") priceStr =\n        strings.ReplaceAll(priceStr, \",\", \"\") reDigits := regexp.MustCompile(`\\D`) priceStr =\n        reDigits.ReplaceAllString(priceStr, \"\") // Convert Persian digits to Arabic numerals\n        priceStr = convertPersianDigits(priceStr)\n    }\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Price Normalization</strong>:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Removes currency labels (“تومان”)",
        "Strips thousands separators (commas)",
        "Extracts only numeric values",
        "Converts Persian digits (۰-۹) to ASCII digits (0–9)"
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>Result</strong>: Structured data ready for database storage:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type ProductClean struct {\n    Title string\n    Price int64 // 749000 instead of \"749,000 تومان\"\n    Link string\n    Image string\n    Description string\n}"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Testing Strategy"
    },
    {
      "type": "paragraph",
      "html": "The project includes comprehensive tests demonstrating different crawling scenarios:"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Basic Crawling Test"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func TestWebsites(t *testing.T) {\n    webs := website.New(logger)\n    web := webs[0]\n    pr := product.New(logger, &web)\n    if err := pr.Run();\n    err != nil {\n        t.Fatalf(\"failed to run service: %v\", err)\n    }\n    // Validate output\n    cleanCategories := pr.CleanedProducts()\n    if len(cleanCategories) == 0 {\n        t.Fatalf(\"expected at least 1 category, got 0\")\n    }\n}"
    },
    {
      "type": "paragraph",
      "html": "Persian Language Support Test"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func TestProducts(t *testing.T) {\n    web.Categories[0] = domain.Category{\n        Name: \"آجیل شب یلدا\", // Yalda Night Nuts Link: \"https://hosseinibrothers.ir/...\",\n    }\n    // Verify Persian category names are handled correctly\n}"
    },
    {
      "type": "paragraph",
      "html": "End-to-End Description Fetching Test"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func TestDescription(t *testing.T) {\n    // Tests full pipeline: discovery → listing → description → cleanup\n    desc := description.New(logger, &web)\n    w, err := desc.FetchDescriptions(&web)\n    cleanup.Cleans(w.Categories)\n    if w.Categories[1].CleanedProducts[0].Description == \"\" {\n        t.Errorf(\"expected description, got empty string\")\n    }\n}"
    },
    {
      "type": "paragraph",
      "html": "These tests validate the complete pipeline with real data structures, including Persian content handling."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Storage and API Layers"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Dual API Support"
    },
    {
      "type": "paragraph",
      "html": "The system provides both HTTP REST and gRPC endpoints through multiple controllers:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "fx.Provide(\n    productController.NewAdmin,  // Admin REST API\n    productController.NewClient, // Client REST API\n    productController.NewGRPC,   // gRPC API\n)"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Storage Architecture"
    },
    {
      "type": "paragraph",
      "html": "<strong>SQL Database</strong>: Primary storage with migrations managed automatically:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "fx.Provide(\n    sql.InitialDB,\n    migrate.NewRunner,\n    sqlc.New,\n)"
    },
    {
      "type": "paragraph",
      "html": "<strong>Redis Cache</strong>: Fast access layer for frequently requested products:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "fx.Provide(\n    cache.NewClient,\n    cache.NewCacheStore,\n)"
    },
    {
      "type": "paragraph",
      "html": "This dual-layer approach provides both persistence and performance."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Key Design Patterns"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1. Configuration-Driven Crawling"
    },
    {
      "type": "paragraph",
      "html": "Rather than hardcoding CSS selectors, the system loads them from JSON:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "ProductList: domain.ProductListConfig{\n    Item: \"div.product\",\n    Fields: map[string]FieldConfig{\n        \"title\": {Selector: \"h2.name\", Attr: \"text\"},\n        \"price\": {Selector: \".price\", Attr: \"text\"},\n        // ...\n    },\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Benefit</strong>: Adapt to different WordPress themes without code changes."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2. Graceful Degradation"
    },
    {
      "type": "paragraph",
      "html": "When a category fails to scrape, the system logs the error and continues:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "if err := c.Visit(cat.Link); err != nil {\n    s.logger.Error(\"Visit failed, skipping category\", ...)\n    continue\n}"
    },
    {
      "type": "paragraph",
      "html": "This ensures that failure in one category doesn’t prevent scraping others."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Production Considerations"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Rate Limiting and Politeness"
    },
    {
      "type": "paragraph",
      "html": "The system implements multiple courtesy measures:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Configurable Delays</strong>: 500ms to 2s between requests",
        "<strong>Batch Pauses</strong>: Sleeps between batches of products",
        "<strong>Concurrency Limits</strong>: Maximum 4 concurrent requests per domain",
        "<strong>Error Reporting</strong>: Logs blocked or failed requests"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Robustness"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Timeout Management</strong>: 30-second request timeouts prevent hanging",
        "<strong>Context Propagation</strong>: Uses Go’s context for cancellation",
        "<strong>Mutex Protection</strong>: Thread-safe data collection across goroutines",
        "<strong>Migration Support</strong>: Automatic database schema updates"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Observability"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Structured Logging</strong>: All operations logged with context using zap",
        "<strong>Progress Tracking</strong>: Batch-level visibility into crawling progress",
        "<strong>Error Categories</strong>: Distinguishes between network errors, parsing errors, and validation errors"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Future Enhancement Possibilities"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "<strong>Distributed Crawling</strong>: Use message queues to distribute crawling across multiple workers",
        "<strong>ML-Based Selectors</strong>: Train models to identify product elements rather than manual configuration",
        "<strong>Proxy Rotation</strong>: Integrate proxy rotation for large-scale crawls",
        "<strong>Data Versioning</strong>: Track product price history over time",
        "<strong>Webhook Integration</strong>: Notify external systems when products are updated",
        "<strong>GraphQL API</strong>: Add GraphQL support alongside gRPC and REST"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Conclusion"
    },
    {
      "type": "paragraph",
      "html": "The go-wordpress project demonstrates how to build a production-grade web scraping system that balances performance, reliability, and maintainability. Its thoughtful architecture with dependency injection, concurrent batch processing, comprehensive data cleaning, and graceful error handling makes it suitable as a foundation for e-commerce data aggregation platforms."
    },
    {
      "type": "paragraph",
      "html": "The project’s handling of Persian language content, configurable scraping selectors, and sophisticated concurrency control provide templates for anyone building similar systems. Most importantly, it shows that scrapers don’t need to be brittle, chaotic systems — they can be well-structured, testable, and production-ready."
    },
    {
      "type": "paragraph",
      "html": "Whether you’re scraping WordPress sites, aggregating product data, or building a data pipeline, go-wordpress offers proven patterns and a solid starting point for your web data extraction needs."
    }
  ]
}