Introduction

The go-wordpress project represents a sophisticated approach to building a production-grade web scraping system in Go. Designed to extract, process, and manage product data from WordPress-based e-commerce websites, this project demonstrates best practices in concurrent scraping, data transformation, and enterprise application architecture. Whether you’re aggregating products from multiple sources or building a distributed crawling system, go-wordpress offers valuable insights into handling complex web extraction challenges at scale.

Project Overview

Go-WordPress is a comprehensive product scraping and management system built with Go, featuring concurrent web crawling, data normalization, caching layers, and both gRPC and HTTP APIs. The system is designed to handle Persian language content (with full Unicode support) and implements sophisticated error handling and batch processing strategies.

Git Commit

Key Components:

  • Web scraping with dynamic category discovery
  • Concurrent product listing and description extraction
  • Data cleaning and normalization (especially for Persian prices)
  • Multiple storage backends (SQL database and Redis cache)
  • gRPC and HTTP REST APIs
  • Dependency injection with Uber’s fx framework
  • Comprehensive test coverage

Architecture Overview

Modular Design with Dependency Injection

The project leverages Uber’s fx framework to create a clean, modular architecture. The NewApp() function demonstrates this elegantly:

go
func NewApp() *fx.App {
    return fx.New(fx.Provide(logger.NewLogger, config.NewConfig, sql.InitialDB, //...services and
    controllers), fx.Invoke(// Initialization and lifecycle management),
    )
}

This approach provides several advantages:

Loose Coupling: Each component is defined independently and dependencies are automatically resolved by the container. Services don’t need to know how their dependencies are constructed.

Testability: Components can be easily mocked or substituted for testing. The dependency graph ensures that all dependencies are satisfied before initialization.

Lifecycle Management: The framework handles initialization order and graceful shutdown sequences automatically.

Configuration Centralization: All providers are declared in one place, making it easy to understand the system’s composition.

The Scraping Pipeline

1. Category Discovery

The journey begins with automatic category discovery. The category package implements an intelligent crawler that finds all product categories from a website's navigation:

go
func (cd *Service) Discover() error {
    c := colly.NewCollector(colly.AllowedDomains(cd.domain),
    ) // Find categories in navigation
    c.OnHTML("nav a[href]",
    func(e *colly.HTMLElement) {
        // Extract and validate category links
    })
}

Smart Link Validation: The isValidLink() function filters out irrelevant URLs:

  • Blocks different domains (prevents external crawling)
  • Excludes common non-product pages (login, registration, admin)
  • Handles Persian URL encoding (specifically filters form registration links)
  • Validates URL schemes (removes javascript:, mailto:, tel:)

Persistent Configuration: Discovered categories are saved to JSON files for offline processing:

text
categories.json

├── name: "آجیل شب یلدا"

├── link: "https://hosseinibrothers.ir/آجیل/..."

└── domain:
"hosseinibrothers.ir"

This separation allows the discovery phase to run once, and subsequent runs can use cached categories, significantly improving performance.

2. Product Listing with Concurrent Scraping

The product package implements sophisticated concurrent scraping using the Colly framework:

go
func (s *Service) List() error {
    c := colly.NewCollector(colly.AllowedDomains(s.web.Domain), colly.Async(true), // Enable native
    parallelism
    ) c.Limit(&colly.LimitRule{
        DomainGlob: "*" + s.web.Domain + "*", Parallelism: 4, Delay: 500 * time.Millisecond,
    })
}

Concurrency Control: Rather than serial requests, the system makes 4 concurrent requests with 500ms delays between them. This balances efficiency with server courtesy.

Configurable Selectors: The system uses configuration-driven CSS selectors, allowing it to adapt to different WordPress theme structures:

json
{
  "product_list": {
    "item": "div.product-item",
    "fields": {
      "title": {
        "selector": ".product-name",
        "attr": "text"
      },
      "price": {
        "selector": ".price",
        "attr": "text"
      },
      "link": {
        "selector": "a.product-link",
        "attr": "href"
      },
      "image": {
        "selector": "img.product-image",
        "attr": "src"
      }
    }
  }
}

Pagination Handling: The crawler automatically follows pagination links:

go
c.OnHTML(s.web.Pagination.Next, func(e *colly.HTMLElement) {
    nextPage := e.Attr("href")
    e.Request.Visit(nextPage)
})

Thread-Safe Data Collection: Products are collected using a mutex to prevent race conditions:

go
s.mu.Lock()
products = append(products, p)
s.mu.Unlock()

Graceful Error Handling: Instead of failing entirely, the system logs errors and continues:‍

go
if err != nil {
    s.logger.Error("Visit failed, skipping category", ...)
    continue
}

c.Wait()

3. Description: Fetching with Batch Processing

The description package implements intelligent batch processing for fetching product descriptions:

go
func (s *Service) processBatches(web *website.Service, categoryIndex int, validProducts []int) error
{
    for batchStart := 0;
    batchStart < len(validProducts);
    batchStart += s.batchSize {
        // Process 4 products at a time semaphore := make(chan struct{
        }, 4) for _, productIdx := range validProducts[batchStart:batchEnd] {
            go func(productIndex int, categoryIdx int) {
                semaphore <- struct{
                }
                {
                }
                defer func() {
                    <-semaphore
                }
                () // Fetch description
            }
            (productIdx, categoryIndex)
        }
        wg.Wait() time.Sleep(s.sleepBatch) // Pause between batches
    }
}

Batch Processing Benefits:

  • Server-Friendly: Pauses between batches prevent overwhelming target servers
  • Efficient: Processes multiple products concurrently while maintaining control
  • Reliable: Reduces the risk of being blocked by rate-limit detection
  • Observable: Logs batch progress for monitoring large crawls

Semaphore Pattern: Uses Go channels as semaphores to limit concurrent goroutines while allowing flexibility in request management.

Data Cleaning and Normalization

One of the project’s standout features is its comprehensive data cleaning pipeline, essential for handling Persian e-commerce sites with varied formatting.

The Cleanup Service

go
func Clean(cat *domain.Category) []domain.ProductClean {
    for _, p := range cat.Products {
        // Strip HTML tags title := reTags.ReplaceAllString(p.Title, "") // Clean price:
        "749,000 تومان" → 749000 priceStr := strings.ReplaceAll(p.Price, "تومان", "") priceStr =
        strings.ReplaceAll(priceStr, ",", "") reDigits := regexp.MustCompile(`\D`) priceStr =
        reDigits.ReplaceAllString(priceStr, "") // Convert Persian digits to Arabic numerals
        priceStr = convertPersianDigits(priceStr)
    }
}

Price Normalization:

  • Removes currency labels (“تومان”)
  • Strips thousands separators (commas)
  • Extracts only numeric values
  • Converts Persian digits (۰-۹) to ASCII digits (0–9)

Result: Structured data ready for database storage:

go
type ProductClean struct {
    Title string
    Price int64 // 749000 instead of "749,000 تومان"
    Link string
    Image string
    Description string
}

Testing Strategy

The project includes comprehensive tests demonstrating different crawling scenarios:

Basic Crawling Test

go
func TestWebsites(t *testing.T) {
    webs := website.New(logger)
    web := webs[0]
    pr := product.New(logger, &web)
    if err := pr.Run();
    err != nil {
        t.Fatalf("failed to run service: %v", err)
    }
    // Validate output
    cleanCategories := pr.CleanedProducts()
    if len(cleanCategories) == 0 {
        t.Fatalf("expected at least 1 category, got 0")
    }
}

Persian Language Support Test

go
func TestProducts(t *testing.T) {
    web.Categories[0] = domain.Category{
        Name: "آجیل شب یلدا", // Yalda Night Nuts Link: "https://hosseinibrothers.ir/...",
    }
    // Verify Persian category names are handled correctly
}

End-to-End Description Fetching Test

go
func TestDescription(t *testing.T) {
    // Tests full pipeline: discovery → listing → description → cleanup
    desc := description.New(logger, &web)
    w, err := desc.FetchDescriptions(&web)
    cleanup.Cleans(w.Categories)
    if w.Categories[1].CleanedProducts[0].Description == "" {
        t.Errorf("expected description, got empty string")
    }
}

These tests validate the complete pipeline with real data structures, including Persian content handling.

Storage and API Layers

Dual API Support

The system provides both HTTP REST and gRPC endpoints through multiple controllers:

go
fx.Provide(
    productController.NewAdmin,  // Admin REST API
    productController.NewClient, // Client REST API
    productController.NewGRPC,   // gRPC API
)

Storage Architecture

SQL Database: Primary storage with migrations managed automatically:

go
fx.Provide(
    sql.InitialDB,
    migrate.NewRunner,
    sqlc.New,
)

Redis Cache: Fast access layer for frequently requested products:

go
fx.Provide(
    cache.NewClient,
    cache.NewCacheStore,
)

This dual-layer approach provides both persistence and performance.

Key Design Patterns

1. Configuration-Driven Crawling

Rather than hardcoding CSS selectors, the system loads them from JSON:

go
ProductList: domain.ProductListConfig{
    Item: "div.product",
    Fields: map[string]FieldConfig{
        "title": {Selector: "h2.name", Attr: "text"},
        "price": {Selector: ".price", Attr: "text"},
        // ...
    },
}

Benefit: Adapt to different WordPress themes without code changes.

2. Graceful Degradation

When a category fails to scrape, the system logs the error and continues:

go
if err := c.Visit(cat.Link); err != nil {
    s.logger.Error("Visit failed, skipping category", ...)
    continue
}

This ensures that failure in one category doesn’t prevent scraping others.

Production Considerations

Rate Limiting and Politeness

The system implements multiple courtesy measures:

  • Configurable Delays: 500ms to 2s between requests
  • Batch Pauses: Sleeps between batches of products
  • Concurrency Limits: Maximum 4 concurrent requests per domain
  • Error Reporting: Logs blocked or failed requests

Robustness

  • Timeout Management: 30-second request timeouts prevent hanging
  • Context Propagation: Uses Go’s context for cancellation
  • Mutex Protection: Thread-safe data collection across goroutines
  • Migration Support: Automatic database schema updates

Observability

  • Structured Logging: All operations logged with context using zap
  • Progress Tracking: Batch-level visibility into crawling progress
  • Error Categories: Distinguishes between network errors, parsing errors, and validation errors

Future Enhancement Possibilities

  1. Distributed Crawling: Use message queues to distribute crawling across multiple workers
  2. ML-Based Selectors: Train models to identify product elements rather than manual configuration
  3. Proxy Rotation: Integrate proxy rotation for large-scale crawls
  4. Data Versioning: Track product price history over time
  5. Webhook Integration: Notify external systems when products are updated
  6. GraphQL API: Add GraphQL support alongside gRPC and REST

Conclusion

The go-wordpress project demonstrates how to build a production-grade web scraping system that balances performance, reliability, and maintainability. Its thoughtful architecture with dependency injection, concurrent batch processing, comprehensive data cleaning, and graceful error handling makes it suitable as a foundation for e-commerce data aggregation platforms.

The project’s handling of Persian language content, configurable scraping selectors, and sophisticated concurrency control provide templates for anyone building similar systems. Most importantly, it shows that scrapers don’t need to be brittle, chaotic systems — they can be well-structured, testable, and production-ready.

Whether you’re scraping WordPress sites, aggregating product data, or building a data pipeline, go-wordpress offers proven patterns and a solid starting point for your web data extraction needs.