Introduction

In this project, we explored how to design and implement a Go‑based microservice that crawls products from a WooCommerce WordPress site. The journey started with a simple Colly scraper and evolved into a clean, testable service integrated into a larger application framework using Uber’s Fx dependency injection.

The goal was to reliably fetch product data, normalize it (clean titles and prices), and expose it for downstream services such as storage, analytics, or APIs.

Go-Wordpress

Application Architecture

At the top level, the application is composed with Fx:

go
func NewApp() *fx.App {
    return fx.New(fx.Provide(logger.NewLogger, config.NewConfig, sql.InitialDB, health.New,
    server.NewGinEngine, server.CreateHTTPServer, server.CreateGRPCServer, migrate.NewRunner,
    sqlc.New, cache.NewClient, cache.NewCacheStore, productController.NewAdmin,
    productController.NewClient, productController.NewGRPC, productService.New, crawler.New,
    dispatcher.New, poller.New,), fx.Invoke(dispatcher.RegisterServices,
    dispatcher.RegisterLifecycle, poller.RegisterLifecycle, server.RegisterRoutes,
    server.StartHTTPServer, server.StartGRPCServer, migrate.RunMigrations,
    logger.RegisterLoggerLifecycle, server.GRPCLifeCycle,),
    )
}

This composition wires together logging, configuration, SQL storage, cache, controllers, services, and our new crawler service.

badomjip

The Crawler Service

The crawler is responsible for scraping product data from the WooCommerce category page:

go
type Product struct {
    Title string
    Price string
    Link string
    Image string
}
type Service struct {
    products []Product
    cleanedProducts []Product
}

Scraping Logic

Using Colly, the service visits the target page and extracts product information:

go
func (s *Service) ListProducts() error {
    c := colly.NewCollector(colly.AllowedDomains("badomjip.com"),
    )
    products := []Product{
    }
    c.OnHTML("div.product-grid-item", func(e *colly.HTMLElement) {
        p := Product{
            Title: e.ChildText("h3.wd-entities-title a"), Price: e.ChildText("span.price"), Link:
            e.ChildAttr("a.product-image-link", "href"), Image:
            e.ChildAttr("a.product-image-link img", "src"),
        }
        products = append(products, p)
    })
    err := c.Visit("https://badomjip.com/product-category/%d9%be%da%a9%db%8c%d8%ac%d9%87%d8%a7/")
    if err != nil {
        return err
    }
    s.products = products
    s.cleanUP()
    return nil
}

Data Normalization

Raw HTML often contains extra markup and localized currency strings. The cleanUP method ensures titles and prices are normalized:

  • Title: strip HTML tags and trim whitespace
  • Price: remove commas, currency symbols, and convert to an integer
go
func (s *Service) cleanUP() {
    cleaned := make([]Product, 0, len(s.products))
    reTags := regexp.MustCompile(`<[^>]*>`)
    for _, p := range s.products {
        title := reTags.ReplaceAllString(p.Title, "") title = strings.TrimSpace(title) priceStr :=
        strings.ReplaceAll(p.Price, "تومان", "") priceStr = strings.ReplaceAll(priceStr, ",", "")
        priceStr = strings.TrimSpace(priceStr) reDigits := regexp.MustCompile(`\D`) priceStr =
        reDigits.ReplaceAllString(priceStr, ""
        )
        var priceInt int if priceStr != "" {
            if val, err := strconv.Atoi(priceStr);
            err == nil {
                priceInt = val
            }
        }
        cleaned = append(cleaned, Product{
            Title: title, Price: strconv.Itoa(priceInt), Link: p.Link, Image: p.Image,
        })
    }
    s.cleanedProducts = cleaned
}

Integration Testing

To validate the crawler against the live site, we wrote an integration test:

go
func TestListProductsIntegration(t *testing.T) {
    svc := crawler.New()
    err := svc.ListProducts()
    if err != nil {
        t.Fatalf("ListProducts failed: %v", err)
    }
    products := svc.CleanedProducts()
    if len(products) == 0 {
        t.Fatalf("expected at least 1 product, got 0")
    }
    p := products[0]
    if p.Title == "" {
        t.Errorf("product title should not be empty")
    }
    if p.Price == "" {
        t.Errorf("product price should not be empty")
    }
    if p.Link == "" {
        t.Errorf("product link should not be empty")
    }
    if p.Image == "" {
        t.Errorf("product image should not be empty")
    }
}

This test ensures that:

  • The crawler successfully fetches products from the live WooCommerce page
  • Cleaned products contain non‑empty fields
  • Price normalization works as expected

Lessons Learned

  1. Colly selectors must match the theme HTML: WooCommerce themes vary, so we adapted selectors to div.product-grid-item instead of the default ul.products li.product.
  2. Normalization is essential: Prices often include commas, non‑breaking spaces, and localized currency symbols. Cleaning ensures consistent numeric values.
  3. Integration tests provide confidence: By hitting the real page, we validated that our crawler works end‑to‑end.
  4. Fx integration makes services composable: The crawler plugs into the larger application seamlessly, benefiting from DI and lifecycle management.

Git Commits

Conclusion

Through iterative refinement, we built a robust Go service that crawls WooCommerce products, cleans the data, and integrates into a modern microservice architecture. The combination of Colly for scraping, Fx for dependency injection, and Go’s testing framework for integration tests provides a solid foundation for extending this service — whether storing products in SQL/ClickHouse, exposing them via APIs, or scheduling periodic crawls with a poller.

This project demonstrates how backend engineering principles — modularity, testability, and clean architecture — apply even to something as seemingly simple as a web scraper.