Building a Safe, Parallel WooCommerce Crawler in Go with Colly

Web crawling often starts simple: fetch a listing page, extract some fields, and move on. The complexity appears when you need to enrich data by visiting detail pages, do it in parallel, and still keep your code race-free, testable, and maintainable.

This article walks through the evolution of a Go-based crawler built with Colly, focusing on the architectural decisions that make it safe and production-ready.

Go-wordpress Project

Problem Statement

The goal was to crawl a WooCommerce website and extract:

Product title
Price
Image
Product URL
Product description from the product page

The challenge was to:

Separate listing crawling from detail crawling
Fetch product pages in parallel
Avoid data races
Normalize the extracted data
Keep the crawler testable

Data Model Evolution

The first change was extending the product model to support enrichment from product pages:

type Product struct {
    Title string
    Price string
    Link string
    Image string
    Description string
}

This makes it explicit that descriptions are not available at listing time and must be fetched later.

Service Architecture

The crawler is implemented as a service with a clear lifecycle:

type Service struct {
    products []Product
    cleanedProducts []Product
    logger *zap.Logger
    mu sync.Mutex
}

Key points:

products holds raw crawl results
cleanedProducts holds normalized output
sync.Mutex protects concurrent writes
Logging is injected (no global state)

Execution Flow

The crawler runs in three deterministic phases:

func (s *Service) Run() error {
    if err := s.ListProducts();
    err != nil {
        return err
    }
    if err := s.FetchDescriptions();
    err != nil {
        return err
    }
    s.cleanUP() return nil
}

This separation ensures:

Listing logic is isolated
Detail crawling can be parallelized independently
Cleanup runs only after all data is available

Phase 1: Listing Crawl

The listing crawler only extracts what is available on category pages:

c.OnHTML("div.product-grid-item", func(e *colly.HTMLElement) {
    p := Product{
        Title: e.ChildText("h3.wd-entities-title a"),
        Price: e.ChildText("span.price"),
        Link:  e.ChildAttr("a.product-image-link", "href"),
        Image: e.ChildAttr("a.product-image-link img", "src"),
    }

    if p.Link != "" {
        products = append(products, p)
    }
})

No concurrency is needed here. The result is a stable slice of products that becomes the input for the next phase.

Phase 2: Parallel Description Fetching

This is where most crawlers break.

Enabling Parallelism Correctly

Colly already provides a safe async engine:

c := colly.NewCollector(
    colly.AllowedDomains("badomjip.com"),
    colly.Async(true),
)

Concurrency is controlled via LimitRule:

c.Limit(&colly.LimitRule{
    DomainGlob:  "badomjip.com",
    Parallelism: 4,
    Delay:       800 * time.Millisecond,
})

No goroutines around Visit() are used — Colly handles scheduling internally.

Passing Context Safely

Each request carries the index of the product it belongs to:

ctx := colly.NewContext()
ctx.Put("index", strconv.Itoa(i))

This allows mapping a product page back to the correct slice element without lookups or maps.

Preventing Data Races

Parallel callbacks mean concurrent writes. This line is not safe without protection:

s.products[idx].Description = desc

The fix is explicit synchronization:

s.mu.Lock()
s.products[idx].Description = desc
s.mu.Unlock()

This guarantees:

No slice corruption
Clean go test -race
Predictable behavior under load

Phase 3: Cleanup and Normalization

The cleanup step converts scraped strings into normalized data:

HTML tags removed
Prices converted to numeric strings
Currency stripped
Whitespace trimmed

cleaned = append(cleaned, Product{
    Title:       title,
    Price:       strconv.Itoa(priceInt),
    Link:        p.Link,
    Image:       p.Image,
    Description: p.Description,
})

This separation ensures that scraping concerns never leak into business logic.

Testing the Crawler

An integration test validates the full pipeline:

func TestListProductsIntegration(t *testing.T) {
    cr := crawler.New(logger) if err := cr.Run();
    err != nil {
        t.Fatalf("failed to run service: %v", err)
    }
    products := cr.CleanedProducts() if len(products) == 0 {
        t.Fatalf("expected at least 1 product")
    }
}

Key benefits:

Tests real HTML parsing
Verifies async completion
Catches selector regressions early

Key Takeaways

Never mix listing and detail crawling
Parallelism must be controlled, not improvised
Async callbacks require explicit synchronization
Context is safer than global maps
Cleanup belongs in its own phase
Integration tests matter for crawlers

Git-commit

Final Result

The crawler is now:

Parallel but race-free
Deterministic
Easy to extend (pagination, retries, specs)
Safe under load
Testable

This pattern scales cleanly to thousands of products and is suitable for production use.

Your Business — On AutoPilot with DDImedia AI Assistant
(Join Our Waitlist)

Visit us at DataDrivenInvestor.com

Join our creator ecosystem here.

DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1

Building a Safe, Parallel WooCommerce Crawler in Go with Colly