{
"slug": "Building-a-Safe--Parallel-WooCommerce-Crawler-in-Go-with-Colly-8786353b4b16",
"title": "Building a Safe, Parallel WooCommerce Crawler in Go with Colly",
"subtitle": "Web crawling often starts simple: fetch a listing page, extract some fields, and move on. The complexity appears when you need to enrich…",
"excerpt": "Web crawling often starts simple: fetch a listing page, extract some fields, and move on. The complexity appears when you need to enrich…",
"date": "2026-01-07",
"tags": [
"Go",
"Machine Learning"
],
"readingTime": "3 min",
"url": "https://medium.com/@mobinshaterian/building-a-safe-parallel-woocommerce-crawler-in-go-with-colly-8786353b4b16",
"hero": "https://cdn-images-1.medium.com/max/800/1*LOSYb3H5JRoDenSvFH5p3Q.png",
"content": [
{
"type": "heading",
"level": 2,
"text": "Building a Safe, Parallel WooCommerce Crawler in Go with Colly"
},
{
"type": "paragraph",
"html": "Web crawling often starts simple: fetch a listing page, extract some fields, and move on. The complexity appears when you need to enrich data by visiting detail pages, do it in parallel, and still keep your code race-free, testable, and maintainable."
},
{
"type": "paragraph",
"html": "This article walks through the evolution of a Go-based crawler built with Colly, focusing on the architectural decisions that make it safe and production-ready."
},
{
"type": "heading",
"level": 2,
"text": "Go-wordpress Project"
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*LOSYb3H5JRoDenSvFH5p3Q.png",
"alt": "Building a Safe, Parallel WooCommerce Crawler in Go with Colly",
"caption": "",
"width": 1024,
"height": 1536
},
{
"type": "heading",
"level": 2,
"text": "Problem Statement"
},
{
"type": "paragraph",
"html": "The goal was to crawl a WooCommerce website and extract:"
},
{
"type": "list",
"ordered": false,
"items": [
"Product title",
"Price",
"Image",
"Product URL",
"Product description from the product page"
]
},
{
"type": "paragraph",
"html": "The challenge was to:"
},
{
"type": "list",
"ordered": true,
"items": [
"Separate listing crawling from detail crawling",
"Fetch product pages in parallel",
"Avoid data races",
"Normalize the extracted data",
"Keep the crawler testable"
]
},
{
"type": "heading",
"level": 2,
"text": "Data Model Evolution"
},
{
"type": "paragraph",
"html": "The first change was extending the product model to support enrichment from product pages:"
},
{
"type": "code",
"lang": "go",
"code": "type Product struct {\n Title string\n Price string\n Link string\n Image string\n Description string\n}"
},
{
"type": "paragraph",
"html": "This makes it explicit that descriptions are not available at listing time and must be fetched later."
},
{
"type": "heading",
"level": 2,
"text": "Service Architecture"
},
{
"type": "paragraph",
"html": "The crawler is implemented as a service with a clear lifecycle:"
},
{
"type": "code",
"lang": "go",
"code": "type Service struct {\n products []Product\n cleanedProducts []Product\n logger *zap.Logger\n mu sync.Mutex\n}"
},
{
"type": "paragraph",
"html": "Key points:"
},
{
"type": "list",
"ordered": false,
"items": [
"products holds raw crawl results",
"cleanedProducts holds normalized output",
"sync.Mutex protects concurrent writes",
"Logging is injected (no global state)"
]
},
{
"type": "heading",
"level": 2,
"text": "Execution Flow"
},
{
"type": "paragraph",
"html": "The crawler runs in three deterministic phases:"
},
{
"type": "code",
"lang": "go",
"code": "func (s *Service) Run() error {\n if err := s.ListProducts();\n err != nil {\n return err\n }\n if err := s.FetchDescriptions();\n err != nil {\n return err\n }\n s.cleanUP() return nil\n}"
},
{
"type": "paragraph",
"html": "This separation ensures:"
},
{
"type": "list",
"ordered": false,
"items": [
"Listing logic is isolated",
"Detail crawling can be parallelized independently",
"Cleanup runs only after all data is available"
]
},
{
"type": "heading",
"level": 2,
"text": "Phase 1: Listing Crawl"
},
{
"type": "paragraph",
"html": "The listing crawler only extracts what is available on category pages:"
},
{
"type": "code",
"lang": "go",
"code": "c.OnHTML(\"div.product-grid-item\", func(e *colly.HTMLElement) {\n p := Product{\n Title: e.ChildText(\"h3.wd-entities-title a\"),\n Price: e.ChildText(\"span.price\"),\n Link: e.ChildAttr(\"a.product-image-link\", \"href\"),\n Image: e.ChildAttr(\"a.product-image-link img\", \"src\"),\n }\n\n if p.Link != \"\" {\n products = append(products, p)\n }\n})"
},
{
"type": "paragraph",
"html": "No concurrency is needed here. The result is a stable slice of products that becomes the input for the next phase."
},
{
"type": "heading",
"level": 2,
"text": "Phase 2: Parallel Description Fetching"
},
{
"type": "paragraph",
"html": "This is where most crawlers break."
},
{
"type": "heading",
"level": 3,
"text": "Enabling Parallelism Correctly"
},
{
"type": "paragraph",
"html": "Colly already provides a safe async engine:"
},
{
"type": "code",
"lang": "go",
"code": "c := colly.NewCollector(\n colly.AllowedDomains(\"badomjip.com\"),\n colly.Async(true),\n)"
},
{
"type": "paragraph",
"html": "Concurrency is controlled via LimitRule:"
},
{
"type": "code",
"lang": "go",
"code": "c.Limit(&colly.LimitRule{\n DomainGlob: \"badomjip.com\",\n Parallelism: 4,\n Delay: 800 * time.Millisecond,\n})"
},
{
"type": "paragraph",
"html": "No goroutines around Visit() are used — Colly handles scheduling internally."
},
{
"type": "heading",
"level": 2,
"text": "Passing Context Safely"
},
{
"type": "paragraph",
"html": "Each request carries the index of the product it belongs to:"
},
{
"type": "code",
"lang": "go",
"code": "ctx := colly.NewContext()\nctx.Put(\"index\", strconv.Itoa(i))"
},
{
"type": "paragraph",
"html": "This allows mapping a product page back to the correct slice element without lookups or maps."
},
{
"type": "heading",
"level": 2,
"text": "Preventing Data Races"
},
{
"type": "paragraph",
"html": "Parallel callbacks mean concurrent writes. This line is not safe without protection:"
},
{
"type": "code",
"lang": "go",
"code": "s.products[idx].Description = desc"
},
{
"type": "paragraph",
"html": "The fix is explicit synchronization:"
},
{
"type": "code",
"lang": "go",
"code": "s.mu.Lock()\ns.products[idx].Description = desc\ns.mu.Unlock()"
},
{
"type": "paragraph",
"html": "This guarantees:"
},
{
"type": "list",
"ordered": false,
"items": [
"No slice corruption",
"Clean go test -race",
"Predictable behavior under load"
]
},
{
"type": "heading",
"level": 2,
"text": "Phase 3: Cleanup and Normalization"
},
{
"type": "paragraph",
"html": "The cleanup step converts scraped strings into normalized data:"
},
{
"type": "list",
"ordered": false,
"items": [
"HTML tags removed",
"Prices converted to numeric strings",
"Currency stripped",
"Whitespace trimmed"
]
},
{
"type": "code",
"lang": "go",
"code": "cleaned = append(cleaned, Product{\n Title: title,\n Price: strconv.Itoa(priceInt),\n Link: p.Link,\n Image: p.Image,\n Description: p.Description,\n})"
},
{
"type": "paragraph",
"html": "This separation ensures that scraping concerns never leak into business logic."
},
{
"type": "heading",
"level": 2,
"text": "Testing the Crawler"
},
{
"type": "paragraph",
"html": "An integration test validates the full pipeline:"
},
{
"type": "code",
"lang": "go",
"code": "func TestListProductsIntegration(t *testing.T) {\n cr := crawler.New(logger) if err := cr.Run();\n err != nil {\n t.Fatalf(\"failed to run service: %v\", err)\n }\n products := cr.CleanedProducts() if len(products) == 0 {\n t.Fatalf(\"expected at least 1 product\")\n }\n}"
},
{
"type": "paragraph",
"html": "Key benefits:"
},
{
"type": "list",
"ordered": false,
"items": [
"Tests real HTML parsing",
"Verifies async completion",
"Catches selector regressions early"
]
},
{
"type": "heading",
"level": 2,
"text": "Key Takeaways"
},
{
"type": "list",
"ordered": true,
"items": [
"Never mix listing and detail crawling",
"Parallelism must be controlled, not improvised",
"Async callbacks require explicit synchronization",
"Context is safer than global maps",
"Cleanup belongs in its own phase",
"Integration tests matter for crawlers"
]
},
{
"type": "heading",
"level": 2,
"text": "Git-commit"
},
{
"type": "heading",
"level": 2,
"text": "Final Result"
},
{
"type": "paragraph",
"html": "The crawler is now:"
},
{
"type": "list",
"ordered": false,
"items": [
"Parallel but race-free",
"Deterministic",
"Easy to extend (pagination, retries, specs)",
"Safe under load",
"Testable"
]
},
{
"type": "paragraph",
"html": "This pattern scales cleanly to thousands of products and is suitable for production use."
},
{
"type": "paragraph",
"html": "Your Business — On AutoPilot with DDImedia AI Assistant
(Join Our Waitlist)"
},
{
"type": "paragraph",
"html": "Visit us at DataDrivenInvestor.com"
},
{
"type": "paragraph",
"html": "Join our creator ecosystem here."
},
{
"type": "paragraph",
"html": "DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1"
},
{
"type": "paragraph",
"html": "Follow us on LinkedIn, Twitter, YouTube, and Facebook."
}
]
}