{
  "slug": "Go-WordPress-Crawler--Advanced-Structural-Improvements-ec460d05c4f0",
  "title": "Go WordPress Crawler: Advanced Structural Improvements",
  "subtitle": "A Deep Dive into Enhanced Web Scraping Architecture and Configuration",
  "excerpt": "A Deep Dive into Enhanced Web Scraping Architecture and Configuration",
  "date": "2026-02-02",
  "tags": [
    "Go",
    "Architecture"
  ],
  "readingTime": "14 min",
  "url": "https://medium.com/@mobinshaterian/go-wordpress-crawler-advanced-structural-improvements-ec460d05c4f0",
  "hero": "https://cdn-images-1.medium.com/max/800/1*WH158q_3JcIFbsg43BM19w.png",
  "content": [
    {
      "type": "paragraph",
      "html": "A Deep Dive into Enhanced Web Scraping Architecture and Configuration"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Introduction"
    },
    {
      "type": "paragraph",
      "html": "The go-wordpress project is a sophisticated web scraping framework written in Go that specializes in extracting product data from WordPress-based e-commerce websites. The recent architectural changes represent a significant evolution in the project’s capabilities, introducing a more robust and flexible data model, improved separation of concerns, and enhanced configuration management. This article provides a comprehensive analysis of these changes, their implications, and their benefits to the development workflow."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Git commit"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*WH158q_3JcIFbsg43BM19w.png",
      "alt": "Go WordPress Crawler: Advanced Structural Improvements",
      "caption": "",
      "width": 1024,
      "height": 1024
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1. Core Data Structures"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1.1 Product and ProductClean Types"
    },
    {
      "type": "paragraph",
      "html": "The foundation of the enhanced data model includes two complementary product structures:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Product</strong>: Raw data directly extracted from HTML. All fields are stored as strings, preserving the original format from the website. This includes the title, price, link, image URL, and description exactly as they appear in the HTML.",
        "<strong>ProductClean</strong>: Normalized and validated data with type safety. The price field is converted from a string to int64, ensuring type consistency and enabling mathematical operations. Other fields maintain their string types, but with cleaned and trimmed values."
      ]
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*dh0vrndPbhutUFNQf4-YMQ.png",
      "alt": "Go WordPress Crawler: Advanced Structural Improvements",
      "caption": "",
      "width": 712,
      "height": 312
    },
    {
      "type": "paragraph",
      "html": "This dual-structure approach provides several benefits:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Data Integrity</strong>: Original raw data is preserved in the Product struct, enabling audit trails and debugging",
        "<strong>Type Safety</strong>: The ProductClean struct ensures prices are numeric, preventing type errors downstream",
        "<strong>Flexibility</strong>: Consumers can choose between raw or cleaned data based on their needs",
        "<strong>Validation</strong>: The cleaning process validates data quality before it reaches production systems"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1.2 Category Structure"
    },
    {
      "type": "paragraph",
      "html": "Categories represent product collections on an e-commerce site. Each category contains metadata and dual product collections that mirror the Product/ProductClean pattern."
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type Category struct {\n    Name string\n    Link string\n    Products []Product // Raw, uncleaned products\n    CleanedProducts []ProductClean // Validated, normalized products\n}"
    },
    {
      "type": "paragraph",
      "html": "The Category structure includes:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Name</strong>: Human-readable category identifier (e.g., “پکیجها” — Packages in Persian)",
        "<strong>Link</strong>: URL to the category listing page",
        "<strong>Products</strong>: Slice of raw Product structs extracted from the website",
        "<strong>CleanedProducts</strong>: Slice of ProductClean structs after validation and normalization"
      ]
    },
    {
      "type": "paragraph",
      "html": "This dual-collection approach enables the system to preserve original data while providing validated, normalized data for consumption."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "1.3 Website Configuration"
    },
    {
      "type": "paragraph",
      "html": "The Website structure serves as the primary configuration container for the crawler. It defines all necessary parameters for scraping a specific website:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type Website struct {\n    Domain string\n    Categories []Category\n    StartURLs []string `json:\"start_urls\"`\n    ProductList ProductListConfig `json:\"product_list\"`\n    Pagination *PaginationConfig `json:\"pagination,omitempty\"`\n    ProductDetail ProductDetailConfig `json:\"product_detail\"`\n}"
    },
    {
      "type": "paragraph",
      "html": "Key components:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Domain</strong>: The target website domain (e.g., “badomjip.com”) used for allowed domain validation",
        "<strong>Categories</strong>: Array of category objects with links to product listing pages",
        "<strong>StartURLs</strong>: Array of initial URLs to begin crawling (optional, categories provide the primary entry points)",
        "<strong>ProductList</strong>: Configuration for extracting products from list pages",
        "<strong>Pagination</strong>: Configuration for handling multi-page results (optional, indicated by pointer type)",
        "<strong>ProductDetail</strong>: Configuration for extracting product descriptions from detail pages"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2. Advanced Configuration System"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2.1 Field Configuration"
    },
    {
      "type": "paragraph",
      "html": "The FieldConfig structure defines how to extract a specific piece of data from HTML:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type FieldConfig struct {\n    Selector string `json:\"selector\"`\n    Attr string `json:\"attr\"`\n}"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Selector</strong>: CSS selector for targeting elements (e.g., “h3.wd-entities-title a”)",
        "<strong>Attr</strong>: Attribute name if extracting an attribute value. An empty string means extract text content"
      ]
    },
    {
      "type": "paragraph",
      "html": "This generic approach decouples the crawler logic from website-specific HTML structures. Whether you need to extract text content or an href attribute, the same FieldConfig structure handles both cases."
    },
    {
      "type": "paragraph",
      "html": "<strong>Example usage in extraction:</strong>"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "// Extract text content.\nTitle: e.ChildText(s.web.ProductList.Fields[\"title\"].Selector)"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "// Extract an attribute value.\nLink: e.ChildAttr(\n    s.web.ProductList.Fields[\"link\"].Selector,\n    s.web.ProductList.Fields[\"link\"].Attr,\n)"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2.2 ProductListConfig"
    },
    {
      "type": "paragraph",
      "html": "Defines how products are extracted from listing pages:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type ProductListConfig struct {\n    Item string `json:\"item\"`\n    Fields map[string]FieldConfig\n    `json:\"fields\"`\n}"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Item</strong>: CSS selector matching each product element on the listing page",
        "<strong>Fields</strong>: Map of field names (title, price, link, image) to their extraction configurations"
      ]
    },
    {
      "type": "paragraph",
      "html": "This map-based approach allows flexible field extraction without modifying code. For example, the configuration can specify that product titles are found at “h3.wd-entities-title a” and prices at “span.price”. The crawler dynamically applies these selectors to each product item."
    },
    {
      "type": "paragraph",
      "html": "<strong>Configuration example:</strong>"
    },
    {
      "type": "code",
      "lang": "json",
      "code": "{\n  \"product_list\": {\n    \"item\": \"div.product-grid-item\",\n    \"fields\": {\n      \"title\": {\n        \"selector\": \"h3.wd-entities-title a\",\n        \"attr\": \"\"\n      },\n      \"price\": {\n        \"selector\": \"span.price\",\n        \"attr\": \"\"\n      },\n      \"link\": {\n        \"selector\": \"a.product-image-link\",\n        \"attr\": \"href\"\n      },\n      \"image\": {\n        \"selector\": \"a.product-image-link img\",\n        \"attr\": \"src\"\n      }\n    }\n  }\n}"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2.3 PaginationConfig"
    },
    {
      "type": "paragraph",
      "html": "Handles multi-page product listings:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type PaginationConfig struct {\n    Next string `json:\"next\"`\n}"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Next</strong>: CSS selector for the “next page” link element"
      ]
    },
    {
      "type": "paragraph",
      "html": "When the crawler encounters a page with products, it looks for a “next” link using this selector. When found, the Gocolly collector automatically visits the next page and continues extraction. This eliminates the need for hardcoded pagination logic and makes the crawler adaptable to different pagination schemes."
    },
    {
      "type": "paragraph",
      "html": "<strong>How it works:</strong>"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "c.OnHTML(s.web.Pagination.Next, func(e *colly.HTMLElement) {\n    nextPage := e.Attr(\"href\")\n    s.logger.Info(\"Visiting next page\", zap.String(\"url\", nextPage))\n    e.Request.Visit(nextPage)\n})"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "2.4 ProductDetailConfig"
    },
    {
      "type": "paragraph",
      "html": "Specifies how to extract detailed product information from individual product pages:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type ProductDetailConfig struct {\n    Description FieldConfig `json:\"description\"`\n}"
    },
    {
      "type": "paragraph",
      "html": "Currently, it defines the Description field configuration. This structure can be extended to include other details like specifications, reviews, availability information, or ratings without modifying the core crawler logic."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3. The Service Layer Architecture"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3.1 Service Structure"
    },
    {
      "type": "paragraph",
      "html": "The Service struct encapsulates the crawler logic, managing state through three fields:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "type Service struct {\n    web *Website\n    logger *zap.Logger\n    mu sync.Mutex\n}"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>web</strong>: Pointer to the Website configuration containing all extraction rules",
        "<strong>logger</strong>: Structured logger (zap) for comprehensive logging",
        "<strong>mu</strong>: Mutex for thread-safe concurrent operations during async description fetching"
      ]
    },
    {
      "type": "paragraph",
      "html": "This design follows Go idioms and enables graceful concurrent scraping of product pages while maintaining data integrity."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3.2 The Run Method: Orchestrating the Pipeline"
    },
    {
      "type": "paragraph",
      "html": "The Run method orchestrates the entire crawling workflow with a clear three-step pipeline:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func (s *Service) Run() error {\n    if err := s.ListProducts();\n    err != nil {\n        s.logger.Error(\"ListProducts failed\", zap.Error(err)) return err\n    }\n    if err := s.FetchDescriptions();\n    err != nil {\n        s.logger.Error(\"FetchDescriptions failed\", zap.Error(err)) return err\n    }\n    s.cleanUP()\n    return nil\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>The workflow:</strong>"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "<strong>ListProducts</strong>: Fetches all products from category listing pages with pagination support",
        "<strong>FetchDescriptions</strong>: Asynchronously retrieves product descriptions from individual product pages",
        "<strong>cleanUP</strong>: Validates and normalizes all extracted data"
      ]
    },
    {
      "type": "paragraph",
      "html": "This separation enables:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Monitoring</strong>: Track which step fails and log detailed errors",
        "<strong>Optimization</strong>: Each step can be optimized independently",
        "<strong>Reusability</strong>: Steps can be called in different orders or combinations",
        "<strong>Testability</strong>: Each step can be tested in isolation"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3.3 ListProducts Method: Extraction with Dynamic Selectors"
    },
    {
      "type": "paragraph",
      "html": "This method iterates through configured categories and creates a Gocolly collector for each:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func (s *Service) ListProducts() error {\n    for index, cat := range s.web.Categories {\n        c := colly.NewCollector(colly.AllowedDomains(s.web.Domain),) products := []Product{\n        }\n        // Use configured selectors from web.ProductList c.OnHTML(s.web.ProductList.Item,\n        func(e *colly.HTMLElement) {\n            p := Product{\n                Title: e.ChildText(s.web.ProductList.Fields[\"title\"].Selector), Price:\n                e.ChildText(s.web.ProductList.Fields[\"price\"].Selector), Link:\n                e.ChildAttr(s.web.ProductList.Fields[\"link\"].Selector,\n                s.web.ProductList.Fields[\"link\"].Attr), Image:\n                e.ChildAttr(s.web.ProductList.Fields[\"image\"].Selector,\n                s.web.ProductList.Fields[\"image\"].Attr),\n            }\n            if p.Link != \"\" {\n                products = append(products, p)\n            }\n            else {\n                s.logger.Debug(\"Product link is not valid\", zap.Any(\"product\", p))\n            }\n        }) // Handle Pagination c.OnHTML(s.web.Pagination.Next,\n        func(e *colly.HTMLElement) {\n            nextPage := e.Attr(\"href\") s.logger.Info(\"Visiting next page\", zap.String(\"url\",\n            nextPage)) e.Request.Visit(nextPage)\n        }) err := c.Visit(cat.Link) if err != nil {\n            return err\n        }\n        s.web.Categories[index].Products = products\n    }\n    return nil\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Key features:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Dynamic selector-based extraction</strong>: Uses FieldConfig to determine where to find each field",
        "<strong>Support for text and attribute extraction</strong>: Handles both content extraction and attribute parsing",
        "<strong>Automatic pagination</strong>: Follows “next” links without manual intervention",
        "<strong>Validation</strong>: Ensures product links exist before storing products",
        "<strong>Detailed logging</strong>: Logs skipped products for debugging"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3.4 FetchDescriptions Method: Concurrent Scraping with Rate Limiting"
    },
    {
      "type": "paragraph",
      "html": "This asynchronous method demonstrates advanced concurrency patterns essential for production web scraping:"
    },
    {
      "type": "paragraph",
      "html": "go"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func (s *Service) FetchDescriptions() error {\n    for index, cat := range s.web.Categories {\n        c := colly.NewCollector(colly.AllowedDomains(s.web.Domain), colly.Async(true), // ENABLE\n        ASYNC) // Control parallelism c.Limit(&colly.LimitRule{\n            DomainGlob: s.web.Domain, Parallelism: 4, // number of concurrent requests Delay: 800 *\n            time.Millisecond,\n        }) // Extract description from product page\n        c.OnHTML(s.web.ProductDetail.Description.Selector,\n        func(e *colly.HTMLElement) {\n            desc := strings.TrimSpace(e.Text) idxAny := e.Request.Ctx.Get(\"index\") idx, _ :=\n            strconv.Atoi(idxAny) if desc == \"\" {\n                s.logger.Warn(\"empty description\", zap.String(\"url\", e.Request.URL.String())) return\n            }\n            s.mu.Lock() s.web.Categories[index].Products[idx].Description = desc s.mu.Unlock()\n        },) for i := range cat.Products {\n            if s.web.Categories[index].Products[i].Link == \"\" {\n                continue\n            }\n            ctx := colly.NewContext() ctx.Put(\"index\", strconv.Itoa(i)) if err := c.Request(\"GET\",\n            s.web.Categories[index].Products[i].Link, nil, ctx, nil,);\n            err != nil {\n                s.logger.Warn(\"Failed to visit product\", zap.String(\"url\",\n                s.web.Categories[index].Products[i].Link), zap.Error(err),)\n            }\n        }\n        c.Wait()\n    }\n    return nil\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Advanced features:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Async(true)</strong>: Enables concurrent requests instead of sequential processing",
        "<strong>LimitRule</strong>: Controls parallelism to 4 concurrent requests with 800ms delay between them",
        "Prevents overwhelming target servers",
        "Respects robots.txt conventions",
        "Reduces the risk of IP blocking",
        "<strong>Context objects</strong>: Pass product indices to callbacks for accurate data mapping without relying on shared state",
        "<strong>Mutex protection</strong>: Ensures thread-safe modification of concurrent slice updates",
        "<strong>Error handling</strong>: Logs failures without stopping the entire process",
        "<strong>Wait()</strong>: Blocks until all async requests complete"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "3.5 Data Cleanup and Normalization"
    },
    {
      "type": "paragraph",
      "html": "The cleanUP method performs a comprehensive data transformation:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func (s *Service) cleanUP() {\n    for index, cat := range s.web.Categories {\n        cleaned := make([]ProductClean, 0, len(cat.Products)) // regex to strip HTML tags reTags :=\n        regexp.MustCompile(`<[^>]*>`) for _, p := range cat.Products {\n            // --- Clean Title --- title := reTags.ReplaceAllString(p.Title, \"\") title =\n            strings.TrimSpace(title) // --- Clean Price --- priceStr := p.Price // remove currency\n            word priceStr = strings.ReplaceAll(priceStr, \"تومان\", \"\") // remove commas and\n            non-digits priceStr = strings.ReplaceAll(priceStr, \",\", \"\") priceStr =\n            strings.TrimSpace(priceStr) // keep only digits reDigits := regexp.MustCompile(`\\D`)\n            priceStr = reDigits.ReplaceAllString(priceStr, \"\") priceStr =\n            convertPersianDigits(priceStr\n            )\n            var priceInt int if priceStr != \"\" {\n                if val, err := strconv.Atoi(priceStr);\n                err == nil {\n                    priceInt = val\n                }\n            }\n            // --- Build cleaned product --- cleaned = append(cleaned, ProductClean{\n                Title: title, Price: int64(priceInt), Link: p.Link, Image: p.Image, Description:\n                p.Description,\n            })\n        }\n        s.web.Categories[index].CleanedProducts = cleaned\n    }\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Comprehensive data transformation:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>HTML Tag Removal</strong>: Uses regex <code>&lt;[^&gt;]*&gt;</code> to strip HTML tags from titles and descriptions, preserving clean text content",
        "<strong>Price Parsing</strong>: A sophisticated multi-step process:",
        "Removes currency text (e.g., “تومان”)",
        "Removes formatting characters (commas, spaces)",
        "Extracts only digits using regex",
        "Converts Persian digits to ASCII",
        "Parses to int64",
        "<strong>Persian Digit Conversion</strong>: Converts Persian numerals (۰-۹) to ASCII digits (0–9)"
      ]
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func convertPersianDigits(str string) string {\n    persianDigits := []string{\n        \"۰\", \"۱\", \"۲\", \"۳\", \"۴\", \"۵\", \"۶\", \"۷\", \"۸\", \"۹\"\n    }\n    for i, digit := range persianDigits {\n        str = strings.ReplaceAll(str, digit, strconv.Itoa(i))\n    }\n    return str\n}"
    },
    {
      "type": "paragraph",
      "html": "This is crucial for handling Iranian e-commerce sites"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Type Conversion</strong>: Converts price strings to int64, providing type safety and enabling numerical operations"
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>Benefits:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Ensures data consistency across different website formats",
        "Enables downstream systems to rely on predictable data types",
        "Handles international number systems transparently",
        "Validates data quality before it reaches production"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "4. Comprehensive Testing Strategy"
    },
    {
      "type": "paragraph",
      "html": "The test suite includes integration tests demonstrating real-world crawler functionality against actual websites:"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "4.1 TestBadomjip"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func TestBadomjip(t *testing.T) {\n    logger, err := zap.NewDevelopment()\n    if err != nil {\n        t.Fatalf(\"failed to create logger: %v\", err)\n    }\n    defer logger.Sync() web := crawler.NewWebsite(\"badomjip.json\")\n    cr := crawler.New(logger, web)\n    if err := cr.Run();\n    err != nil {\n        t.Fatalf(\"failed to run service: %v\", err)\n    }\n    categories := cr.CleanedProducts()\n    if len(categories) == 0 {\n        t.Fatalf(\"expected at least 1 category, got 0\")\n    }\n    // Check first category\n    cleanedProducts := categories[0].CleanedProducts\n    if len(cleanedProducts) == 0 {\n        t.Fatalf(\"expected at least 1 product, got 0\")\n    }\n    // Basic sanity checks on first product\n    p := cleanedProducts[0]\n    if p.Title == \"\" {\n        t.Errorf(\"product title should not be empty\")\n    }\n    if p.Price == 0 {\n        t.Errorf(\"product price should not be 0\")\n    }\n    if p.Link == \"\" {\n        t.Errorf(\"product link should not be empty\")\n    }\n    if p.Image == \"\" {\n        t.Errorf(\"product image should not be empty\")\n    }\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Test coverage:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Tests the crawler against badomjip.com, a Persian e-commerce site",
        "Validates that the crawler successfully loads the configuration",
        "Confirms the complete pipeline runs without errors",
        "Produces cleaned products with non-zero prices and populated fields",
        "Confirms the crawler correctly handles Persian text and currency formats"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "4.2 TestHosseiniBrothers"
    },
    {
      "type": "paragraph",
      "html": "Test the crawler against Hosseinibrothers. ir, another Persian e-commerce platform with a different HTML structure. This test ensures the configuration-driven approach handles diverse website layouts without code changes. It validates the same assertions as TestBadomjip, confirming crawler robustness across different target sites."
    },
    {
      "type": "paragraph",
      "html": "<strong>Why two tests?</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Structural diversity</strong>: Different websites have different HTML structures",
        "<strong>Configuration validation</strong>: Demonstrates that configurations are portable",
        "<strong>Regression prevention</strong>: Ensures updates don’t break existing functionality",
        "<strong>Real-world proof</strong>: Actual tests against live websites provide confidence"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "5. Configuration Deep Dive"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "5.1 Badomjip Configuration Example"
    },
    {
      "type": "paragraph",
      "html": "The badomjip.json configuration demonstrates selector-based extraction for a WordPress WooCommerce store:"
    },
    {
      "type": "code",
      "lang": "json",
      "code": "{\n  \"domain\": \"badomjip.com\",\n  \"start_urls\": [\n    \"https://badomjip.com/product-category/%d9%be%da%a9%db%8c%d8%ac%d9%87%d8%a7/\"\n  ],\n  \"categories\": [\n    {\n      \"name\": \"پکیجها\",\n      \"link\": \"https://badomjip.com/product-category/%d9%be%da%a9%db%8c%d8%ac%d9%87%d8%a7/\"\n    }\n  ],\n  \"product_list\": {\n    \"item\": \"div.product-grid-item\",\n    \"fields\": {\n      \"title\": {\n        \"selector\": \"h3.wd-entities-title a\",\n        \"attr\": \"\"\n      },\n      \"price\": {\n        \"selector\": \"span.price\",\n        \"attr\": \"\"\n      },\n      \"link\": {\n        \"selector\": \"a.product-image-link\",\n        \"attr\": \"href\"\n      },\n      \"image\": {\n        \"selector\": \"a.product-image-link img\",\n        \"attr\": \"src\"\n      }\n    }\n  },\n  \"pagination\": {\n    \"next\": \"a.next.page-numbers\"\n  },\n  \"product_detail\": {\n    \"description\": {\n      \"selector\": \"div.woocommerce-product-details__short-description, div.woocommerce-Tabs-panel--description\"\n    }\n  }\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Configuration breakdown:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Product Item Selector</strong> (<code>div.product-grid-item</code>): Targets individual product cards",
        "<strong>Title Extraction</strong> (<code>h3.wd-entities-title a</code>): Retrieved as text content",
        "<strong>Price Extraction</strong> (<code>span.price</code>): Obtained as text content, later parsed to an integer",
        "<strong>Link Extraction</strong> (<code>a.product-image-link</code>, attr: <code>href</code>): Retrieved from href attribute",
        "<strong>Image Extraction</strong> (<code>a.product-image-link img</code>, attr: <code>src</code>): Retrieved from the src attribute",
        "<strong>Pagination</strong> (<code>a.next.page-numbers</code>): Automatically follows next page links",
        "<strong>Description</strong> (Multiple selectors): Handles both short and detailed descriptions"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "5.2 Hosseini Brothers Configuration Example"
    },
    {
      "type": "paragraph",
      "html": "This configuration handles a completely different HTML structure:"
    },
    {
      "type": "code",
      "lang": "json",
      "code": "{\n  \"domain\": \"hosseinibrothers.ir\",\n  \"start_urls\": [\n    \"https://hosseinibrothers.ir/%D8%AE%D8%B4%DA%A9%D8%A8%D8%A7%D8%B1/\"\n  ],\n  \"categories\": [\n    {\n      \"name\": \"خشکبار\",\n      \"link\": \"https://hosseinibrothers.ir/%D8%AE%D8%B4%DA%A9%D8%A8%D8%A7%D8%B1/\"\n    },\n    {\n      \"name\": \"زعفران\",\n      \"link\": \"https://hosseinibrothers.ir/%D8%B2%D8%B9%D9%81%D8%B1%D8%A7%D9%86/\"\n    },\n    {\n      \"name\": \"تنقلات\",\n      \"link\": \"https://hosseinibrothers.ir/%D8%AA%D9%86%D9%82%D9%84%D8%A7%D8%AA/\"\n    }\n  ],\n  \"product_list\": {\n    \"item\": \"div.js-product-miniature\",\n    \"fields\": {\n      \"title\": {\n        \"selector\": \"a.stsb_mini_product_name\",\n        \"attr\": \"\"\n      },\n      \"price\": {\n        \"selector\": \"div.stsb_pm_price\",\n        \"attr\": \"\"\n      },\n      \"link\": {\n        \"selector\": \"a.stsb_mini_product_name\",\n        \"attr\": \"href\"\n      },\n      \"image\": {\n        \"selector\": \"img.stsb_pm_image\",\n        \"attr\": \"src\"\n      }\n    }\n  },\n  \"pagination\": {\n    \"next\": \"a[rel='next']\"\n  },\n  \"product_detail\": {\n    \"description\": {\n      \"selector\": \"#description, div.elementor-widget-theme-post-content\"\n    }\n  }\n}"
    },
    {
      "type": "paragraph",
      "html": "<strong>Configuration breakdown:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Product Item Selector</strong> (<code>div.js-product-miniature</code>): Different class names",
        "<strong>Title and Link</strong> (<code>a.stsb_mini_product_name</code>): Same selector, different extraction methods",
        "<strong>Price</strong> (<code>div.stsb_pm_price</code>): Different structure than WooCommerce",
        "<strong>Image</strong> (<code>img.stsb_pm_image</code>): Custom image class",
        "<strong>Multiple Categories</strong>: Demonstrates support for category-specific links",
        "<strong>Pagination</strong> (<code>a[rel='next']</code>): Uses attribute selector instead of class-based selector",
        "<strong>Description</strong>: Handles multiple selector alternatives with comma separation"
      ]
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*T02oXtliCC-lfghm9llJFg.png",
      "alt": "Go WordPress Crawler: Advanced Structural Improvements",
      "caption": "",
      "width": 565,
      "height": 251
    },
    {
      "type": "paragraph",
      "html": "Despite these differences, both sites are fully supported through configuration alone, without code changes."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "6. Benefits and Architectural Improvements"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "6.1 Separation of Concerns"
    },
    {
      "type": "paragraph",
      "html": "Raw data extraction, validation, and cleaning are handled in separate methods, making code easier to test and maintain:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>ListProducts()</strong>: Responsible only for HTML parsing and product extraction",
        "<strong>FetchDescriptions()</strong>: Handles async description fetching",
        "<strong>cleanUP()</strong>: Manages data validation and normalization"
      ]
    },
    {
      "type": "paragraph",
      "html": "Each method has a single responsibility, following the Single Responsibility Principle. This makes the codebase:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Testable</strong>: Each step can be tested independently",
        "<strong>Debuggable</strong>: Issues can be isolated to specific methods",
        "<strong>Maintainable</strong>: Changes to one step don’t affect others",
        "<strong>Reusable</strong>: Methods can be composed in different ways"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "6.2 Configuration-Driven Flexibility"
    },
    {
      "type": "paragraph",
      "html": "New websites can be added with JSON configuration files only, without modifying the core crawler code:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>No code compilation needed</strong>: Administrators can add new websites by creating JSON files",
        "<strong>Dynamic selector support</strong>: Each website can have completely different HTML structures",
        "<strong>Version control friendly</strong>: Configuration files can be versioned separately",
        "<strong>A/B testing ready</strong>: Different configurations can be tested without deployment",
        "<strong>Dramatically reduced effort</strong>: Supporting a new site takes minutes instead of days"
      ]
    },
    {
      "type": "paragraph",
      "html": "This dramatically reduces the effort needed to support new e-commerce platforms."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "6.3 Type Safety"
    },
    {
      "type": "paragraph",
      "html": "The ProductClean struct with proper types (int64 for price) enables compile-time type checking:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Prevents runtime errors</strong>: Type mismatches caught at compile time",
        "<strong>Enables math operations</strong>: int64 prices can be summed, averaged, compared",
        "<strong>Database compatibility</strong>: Proper types map naturally to database schemas",
        "<strong>API contracts</strong>: Clear type definitions for REST/gRPC endpoints",
        "<strong>Reduces bugs</strong>: No more string-to-number conversion errors at runtime"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "6.4 Localization Support"
    },
    {
      "type": "paragraph",
      "html": "The Persian digit conversion and handling of RTL scripts demonstrates built-in support for international e-commerce platforms:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "func convertPersianDigits(str string) string {\n    persianDigits := []string{\n        \"۰\", \"۱\", \"۲\", \"۳\", \"۴\", \"۵\", \"۶\", \"۷\", \"۸\", \"۹\"\n    }\n    for i, digit := range persianDigits {\n        str = strings.ReplaceAll(str, digit, strconv.Itoa(i))\n    }\n    return str\n}"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Multi-language support</strong>: Handles Persian, Arabic, and other number systems",
        "<strong>Currency flexibility</strong>: Removes currency text before parsing (تومان, etc.)",
        "<strong>No additional configuration</strong>: Localization is built into the data cleaning",
        "<strong>Extensible</strong>: Easy to add support for additional numeral systems"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "6.5 Concurrency and Performance"
    },
    {
      "type": "paragraph",
      "html": "The implementation leverages Gocolly’s async capabilities with controlled parallelism:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "c.Limit(&colly.LimitRule{\n    DomainGlob:  s.web.Domain,\n    Parallelism: 4,                      // 4 concurrent requests\n    Delay:       800 * time.Millisecond, // 800ms between requests\n})"
    },
    {
      "type": "paragraph",
      "html": "<strong>Benefits:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Efficiency</strong>: Multiple product pages fetched simultaneously",
        "<strong>Resource control</strong>: Limits prevent server overload and connection exhaustion",
        "<strong>Ethical scraping</strong>: Delays between requests respect server limits",
        "<strong>Scalability</strong>: Same code handles 10 products or 10,000",
        "<strong>Production-ready</strong>: Rate limiting prevents IP blocking"
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>Performance characteristics:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Sequential listing extraction: Fast, single-threaded",
        "Parallel description fetching: 4 concurrent requests, 800ms delay = efficient crawling",
        "Total crawl time reduced from (N * 800ms) to (ceil(N/4) * 800ms)"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "6.6 Structured Logging"
    },
    {
      "type": "paragraph",
      "html": "Integration with Zap Logger provides structured, efficient logging:"
    },
    {
      "type": "code",
      "lang": "go",
      "code": "s.logger.Info(\n    \"Visiting next page\",\n    zap.String(\"url\", nextPage),\n)\n\ns.logger.Debug(\n    \"Product link is not valid\",\n    zap.Any(\"product\", p),\n)\n\ns.logger.Warn(\n    \"Failed to visit product\",\n    zap.String(\"url\", url),\n    zap.Error(err),\n)"
    },
    {
      "type": "paragraph",
      "html": "<strong>Features:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Structured fields</strong>: Each log entry contains typed fields (not just strings)",
        "<strong>Log levels</strong>: DEBUG, INFO, WARN, ERROR for filtering",
        "<strong>Performance</strong>: Allocates minimal memory for log entries",
        "<strong>Monitoring</strong>: Easy to parse logs programmatically",
        "<strong>Debugging</strong>: Context-rich logs help troubleshoot issues",
        "<strong>Compliance</strong>: Audit trails for production systems"
      ]
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/VDlJpx83O2w?feature=oembed"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "7. Extensibility and Future Improvements"
    },
    {
      "type": "paragraph",
      "html": "The architecture supports numerous extensions without breaking changes:"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "7.1 Potential Enhancements"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "<strong>Additional Product Fields</strong>: The ProductDetailConfig can be extended to extract specifications, ratings, stock levels, etc.",
        "<strong>Request Customization</strong>: Add headers configuration for sites requiring authentication or a custom User-Agent",
        "<strong>JavaScript Rendering</strong>: Support for JavaScript-rendered content via Chrome DevTools Protocol",
        "<strong>Proxy Support</strong>: Configuration-based proxy rotation for large-scale crawling",
        "<strong>Error Recovery</strong>: Retry logic with exponential backoff for transient failures",
        "<strong>Data Export</strong>: Built-in formats for CSV, JSON, and database export",
        "<strong>Scheduling</strong>: Background job scheduling for periodic crawls"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "7.2 Architecture Strengths"
    },
    {
      "type": "paragraph",
      "html": "The design naturally supports these extensions because:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "<strong>Configuration is a first-class citizen</strong>: New options can be added to JSON configs",
        "<strong>Plugin pattern</strong>: New extraction types can be registered dynamically",
        "<strong>Clean interfaces</strong>: Core crawler logic is decoupled from infrastructure",
        "<strong>Go’s simplicity</strong>: Easy to add features without massive refactoring"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "8. Production Readiness Checklist"
    },
    {
      "type": "paragraph",
      "html": "The current implementation demonstrates production-ready practices:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "✅ <strong>Error handling</strong>: All operations check for errors",
        "✅ <strong>Logging</strong>: Comprehensive structured logging for debugging",
        "✅ <strong>Rate limiting</strong>: Respects target server constraints",
        "✅ <strong>Concurrency safety</strong>: Mutexes protect shared state",
        "✅ <strong>Configuration validation</strong>: Structures validated on load",
        "✅ <strong>Testing</strong>: Integration tests against real websites",
        "✅ <strong>Data validation</strong>: Cleanup ensures data quality",
        "✅ <strong>Type safety</strong>: Proper types prevent runtime errors"
      ]
    },
    {
      "type": "paragraph",
      "html": "<strong>What’s needed for production:</strong>"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Database integration for storing products",
        "Metrics collection (Prometheus/Grafana)",
        "Health checks and alerting",
        "Backup and recovery procedures",
        "Rate limiting per category/domain",
        "Cache layer for frequently accessed data"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "9. Conclusion"
    },
    {
      "type": "paragraph",
      "html": "The go-wordpress project’s recent architectural changes represent a mature, production-ready approach to web scraping. By combining configuration-driven design with clean separation of concerns, comprehensive data validation, and robust concurrency handling, the framework achieves flexibility without sacrificing code quality."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Key Achievements"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "<strong>The dual data model</strong> (Product/ProductClean) ensures data integrity while maintaining flexibility",
        "<strong>Configuration-driven architecture</strong> enables support for diverse websites without code changes",
        "<strong>Structured concurrency</strong> with rate limiting provides ethical, efficient scraping",
        "<strong>Comprehensive data cleaning</strong> handles international formats and validations",
        "<strong>Architectural clarity</strong> makes the codebase maintainable and extensible"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Why This Matters"
    },
    {
      "type": "paragraph",
      "html": "Whether supporting Persian e-commerce platforms or any other WordPress-based site, the project demonstrates how thoughtful architectural decisions enable scalable, maintainable web scraping infrastructure. The framework isn’t just a one-off solution — it’s a platform for building production-grade data pipelines."
    },
    {
      "type": "paragraph",
      "html": "The separation of concerns, configuration-driven design, and proper handling of concurrency create a solid foundation that can grow with your needs, whether you’re crawling 100 products or 100,000."
    }
  ]
}