“The” Problem: How Elasticsearch’s Stop Word Filtering Broke Our Product Search🤔
Have you ever needed to search for the term “the” in Elasticsearch, only to find it filtered out due to stopword removal? We encountered this issue with one of our products — its official name is literally “the.” Unfortunately, Elasticsearch’s default analyzer automatically removes common stop words, such as “the,” making the product unsearchable by its actual name.
🙏 Special thanks to Amirhossein Hajimohammadi and Behrouz Rfa for their invaluable contributions in researching, troubleshooting, this deep-dive into Elasticsearch’s analyzer behavior — turning a frustrating edge case into a clear, actionable solution.
As a first step, we must confirm whether the issue stems from Elasticsearch itself or from our application code. Developers should first optimize their code and ensure all queries are correctly structured. However, even after these adjustments, the problem persists — and its root cause remains unidentified.
Once we have confirmed that the issue does not originate from the application code, the next step is to use Kibana’s Dev Tools to execute a direct query against Elasticsearch. This allows us to verify how stop words are being handled at the index level and determine whether they are the root cause of the search behavior.
Check ElasticSearch Query
As demonstrated in the cURL request below, no results were returned when searching for the term “the” — indicating that the term may be filtered out during analysis, likely due to stop word removal.
POST /product-index/_search{ "query": { "bool": { "must": [{"term": {"status": true}}],
"should": [ {"match_phrase": {"product_name": "the"}} ], "minimum_should_match": 1
} }, "size": 10}{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
}
}Working Example
If your index is properly configured and free of issues, searches — including those for terms like “the” — should return results without any problems.
POST /product-index-v2/_search{ "query": { "bool": { "must": [{"term": {"status": true}}],
"should": [ {"match_phrase": {"product_name": "the"}} ], "minimum_should_match": 1
} }, "size": 10}{ "took": 6, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0,
"failed": 0 }, "hits": { "total": { "value": 2, "relation": "eq" },
"max_score": 8.438612, "hits": [ { "_index": "product-index-v2", "_id": "118",
"_score": 8.438612, "_source": { "id": 2234, "product_name": "the",
"url_en": "product-the", "status": true, } } ] }}How to find the root cause of the problem
To inspect how a specific text field is configured for analysis in Elasticsearch — including which analyzers are used during indexing and searching — you should query the field’s mapping using the correct endpoint: GET /product-index/_mapping/field/product_name. This returns the field’s type (typically text for full-text search) and explicitly shows the analyzer (used when documents are indexed) and search_analyzer (used when queries are executed), such as "default" and "rebuilt" respectively. Note that this mapping API does not show the actual tokenization behavior — for that, you must use the _analyze API with sample text to see how input is broken into tokens. The original example contained errors, including querying for a field named product-index (which likely doesn’t exist. Always ensure your index and field names align with your actual data structure to avoid confusion and incorrect results.
GET /product-index/_mapping/field/product_name{
"hotel-index": {
"mappings": {
"name_fa": {
"full_name": "product-index",
"mapping": {
"product_name": {
"type": "text",
"analyzer": "default",
"search_analyzer": "rebuilt"
}
}
}
}
}
}GET /product-index/_mapping{
"name_fa": {
"type": "text",
"analyzer": "default",
"search_analyzer": "rebuilt"
}
}To ensure that stopwords are properly applied during text analysis in your product-indexenvironment, you can verify the analyzer configuration — specifically, the rebuilt analyzer — by fetching the index settings with GET /product-index/_settings?include_defaults=true. In the response, under "analysis" → "analyzer" → "rebuilt", you’ll see that the filter chain includes "english_stop", which confirms that English stopwords (like “the”, “and”, “in”, etc.) are being filtered out during both indexing and search — assuming this analyzer is assigned to your field (e.g., product_name). The presence of "english_stop" in the filter list means Elasticsearch will remove common, low-value words before applying stemming or other transformations, improving search relevance and reducing index noise. However, make sure this analyzer is actually assigned to your field via the mapping (e.g., "analyzer": "rebuilt"), and if you want to test its behavior, use the _analyze API with sample text to observe how stopwords are stripped during tokenization.
GET /product-index/_settings?include_defaults=true{
"analyzer": {
"rebuilt": {
"filter": [
"english_possessive_stemmer",
"lowercase",
"english_stop",
"english_keywords",
"english_stemmer"
],
"tokenizer": "standard"
}
}
}✅ Step-by-Step: Define Custom Stopwords + Analyzer
To create a custom English stopword list in Elasticsearch (rather than relying on the built-in "english_stop" filter), you need to define a custom stop filter in your index settings — and then reference it in your custom analyzer. The command GET /product-index-v2/_settings?include_defaults=true only retrieves current settings — it doesn’t create anything. So here’s how to define and apply a custom stopword list when creating or updating your index:
PUT /product-index-v2{ "settings": { "analysis": { "filter": {
"custom_english_stop": { "type": "stop", "stopwords": [ "a", "an",
"and", "or", "but", "in", "on", "at", "to", "for", "of", "with", "by", "is", "are",
"was", "were", "be", "this", "that" ] // You can also load from a file:
"stopwords_path": "analysis/stopwords/english_custom.txt" } }, "analyzer": {
"rebuilt_custom": { "tokenizer": "standard", "filter": [ "lowercase",
"custom_english_stop", // ← Your custom stop filter "english_possessive_stemmer",
"english_stemmer" ] } } } }, "mappings": { "properties": {
"product_name": { "type": "text", "analyzer": "rebuilt_custom", // ← Use your
custom analyzer "search_analyzer": "rebuilt_custom" } } }}🔍 Verify It Works
After creating the index, run:
GET /product-index-v2/_settings?include_defaults=true→ Look under "analysis" → "filter" → "custom_english_stop" to confirm your stopwords list is loaded.
🧪 Test the Analyzer
POST /product-index-v2/_analyze{ "analyzer": "rebuilt_custom", "text":
"The quick brown fox jumps over the lazy dog"}Conclusion:
What started as a seemingly trivial edge case — searching for a product literally named *“the”* — turned into a deep dive into Elasticsearch’s text analysis pipeline, revealing how default configurations can silently break real-world use cases. The culprit? Elasticsearch’s built-in stop word filtering, which — while excellent for general search relevance — fails catastrophically when the search term *is* the stop word.
🔍 Root Cause Confirmed
Through systematic debugging — from validating application code, to querying directly via Kibana Dev Tools, to inspecting index mappings and analyzer settings — we confirmed that the `english_stop` filter in the `rebuilt` analyzer was stripping out “the” during both indexing and search. This rendered our product unsearchable by its actual name.
✅ Solution Implemented
We resolved the issue by:
1. Defining a custom stop word list that excludes semantically critical terms like “the” when they are part of legitimate product names.
2. Creating a custom analyzer (`rebuilt_custom`) that uses this refined stop word filter.
3. Applying the analyzer to the `product_name` field in the index mapping to ensure both indexing and search respect the new rules.
4. Verifying the behavior using the `_analyze` API to confirm “the” is no longer discarded.
💡 Key Takeaways
- Never assume defaults are safe for your domain. Elasticsearch’s “one-size-fits-most” analyzers can break “edge cases” that are actually core business requirements.
- Always inspect analyzer behavior using `_analyze` — mappings alone won’t show you tokenization results.
- Customization is not optional when your data includes meaningful stop words, trademarks, or brand names that overlap with linguistic noise.
- Testing analyzers in isolation before reindexing saves hours of debugging later.
🚀 Moving Forward
This incident highlights the importance of aligning search engine behavior with business semantics — not just linguistic norms. Going forward, we recommend:
- Auditing all product/brand names for potential stop word conflicts.
- Building analyzer configurations that are domain-aware, not just language-aware.
- Documenting analyzer decisions so future engineers understand *why* “the” must be searchable.
In the end, “the” wasn’t just a word — it was a product. And in search, every word matters.