Скачать или смотреть How to Ensure Keyword Normalization in Elasticsearch: A Guide to Using Ingest Pipelines

Discover practical solutions for applying `keyword normalization` in Elasticsearch. Learn how to use ingest pipelines to modify source documents effectively.
---
This video is based on the question https://stackoverflow.com/q/73280103/ asked by the user 'Hakim' ( https://stackoverflow.com/u/4551281/ ) and on the answer https://stackoverflow.com/a/73280178/ provided by the user 'Val' ( https://stackoverflow.com/u/4604579/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Keyword normalizer not applied on document

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
How to Ensure Keyword Normalization in Elasticsearch: A Guide to Using Ingest Pipelines

When working with Elasticsearch, you may encounter issues related to keyword normalization, particularly if you expect certain strings in your documents to be transformed in a specific way. A common scenario involves having the keyword fields retain their original casing, even after applying a normalizer. This guide will walk you through understanding this issue and how to properly ensure your data is indexed as intended.

The Problem

In Elasticsearch, a user encountered a situation where the country field of a document retained its uppercase representation after being indexed, despite having applied a normalizer. The user had the following mapping in place:

[[See Video to Reveal this Text or Code Snippet]]

However, indexing a document with "country": "MX" resulted in the search still returning the uppercase value:

[[See Video to Reveal this Text or Code Snippet]]

This raised the question: What went wrong?

Understanding Normalizers vs. Analyzers

To address this issue, it’s essential to clarify the roles of normalizers and analyzers:

Normalizers: These are designed to modify the index data without altering the source document. They apply transformations such as case adjustments but only on the indexed content.

Analyzers: More comprehensive than normalizers, analyzers tokenize text and can apply multiple transformations simultaneously, including filtering out stop words.

In your case, the normalizer correctly changed the indexed value of the field but did not affect the original source document, which retains "country": "MX".

The Solution

Since normalizers do not alter the source document but only the indexed data, it’s not enough if you expect the source document to reflect these changes.

Step 1: Set Up an Ingest Pipeline

To modify your source document's casing before it gets indexed, you should use an ingest pipeline with a lowercase processor. The following example demonstrates how to create such a pipeline:

[[See Video to Reveal this Text or Code Snippet]]

Step 2: Index Documents Using the Pipeline

Once the pipeline is set up, you can index your documents while specifying the pipeline. Here’s how you would index your original document:

[[See Video to Reveal this Text or Code Snippet]]

Step 3: Verify the Results

You can retrieve the document to confirm that the country field is now in lowercase:

[[See Video to Reveal this Text or Code Snippet]]

The expected response will be:

[[See Video to Reveal this Text or Code Snippet]]

This confirms that by using the ingest pipeline, the original source document is altered before indexing, achieving your desired outcome.

Conclusion

In conclusion, when working with keyword normalization in Elasticsearch, it’s vital to understand the behavior of normalizers and their limitations regarding source documents. By using an ingest pipeline with a lowercase processor, you can effectively ensure that your data is indexed and stored in the format you prefer. This approach will help maintain consistency across your dataset and enhance your search functionality.

Happy indexing!