<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>Full-text search is an interesting field for backend developers: it requires raw technical knowledge, knowledge about the input languages and lots of feedback gathering on the usability side. In this article, I&apos;d like to give you a rough overview of the challenges in each area and inform you about basic solutions for them.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>I&#x2019;ve been implementing search solutions using Elasticsearch for the last 2 years. One of the things I start off with is gathering as much information about the data I&#x2019;m going to work with as soon as possible.
</p>
<p>This focus on the input data is often lacking for Elasticsearch beginners. Very often, they end up with managing to get all data <em>into</em> Elasticsearch and then trying to fix everything while trying to get things out again. Instead, they should have worked through the whole chain of things. In this post, I want to give a few pointers on things that are implemented in an afternoon, but save you headaches for days.
</p>
<p>There&#x2019;s two dangers lurking here: The real world is complex and clearing up that complexity is your first and foremost task. The second is that search rarely crashes, but degrades in quality if you make an error. Those errors will pop up over time when you cannot find a document for some reason. We&#x2019;ll try to make sure these aren&#x2019;t standard cases.
</p>
<p>Most of this comes down to proper analysis. For the technical details, see <a href="/blog/found-text-analysis-part-1/">this series on analyzers</a>. Always keep the shell open and the <a href="/guide/en/elasticsearch/reference/current/indices-analyze.html">analyze API</a> around.
</p>
</section><section id="inverse" class="level2">
<h2><a href="#inverse">Inverse</a></h2>
<p>Elasticsearch is based on Lucene, which implements a so-called <a href="http://en.wikipedia.org/wiki/Inverted_index">inverted index</a>. In contrast to key-value stores and to a lesser extent SQL databases, where the main way to access stored data is a (usually non-natural) key, inverted indices use <em>the data itself</em> as the key into the database. Following that line of thinking, it is pretty clear that messy input leads to messy retrieval. This means, we should take a close look at the data first.
</p>
</section><section id="dont-fake-the-real-world" class="level2">
<h2><a href="#dont-fake-the-real-world">Don&#x2019;t fake the real world</a></h2>
<p>First thing you should do before setting up a search project is get hold of some data. Don&#x2019;t fake it (for example by randomly generating text using a word list), use the real data. If your project has not produced any data yet, find text that matches your domain. Alternatively, if you intend to gather data later, build that part first.
</p>
<p>Let&#x2019;s have a look at real real world data issues.
</p>
</section><section id="encodings" class="level2">
<h2><a href="#encodings">Encodings</a></h2>
<p>Encodings are the most technical of issues when talking about text. Sadly, the situation here can still be summed up like this:
</p>
<blockquote class="twitter-tweet" lang="en">
	On encodings: Americans: &#x201C;I don&apos;t want to know&#x201D;; Japanese: &#x201C;How can you not know?!&#x201D;; Europeans: &#x201C;Unicode solves everything&#x201D;
	<br>
	&#x2014; Yehuda Katz (<span class="citation" data-cites="wycats"><a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a></span>) <a href="https://twitter.com/wycats/status/14149373552">May 17, 2010</a>
</blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>For all Western languages, Unicode works well though. The issue is less the Western-Eastern language split (which are very different), but the tendency of Western programmers to not care about those issues too much - especially if they think the language they are working in is free of any specialities.
</p>
<p>Case in point: (American) English. English writing is generally considered free of all those &#x201C;nasty&#x201D; things like diacritics. This assumption is na&#xEF;ve, especially in a language using loan words heavily. Some people might also just use fancy words like <a href="https://leanpub.com/javascript-allonge">allong&#xE9;</a> in book titles - and suddenly you have to take care of that stuff anyway. No amount of struggling will help you, unless you are the BBC and can just <a href="http://www.bbc.co.uk/academy/journalism/news-style-guide/article/art20130702112133548">ban usage of any diacritics</a>. Also consider that dropping diacritics from names can be seen as an insult.
</p>
<p>The good news is: Elasticsearch is focused on handling text and as such comes with all necessary things to handle Unicode text. There is one issue though: they come as a plugin called <a href="https://github.com/elasticsearch/elasticsearch-analysis-icu">analysis-icu</a>. <a href="http://site.icu-project.org/">ICU</a> is a vast library, implementing all interesting operations for Unicode. Today, we will just skim the surface, by using three standard components:
</p>
<ul>
	<li><code>icu_tokenizer</code>: this is a tokenizer breaking strings at everything unicode considers as &#x201C;segmenting words&#x201D;.</li>
	<li><code>icu_folding</code>: <a href="http://en.wikipedia.org/wiki/Letter_case#Unicode_case_folding_and_script_identification">folding</a> is what people often refer to as &#x201C;lowercasing&#x201D;. The difference is that not all characters have a lowercase equivalent or the mapping might be non-trivial. Don&#x2019;t think too hard about it, this is the operation you want.</li>
	<li><code>icu_normalize</code>: How do I compare &#x201C;&#xFC;&#x201D; to &#x201C;u&#x201D;? Don&#x2019;t think too hard about that question either. Unicode characters have a &#x201C;normal&#x201D; form, fit for comparison. <code>normalize</code> will give that to you. (Answer: for the sake of comparison, they are considered the same).</li>
</ul>
<p>All in all, don&#x2019;t fear Unicode. If you are handling text, consider it the default. Install the ICU plugin first, start using it right away and don&#x2019;t look back. It makes standard Unicode issues a no-brainer and will not be any worse than the standard implementations. If in doubt, use it.
</p>
</section><section id="fun-with-languages" class="level2">
<h2><a href="#fun-with-languages">Fun with languages</a></h2>
<p>Now, with those pesky encodings out of the way, let&#x2019;s have a look at input languages. Even if we don&#x2019;t have any problems with encoding them, they still all come with their own particularities. All these have to be dealt with to create a proper search people want to use. One of the problems here is: your input is usually rather cleanly structured and thought-out, search input is rather ad-hoc. So we need to match these two kinds of input still. Let&#x2019;s look at a few examples:
</p>
<ul>
	<li>French and English have elisions: &#x201C;j&#x2019;aime&#x201D; (I love), &#x201C;it&#x2019;s&#x201D;, people rarely enter them as search phrases</li>
	<li>Germans can use &#x201C;ue&#x201D; for input when they mean &#x201C;&#xFC;&#x201D; (e.g. because they use a foreign keyboard)</li>
	<li>Germans can combine words almost <a href="http://de.wikipedia.org/wiki/Rindfleischetikettierungs%C3%BCberwachungsaufgaben%C3%BCbertragungsgesetz">arbitrarily</a> to form a new one.</li>
	<li>Other languages have that in a more constrained fashion, consider &#x201C;football&#x201D;. People might search for &#x201C;ball&#x201D; and expect to find it.</li>
	<li>English has two major spellings: American and English (&#x201C;Consistent English spelling is a question of honor and labour&#x201D;)</li>
</ul>
<p>From a users perspective, they have a reasonable expectation that your search function handles those things. As with all things to do with search, a 100% solution is extremely hard or even impossible. But after finding such a problem statement, pleasing solutions are often easy to find.
</p>
<ul>
	<li>Elisions can be cleaned using the <a href="/guide/en/elasticsearch/reference/current/analysis-elision-tokenfilter.html">elision token filter</a></li>
	<li>The <a href="http://snowball.tartarus.org/algorithms/german2/stemmer.html">german2</a> stemmer variant transforms all &#x201C;ue&#x201D; into &#x201C;&#xFC;&#x201D; in most cases, at the cost of breaking a very low number of words.</li>
	<li>The <a href="https://github.com/jprante/elasticsearch-analysis-german">elasticsearch-analysis-german</a> plugin ships an algorithmic decompounder usable in German, turning words back into subwords. For easier languages, the <a href="/guide/en/elasticsearch/reference/current/analysis-compound-word-tokenfilter.html#analysis-compound-word-tokenfilter">compound token filter</a> can help you.</li>
	<li>For a known set of spelling variants, try to get hold of a word list from a free source (e.g. <a href="http://en.wikipedia.org/wiki/Wikipedia:List_of_spelling_variants">wikipedia</a>) and feed them to a <a href="/guide/en/elasticsearch/reference/current/analysis-synonym-tokenfilter.html">synonym filter</a></li>
</ul>
<p>Finding and dealing with these particularities will improve your search functionality. The interesting thing here is that a lot of these issues need understanding of the input language, especially when validating. Don&#x2019;t hesitate to ask a native speaker for help! Also: start with the nasty examples first.
</p>
<p>The Elasticsearch guide has a <a href="/guide/en/elasticsearch/guide/current/languages.html">great article</a> on these particularities.
</p>
</section><section id="fun-with-relevancy" class="level2">
<h2><a href="#fun-with-relevancy">Fun with relevancy</a></h2>
<p>&#x201C;We want great search functionality, can Elasticsearch do that?&#x201D;
</p>
<p>Sure, it can, but only if you define what &#x201C;great&#x201D; is. And this is where things become interesting: there is not single definition of &#x201C;relevant&#x201D;. For example, if the search is for restaurants, I can match for names and dishes as much as I want, if the restaurant is on the other side of the country, it&#x2019;s probably not relevant. <a href="/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#_decay_functions">Distance</a> is an important criterion there. On the other hand, if we are talking about a specialised website for three-star-restaurants, people will probably be ready to travel for that. <a href="http://restaurant.michelin.fr/restaurants/restaurants-3-etoiles-michelin">France has 26 of them</a>, so anything but the broadest location filter by name will not add a lot to the user experience. Shops need important products up, TV production company employees will search for internal three-letter codes for their shows, while TV viewers will search for a popular abbrevation of the same shows. In some cases, a narrow set of results is wanted, while broad results are preferred somewhere else. Criteria might work against each other, e.g. putting too much weight on recency of a document can lead to new documents overlaying better documents where the text matches better.
</p>
<p>The most important part of implementing search with relevant results is realizing that &#x201C;relevancy&#x201D; is not a technical term. It can be modelled using technology later, but it is a business and ultimately a management task to define what &#x201C;relevant&#x201D; is. This is highly dependent on what the data in question represents and what people are interested in. Receiving feedback from many channels is important. Finding strategies is as important as understanding the technical parts. Here are some suggestions:
</p>
<ul>
	<li>Have a list of all queries that were problematic, or illustrate what you want well. Don&#x2019;t just put that in tickets, have a central list for them. Annotate them with your expectations. Either as a wiki page or even as a shared spreadsheet somewhere. Be aggressive about adding things to that list.</li>
	<li>Consider building an interface that allows you to play with the parameters of your queries. Build it mirrored: the left hand side shows the current parameters, the right hand side allows you to play and compare. This allows you to get someone non-technical over to your desk and play for a few minutes. Use that regularly.</li>
	<li>And most importantly: Log all your queries. Find popular ones. Check their results. See whether they fit your expectations.</li>
	<li>Interview people about what they would like to know about your data.</li>
</ul>
<p>Your relevancy model is the core of your product. Sometimes, it&#x2019;s simple. At other times, it&#x2019;s a full-time job. Try to keep the feedback wheel turning.
</p>
</section><section id="putting-it-all-together" class="level2">
<h2><a href="#putting-it-all-together">Putting it all together</a></h2>
<p>All this shows why I love working in this field so much. It comes down to a very diverse and interesting set of tasks.
</p>
<p>First is the die-hard technical component: how do I work with the input structure? What are the pitfalls (Encoding, structural issues)? How does this all work in detail?
</p>
<p>Then we have the required knowledge beyond the technical: what particularities do the input languages have? Are there multiple? What makes them special?
</p>
<p>And finally we have the task of interacting with others to find out what they need and want to later put that into writing. Interviewing people or having them find new stuff is a very pleasant break from coding.
</p>
<p>Elasticsearch supports you with all those tiny details.
</p>
<p>And with that, I&#x2019;d like to wish you a great holiday and a happy new year!
</p>
</section></body></html>

Full-Text Search is an Input-Oriented Task

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

Florian Gilcher

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Author

Articles by Florian Gilcher

Full-Text Search is an Input-Oriented Task

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS