Two common ways to address this problem are relevance feedback and query expansion. In relevance feedback, the system adjusts the query based on the documents that seem relevant in the first round of retrieval. In query expansion, a query is expanded or reformulated to match other semantically similar terms.

Relevance Feedback

Relevance feedback is used to improve both recall and precision. The main idea is simple: use feedback from users to refine the query.

The process is typically as follows:

1. The user issues a query.
2. The system returns an initial set of results.
3. The user judges relevance of returned documents.
4. The system updates the query based on the user feedback.
5. The system returns a revised set of results.

Based on how step 3 is handled, there are three main types of relevance feedback:

• Explicit relevance feedback
• Implicit relevance feedback
• Pseudo relevance feedback (also called blind or automatic relevance feedback)

Explicit Relevance Feedback

In explicit relevance feedback, or just relevance feedback, users are asked to judge returned documents. In practice, though, this approach is often limited because most users do not want to spend extra effort during search. This is especially true when they have no strong incentive to provide that feedback. It also becomes less practical when the result set is large, since judging many documents takes time.

Even so, explicit relevance feedback can still be useful in laboratory or experimental settings, where users may be motivated to provide relevance judgments.

Implicit Relevance Feedback

In implicit, or indirect, relevance feedback, the system uses user behavior logs such as clickthrough as relevance signals. For example, documents clicked for a query may be treated as relevant, while documents that are not clicked may be treated as less relevant. A higher number of clicks may also be interpreted as a stronger relevance signal.

This is probably the most common approach in real retrieval systems, because it does not ask users to provide explicit judgments. It is also easier to collect at scale in systems with many users and large amounts of interaction data.

One limitation is that these signals are not always reliable. A click does not always mean a document is relevant: it may be accidental, or even malicious.

Ranking advertisements for a search query in a web search engine is a closely related application of this idea.

Pseudo Relevance Feedback

In pseudo relevance feedback, also called blind or automatic relevance feedback, the top-k documents in the initial results are assumed to be relevant. So, unlike explicit or implicit feedback, this method does not require any user input.

Because it makes that assumption automatically, it is generally less reliable than other forms of relevance feedback. For example, if the query is “john wick” and the top-k documents are mostly about Keanu Reeves, the updated query may drift toward documents about Keanu Reeves rather than the film itself.

One useful application of this method is association anaylsis. For example, we can build a topic language model and compare it with the collection language model to extract terms that are strongly related to a given query. These related terms can then be added to the original query. This is a form of query expansion, which we will discuss later in this post.

The Rocchio Algorithm

The Rocchio algorithm is the most effective classic relevance feedback method in vector space model. It is used in step 4, where the system updates the query based on user feedback.

The basic idea is to build a new query vector $\vec{q}$ that is closer to relevant documents and farther from nonrelevant ones. In other words, we want a query vector that increases similarity to relevant documents while decreasing similarity to nonrelevant documents.

To do so, we need some knowledge of what documents are relevant (positive) and nonrelevant (negative). This is where relevance feedback comes in. Once we have the knowledge, we can move the initial query vector toward the centroid of the relevant documents and away from the centroid of the nonrelevant documents.

The formula is:

Where:

• $\vec{q_m}$: the modified (new) vector
• $\vec{q_o}$: the original query vector
• $D_r$: the set of known relevant documents
• $D_{nr}$: the set of known nonrelevant documents
• $\alpha$: the original query weight
• $\beta$: the related documents weight
• $\gamma$: the nonrelated documents weight
• $\frac{1}{\vert D_r \vert}\sum_{\vec{d}_j \in D_r}\vec{d}_j$: the centroid (vector) of positive examples
• \(\frac{1}{\vert D_{nr} \vert}\sum_{\vec{d}_j \in D_{nr}}\vec{d}_j\): the centroid (vector) of negative examples

As the formula shows, three parameters ($\alpha$, $\beta$, $\gamma$) control how much weight we give to the original query, the relevant documents, and the nonrelevant documents. For example, if we have enough judged documents to trust the feedback, we may want to use higher values for $\beta$ and $\gamma$.

In practice, $\alpha$ is usually kept relatively high to preserve the importance of the original query vector. This also helps avoid overfitting when the number of feedback samples is small. Meanwhile, $\gamma$ is often kept small, or even set to 0. This is because nonrelevant documents are not always clustered together, so they can push the query in many different directions. Therefore, reasonable values might be something like:

• $\alpha$ = 1
• $\beta$ = 0.5
• $\gamma$ = 0.1

Here’s an example. Suppose the query $\vec{q}$ is hybrid search, and the system returns four documents:

The user judges the second and third documents as relevant, and the fourth document as nonrelevant:

We first need to define a consistent term order for the vector dimensions. If we list the unique terms from the query and the four documents in alphabetical order, we get:

('a', 'and', 'can', 'combines', 'consumption', 'Elasticsearch', 'engine', 'fuel', 'hybrid', 'in', 'keyword', 'reduce', 'search', 'semantic', 'successful', 'system')

So this is a 16-dimensional vector space.

Assume the search engine uses term frequency, but not length normalization or IDF. Then each document can be represented as follows:

The query $\vec{q}$ in the same vector space is:

Assume the system uses Rocchio relevance feedback with $\alpha = 1$, $\beta = 1$, and $\gamma = 0$. Then the modified query becomes:

Since $\gamma = 0$, the nonrelevant document $\vec{d}_4$ is ignored.

So the final modified query vector $\vec{q}_m$ is:

In a more readable format:

One potential problem of this method is computational cost. After relevance feedback, the updated query vector may contain many terms from relevant documents. In our example, the original query has only 2 non-zero terms, but the updated query has 8.

In practice, we often truncate the modified vector and keep only the highest-weighted terms for efficiency. In the modified query vector above, the top terms are search (3.0), hybrid (2.0), and then several terms with weight 0.5. If we keep only the top 3 terms, the new query becomes roughly search hybrid Elasticsearch.

As the Rocchio algorithm works within a vector space model, it is not limited to text retrieval systems. It can be applied to any type of data that can be represented as vectors, including images, audio, and learned embeddings.

Demonstration

For this demonstration, I embedded a collection of images from Pixabay. I used openai/clip-vit-base-patch32 to generate the image embeddings, then reduced them to a 256-dimensional vector space.

The first image shows the initial results for the query “cat”:

Now suppose the user marks six images as relevant (blue boxes) and two cat food images as nonrelevant (red boxes):

At this point, the system has useful information about which documents are relevant and which are nonrelevant for the query cat. Using this feedback, we can modify the query vector with the Rocchio algorithm.

The next image shows the revised results after updating the query vector with $\alpha=1$, $\beta=0.5$, and $\gamma=0.1$:

As you can see, the images explicitly marked as non-relevant no longer appear in the results, and more relevant images are shown instead.

However, increasing $\gamma$ can hurt the results. The next image shows the revised results when the query vector is updated with $\alpha=1$, $\beta=0.5$, and $\gamma=0.8$:

In this result set, nonrelevant images are mixed in with the cat images because the query is pulled too strongly by the negative examples.

For reference, here is the Rocchio implementation I used in this demonstration:

import numpy as np


def get_modified_query_vector(
    original_query_vector: np.ndarray,
    relevant_document_vectors: list[np.ndarray],
    nonrelevant_document_vectors: list[np.ndarray],
    alpha: float = 1.0,
    beta: float = 0.1,
    gamma: float = 0,
) -> np.ndarray:
    """
    Calculate the modified query vector using the Rocchio algorithm.

    Args:
        original_query_vector: The original query vector.
        relevant_document_vectors: List of vectors for relevant documents.
        nonrelevant_document_vectors: List of vectors for non-relevant documents.
        alpha: Weight for the original query vector.
        beta: Weight for the relevant document vectors.
        gamma: Weight for the non-relevant document vectors.

    Returns:
        The modified query vector.
    """
    # Calculate the mean vector of relevant documents
    if len(relevant_document_vectors) > 0:
        mean_relevant_vector = np.mean(relevant_document_vectors, axis=0)
    else:
        mean_relevant_vector = np.zeros_like(original_query_vector)

    # Calculate the mean vector of non-relevant documents
    if len(nonrelevant_document_vectors) > 0:
        mean_nonrelevant_vector = np.mean(nonrelevant_document_vectors, axis=0)
    else:
        mean_nonrelevant_vector = np.zeros_like(original_query_vector)

    # Apply the Rocchio formula
    modified_query_vector = (
        alpha * original_query_vector
        + beta * mean_relevant_vector
        - gamma * mean_nonrelevant_vector
    )

    return modified_query_vector

Constraints and Limitations

Relevance feedback alone may not be sufficient in certain conditions:

• Misspellings or typos. These are better handled with spelling correction.
• Cross-language information retrieval. For example, if you search for “global warming” in a collection of Korean documents that use the term “지구 온난화”, relevance feedback may not help if it only learns associations between English terms. This problem is usually handled with machine translation or cross-lingual semantic mapping.
• Synonymy. If you search for “laptop” but the documents use “notebook computer,” the query may miss relevant results. One way to handle this is query expansion using a predefined dictionary or thesaurus.

The Rocchio algorithm represents relevant documents with a single centroid, so it works best when those documents form a cluster. It may not work well when relevant documents are split into groups that use different vocabulary, such as “Burma” and “Myanmar.” (The country of Burma was renamed to Myanmar in 1989.)

Relevance feedback is also less useful in web search. Its main benefit is often improved recall, but most web search users do not care much about high recall. They are usually satisfied if the system returns a small number of highly relevant results.

Query Expansion

In relevance feedback, user feedback is used to modify the query vector. In query expansion, the original query is replaced with an alternative query or expanded with related terms. Like relevance feedback, query expansion is primarily used to improve recall.

There are two main ways to do this:

• Interactive query expansion: The system shows users related queries, and the user chooses one of them. This is often called query suggestion. For example, if a user searches for apple, the interface might suggest queries such as apple store or apple support.
• Automatic query expansion: The system automatically adds synonyms or related terms to the query. For example, if a user searches for Burma, the system may also search for Myanmar. In this case, the user may not even notice that the query has been expanded.

The main problem is how to generate these alternative or expanded queries. The most common approach is to use some form of thesaurus. That is, for each term t in the query, the system can expand the query with synonyms or related words for t taken from the thesaurus. A thesaurus-based approach has the advantage that it does not require user feedback.

The entries in a thesaurus can be created manually or generated automatically.

Manual Thesaurus

In a manual thesaurus, developers or human editors manage sets of synonyms and alternative terms. This approach is especially useful in specialized domains, where term relationships need to be carefully curated. It can also be useful for fast-changing or trendy queries, where developers want to quickly add related terms that may not yet appear in a general thesaurus.

When building a thesaurus from scratch, developers can start from public or open-source thesauri. For example, the UMLS Metathesaurus is a useful resource for building search systems in the biomedical domain.

A simple yet practical way to build up a thesaurus is to use a document database like Elasticsearch. By leveraging an inverted index, each synonym can be looked up quickly, making it easy to retrieve the full synonym list from a query term.

For example, we can create a simple thesaurus index like this:

PUT thesaurus
{
  "mappings": {
    "properties": {
      "synonyms": {
        "type": "keyword"
      }
    }
  }
}

If you need to designate canonical terms for each set of synonyms, then you can add another field such as canonical_term:

PUT thesaurus
{
  "mappings": {
    "properties": {
      "synonyms": {
        "type": "keyword"
      },
      "canonical_term": {
        "type": "keyword"
      }
    }
  }
}

A canonical term is the preferred or normalized form of a synonym group. For example, you might want to treat “burma” as a synonym, but internally normalize it to “myanmar”.

You can add an entry to this thesaurus like this:

POST thesaurus/_doc
{
  "synonyms": [
    "burma",
    "myanmar"
  ]
}

Then, when a user searches for one term, you can look it up in the thesaurus and retrieve the full synonym group. For example, if the query contains “burma”, we can search for thesaurus entries that include that term:

GET thesaurus/_search
{
  "query": {
    "term": {
      "synonyms": "burma"
    }
  }
}

Response:

{
  ...,
  "hits": {
    ...,
    "hits": [
      {
        ...,
        "_source": {
          "synonyms": [
            "burma",
            "myanmar"
          ]
        }
      }
    ]
  }
}

Now the search system can expand the original query using the returned synonym set.

Automatic Thesaurus Generation

Because building a manual thesaurus is expensive, we can try to generate one automatically.

One approach is to use word co-occurrence statistics over a document collection. The idea is that words that co-occur in the same document or paragraph are likely to be semantically similar or related.

The simplest way to build a co-occurrence thesaurus is to compute term-term similarity. This approach starts with a term-document matrix $A$:

Each cell $A_{t,d}$ is a weight $w_{t,d}$ of term $t$ in document $d$. If we compute $C = AA^T$, then each entry $C_{u,v}$ measures how strongly terms $u$ and $v$ are related. The larger the value, the more related the two terms are.

Here’s a simple example. Suppose we have three documents:

• $d_1$ (about cars)
• $d_2$ (about cars)
• $d_2$ (about fruit)

And four terms:

• $t_1$: car
• $t_2$: automobile
• $t_3$: engine
• $t_4$: banana

Let the term-document matrix $A$ be:

Notice that “car”, “automobile”, and “engine” are strong in $d_1$ and $d_2$ while banana only appears in $d_3$.

By multipyling $A$ and $A^T$:

we get the term-term matrix $C$:

Each $C_{u,v}$ is just the dot product of the row for term $u$ and the row for term $v$:

For example, the similarity between “car” and “engine” is:

So, the system can build entries like:

• car -> automobile, engine
• automobile -> car, engine
• banana -> (none of these)

Constraints and Limitations

A manual thesaurus is expensive to build and maintain. For domain-specific systems, a general thesaurus is often not enough, because it may not cover specialized vocabulary well. For example, in medical or legal search systems, specialized terminology and relationships must be covered as well.

A thesaurus generated automatically from co-occurrence statistics also has clear limits. It does not necessarily capture true synonyms. Instead, it captures words that have similar distributions across documents, such as “car” and “engine”. Its quality also depends heavily on the document collection. If the corpus is small, the statistics may be unreliable. If the corpus is biased toward a particular topic, the thesaurus will reflect that bias. For example, in a general-language corpus, likely neighbors of the term “virus” might be “pathogen”, “infection”, and “germ”. But if the corpus is heavily biased toward medicine and public health, virus may co-occur much more often with words like “vaccine”, “patient”, and “outbreak”. As a result, the generated thesaurus may rank those words highly.

Elasticsearch Support for Query Expansion

Elasticsearch, an open-source search and analytics engine, supports the significant_terms and significant_text aggregations. These can be useful for discovering related terms automatically, at least to some extent.

These do not compute plain co-occurrence counts. Instead, they find terms that are unusually common in a foreground set compared with a background set:

• Foreground set: the search results matched by a query
• Background set: the index or indices from which the results were gathered

For example, the following query searches headlines for “health” and finds terms that are unusually common in the headline field of those results compared with the whole index:

GET news_categories/_search
{
  "size": 0,
  "query": {
    "match": {
      "headline": "health"
    }
  },
  "aggregations": {
    "keywords": {
      "significant_text": {
        "field": "headline"
      }
    }
  }
}

Response:

{
  ...
  "hits": {
    "total": {
      "value": 2349,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "keywords": {
      "doc_count": 2349,
      "bg_count": 628581,
      "buckets": [
        ...
        {
          "key": "care",
          "doc_count": 706,
          "score": 14.049006021392186,
          "bg_count": 3957
        },
        {
          "key": "mental",
          "doc_count": 270,
          "score": 6.105789285390884,
          "bg_count": 1335
        },
        {
          "key": "insurance",
          "doc_count": 85,
          "score": 1.3707596512376896,
          "bg_count": 585
        },
        ...
      ]
    }
  }
}

The results suggest that “care”, “mental”, and “insurance” are strongly associated with “health” in headlines.

Take “care” as an example. It appears in the headline field of 3,957 documents in the full index, as shown by bg_count. Of those, 706 appearances are in the 2,349 documents whose headlines matched “health”. That means “care” is unusually common in the foreground set compared with the background set, so Elasticsearch considers it significant.

A user might then choose to add “care” to the query, for example to expand a search for “health” into something like “health care” or to otherwise improve recall.

(The demonstration uses the News Category Dataset by Rishabh Misra from Kaggle, licensed under CC BY 4.0.)

The query above is the simplest way to use the significant_text aggregation. However, re-analyzing many matching documents requires a lot of time and memory. For better performance, it is recommended using significant_text aggregation with a sampler, as shown below:

GET news_categories/_search
{
  "size": 0,
  "query": {
    "match": {
      "headline": "health"
    }
  },
  "aggregations": {
    "my_sample": {
      "sampler": {
        "shard_size": 100
      },
      "aggregations": {
        "keywords": {
          "significant_text": {
            "field": "headline"
          }
        }
      }
    }
  }
}

One advantage of this approach over raw co-occurrence is that it automatically suppresses globally common words. However, just like co-occurrence statistics, the selected terms are not necessarily similar or related terms.

Conclusion

For improving recall, relevance feedback is generally more effective than query expansion. That said, finding good values for the Rocchio weights, $\alpha$, $\beta$, and $\gamma$, is still a problem.

Query expansion, on the other hand, can hurt precision if it is used carelessly. However, it is easier for developers to understand than relevance feedback. Therefore, it is much more straightforward to implement, reason about, and debug.

References

• An Introduction to Information Retrieval by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze
• Text Retrieval and Search Engines by University of Illinois Urbana-Champaign
• Rocchio algorithm - Wikipedia
• Significant terms aggregation - Elasticsearch
• Significant text aggregation - Elasticsearch

The intuition is as follows. When someone seeks information, they have an implicit picture of the kinds of documents that they are looking for and the words those documents are likely to contain. Based on these imaginary documents and their representative terms, they choose search terms and send a query to a search engine, expecting the system to return those documents. The language modeling approach inverts this process. Given a document, it asks: “how likely is it that this document’s model would generate the given query?”

First, let’s review what the probabilistic modeling approach is in IR.

Probabilistic Modeling Approach

Probabilistic approaches to IR model ranking as an estimation problem. That is, given a query $q$ and a document $d$, we aim to estimate the probability that the document is relevant:

Where $R$ is a binary random variable indicating relevance.

Okapi BM25 is a non-binary model of the probabilistic model family.

The key question is: how do we realize this idea?

One intuitive approach is to use click-through logs as implicit relevance feedback. Suppose we have implicit relevance judgments mined from user clicks:

For query $q_1$, we can estimate relevance for documents $d1$, $d2$, $d3$ by computing the fraction of exposures that resulted in a click: $p(R \mid d,q) = \frac{count(q,d,R=1)}{count(q,d)}$.

This yields:

• $P(R=1 \mid d_1,q_1) = 2/2 = 1$
• $P(R=1 \mid d_2,q_1) = 0/2 = 0$
• $P(R=1 \mid d_3,q_1) = 1/2 = 0.5$

So the ranking for $q_1$ would be $d_1 > d_3 > d_2$.

A clear limitation of this approach is that we have no data from which to estimate relevance if a document has never been shown before or if a query has never appeared. In practice, most queries are rare and most documents get little or no interaction.

The language modeling approach provides a different realization of probabilistic IR. Rather than estimating $P(R=1 \mid d,q)$, it reframes the problem as estimating how likely the document would generate the query:

In practice, it builds a probabilistic language model $M_d$ from each document $d$, and ranks documents based on the probability of the model generating the query:

This approach directly tackles the question in the intro: “how likely is it that this document’s model would generate the given query?”.

Let’s briefly go over what language models are and how they are constructed.

Language Models

A language model is a function that puts a probability measure over strings drawn from some vocabulary [An Introduction to Information Retrieval]. In the IR setting, these strings can be thought of as sequences of words or terms.

While often used interchangeably, it is useful to clarify three commonly used concepts:

• words: natural language units as written or spoken.
• terms: normalized tokens that an IR system indexes and matches.
• tokens: implementation-level units produced by a tokenizer, purely technical.

One of the simplest language models is a finite automaton, or simply a state machine, consisting of just a single node that generates terms based on a single probability distribution.

Because all terms are drawn independently from this distribution, the probabilities must sum to 1:

Where:

• $t$: term
• $V$: vocabulary

Since this model has no memory, each term is generated independently of the previous one. To compute the probability of a sequence, we simply multiply the term probabilities.

For example, here’s a partial specification of the term probabilities:

Using this table, the probability of generating a string “chocolate milkshake” is:

Such a model is often called “generative” because it can stochastically produce sequences of terms. A formal generative model would include a stop probability to decide whether to stop or to loop around and then produce another word. For example, if $P(\text{STOP}) = 0.2$, then $P(\text{chocolate milkshake}) = (0.03 \times 0.02) \times (0.8 \times 0.2) = 0.000096$. But in IR this factor cancels out when comparing models and does not affect ranking. Therefore, we omit to include stop probability in the IR context.

One use case of language models is association analysis, where we identify terms that are semantically related to a given seed word. Suppose we want to find words related to “vitamin”. We can build a topic language model $P_{topic}$ from documents containing that term:

Here, common terms like “the”, “is”, and “a” dominate the distribution, but they are not informative. To discount such common terms, we can compare the topic model against a background model constructed from a broad corpus:

By taking the ratio $\frac{P_{topic}(t)}{P_{background}(t)}$, we highlight terms that are unusually frequent in the topic:

From this result, we can see which terms are common to both models and which are disproportionately frequent in the topic model. In particular, “nutrition” stands out as much more characteristic of the topic than generic words such as “the” or “is”, making it more likely to be semantically related to “vitamin”.

Unigram Language Model

The language models introduced so far belong to the simplest class called unigram language models. In a unigram model, the order of words does not matter. Because it treats a document as merely a collection of terms, it is often referred to as a “bag of words” approach. The model ignores all contextual dependencies and assumes that every term occurs independently:

There are more complex language models. For example, a bigram language model conditions each term on the term immediately preceding it:

More complex language models include trigram models and even grammar-based models that capture deeper structure in language. These richer models can be helpful in tasks such as:

• speech recognition
• spelling correction
• machine translation

All of these tasks depend on how likely a term is given the context that surrounds it.

In contrast, most language-modeling work in IR relies on unigram models as IR rarely depends on sentence structure. For determining what a document is about, unigram models are often sufficient. Unigram models are also computationally more efficient to estimate and apply than higher-order models. This efficiency makes them a good fit for large-scale IR systems, where speed and scalability are critical.

Maximum Likelihood Estimation

To estimate $P(q \mid M_d)$, we typically use maximum likelihood estimation (MLE). Under MLE, the probability of a term in a document is simply its frequency divided by the total number of tokens in that document:

Where:

• $tf_{t,d}$ is the (raw) term frequency of term $t$ in document $d$
• $L_d$ is the number of tokens in document $d$

With MLE and the unigram assumption, the probability of producing the query $q$ from the document language model $M_d$ becomes:

Where $\hat{P}$ is the estimated probability.

Suppose a document $d$ contains 100 tokens ($L_d=100$), and we observe the following term frequencies:

The corresponding MLE probabilities are:

So, the probability of generating the query “chocolate milkshake” from $M_d$ is:

Now that we know what language models are and how to estimate $P(q \mid M_d)$, we can move on to applying these ideas to the ranking problem. For simplicity, we will omit the estimator symbol $\hat{}$ on top of $P$ in the rest of the discussion.

Query Likelihood Model

The query likelihood model (often abbreviated as QLM) is the original and basic method for using language models in IR. The core assumption is that a user has some ideal or “prototype” document in mind and generates a query using terms that appear in that document. Whether users literally behave this way is debatable, but the intuition holds. That is, people tend to choose query terms they believe are representative and discriminative for the documents they want.

In QLM, we construct a language model $M_d$ for each document $d$ in the collection. Our goal is to rank documents by $P(d \mid q)$. This probability represents how likely it is that document $d$ is relevant to query $q$. Using Bayes’ rule:

Since $P(q)$ does not vary across documents, it can be ignored for ranking. The prior $P(d)$ is also commonly assumed to be uniform, so it can likewise be dropped. This leaves:

In other words, we rank documents by the likelihood that the document’s language model would generate the query. Retrieval becomes a query-generation problem: the higher the likelihood, the more relevant the document.

The most common way to do this is using the unigram language model with MLE:

This assumes that each term in the query is generated independently, so we multiply their probabilities. Different ways of estimating $P(t \mid M_d)$ lead to different retrieval functions, and MLE is the simplest one.

In practice we can think of QLM as a three step process:

1. Infer a LM for each document.
2. Estimate $P(q \mid M_{d_{i}})$, the probability of generating the query according to each of these document models.
3. Rank the documents according to these probabilities.

Here’s an example. Suppose the query is “chocolate milkshake”, and we have two documents, $d1$ and $d2$. By inferencing the document LMs for both documents, we get:

Using MLE, we can then compute the term probabilities in each document model:

• $P(\text{chocolcate milkshake}\mid M_{d_1})=0.03 \times 0.02=0.0006$
• $P(\text{chocolcate milkshake}\mid M_{d_2})=0.02 \times 0.01=0.0002$

Since $0.0006 \gt 0.0002$, we rank $d_1 \gt d_2$.

In practice, multiplying many small probabilities leads to floating-point underflow, where the outcome is so small that it can no longer be represented by the computer. To avoid this, sums of log probabilities is often used:

Where $tf_{t,q}$ is the (raw) term frequency of term $t$ in query $q$.

This transformation preserves ranking, since the logarithm is a monotonic function.

A major limitation of the basic query likelihood approach arises when a query term does not appear in a document. Under MLE, it gives $P(t \mid M_d)=0$, making the entire query likelihood zero. Even in log space, MLE assigns $\log{P(t \mid M_d)}=-\infty$, making the entire query likelihood $-\infty$. The result is an overly strict conjunctive (AND-style) retrieval where a document contributes only if it contains every query term. In addition, even observed terms are poorly estimated under MLE. Rare terms tend to be overestimated because their single occurrence may be due to chance. The solution is smoothing.

Smoothing

Smoothing is a way to discount non-zero probabilities and to give some probability mass to unseen terms. The general approach is to assign a non-zero probability to unseen terms based on the whole collection. That is:

Where:

• $\alpha_d$ is a smoothing factor of document $d$ for unseen terms.
• $M_c$ is a language model built from the entire collection.

For a term that doesn’t appear in document $d$, we fall back on the collection probability:

Where:

• $cf_t$ is the raw count of the term in the collection
• $T$ is the raw size (number of terms) of the entire collection

With this idea, the ranking function becomes:

We can make this equation simpler by removing the $\sum_{t \in V}tf_{t,q}\log{P(t \mid M_c)}$ part as it’s going to be the same for every document and doesn’t affect ranking. So:

This transformation automatically provides two benefits:

1. More efficient computation as scoring is only based on matched query terms
2. The effects of TF-IDF weighting and document length normalization:
   • $P(t \mid M_d)$ behaves like TF.
   • $\frac{1}{P(t \mid M_c)}$ behaves like IDF.
   • $\vert q \vert \log\alpha_{d}$ acts like document length normalization, because longer documents usually need a smaller $\alpha_{d}$.

A takeaway here is that the role of smoothing is not only to avoid zero probabilities. The smoothing of terms actually implements major parts of the term weighting component.

Next up are the specific smoothing methods that have been shown to perform well in IR experiments:

• Jelinek-Mercer smoothing
• Dirichilet Prior smoothing

Jelinek-Mercer Smoothing

Jelinek-Mercer smoothing, also referred to as as linear interpolation smoothing, is the simpleest smoothing method. The idea is to mix two sources of information. One comes from the document itself and the other comes from the whole collection:

Where $\lambda$ is a smoothing factor that controls the extent of smoothing ($0 < \lambda < 1$). A large value of $\lambda$ means more smoothing.

A useful property is that $\lambda$ doesn’t have to be a fixed constant and you can make it depend on the query length. For example:

• For short queries: use a small $\lambda$
• For long queries: use a larger $\lambda$

If we substitute the JM formula into our scoring function, we get:

Since, $\vert q \vert \log\lambda$ is the same for every document, we can drop it for ranking purposes. So:

Below are the interpretations of some parts in this equation:

• $\lambda$ plays the role of the smoothing factor (similar to $\alpha_d$).
• $tf_{t,q}$ acts like TF weighting.
• $\frac{1}{P(t \mid M_c)}$ acts like IDF weighting.
• $\frac{1}{L_d}$ provides document length normalization.

So even though this looks like a language-modeling formula, it ends up recreating a lot of the same behavior as classic TF-IDF ranking.

Here’s an example. Suppose our collection has two documents:

• $d_1$: “Here is a recipe for a classic, creamy chocolate milkshake”
• $d_2$: “Dark chocolate is a little bitter but very delicious”

We’ll use basic unigram MLEs for both the documents and the whole collection, and we’ll set $\lambda = 0.5$. Now take the query: “chocolate milkshake”

Then:

Only $d_1$ contains “milkshake”, so it gets a strong boost in score. The ranking ends up as $d_1 \gt d_2$. Notice that even though $d_2$ does not contain “milkshake” at all it still receives a non zero score because smoothing gives it some probability mass.

Dirichilet Prior Smoothing

Dirichilet Prior smoothing, also called as Bayesian smoothing, takes a different angle from Jelinek–Mercer. Instead of mixing two models with a fixed weight, it treats the collection language model as a prior and then updates it using evidence from the document.

The formula is:

Where

• $L_d$ is the number of terms in document $d$
• $\alpha$ is a smoothing factor ($\alpha \in [0, +\infty)$). A large value of $\alpha$ means more smoothing.

This equation has an intuitive interpretation:

• If a document is short, $L_d$ is small, so the model leans more toward the collection (more smoothing).
• If a document is long, $L_d$ dominates, so the model trusts the document more (less smoothing).

Using this expression, the scoring function becomes:

Just like JM smoothing, this method naturally gives tf-idf weighting and document length normalization. Specifically:

• $tf_{t,q}$ -> TF weighting
• $\frac{1}{P(t \mid M_c)}$ -> IDF weighting
• $\vert q \vert \log\frac{\alpha}{L_d+\alpha}$ -> document length normalization

A key difference from JM smoothing is that the effective weight ($\alpha_d$) becomes $\frac{\alpha}{L_d+\alpha}$ instead of the fixed linear coefficient $\lambda$. This means the model automatically smooths short documents more and long documents less, which makes the interpolation adaptive rather than fixed.

Other Language Modeling Approaches

Up to this point, we’ve focused on a single perspective that is query likelihood. However, there are other language modeling approaches worth mentioning.

Document Likelihood

Document likelihood looks at how likely a query language model $M_q$ is to generate a document $d$:

In theory, it’s a symmetric idea of query likelihood. In practice, though, it’s not well suited for direct document ranking. Queries are typically short, providing too little data to estimate a reliable query model. As a result, the model becomes unstable and performs poorly when used on its own.

The upside is that this approach naturally supports relevance feedback. We can expand the query model with terms from documents judged relevant, making it easier to refine the query representation.

Model Comparison

Another approach builds a language model for both the document and the query, then measures how different the two models are. One common way to quantify this difference (or “risk”) of returning document $d$ for query $q$ is the Kullback-Leibler (KL) divergence:

The greater the divergence, the less the document model resembles the query model, and the poorer the match.

A limitation of KL-based scoring is that the scores aren’t comparable across different queries. This is fine for ad hoc retrieval, where we only care about ranking documents within a single query. But in applications like topic tracking, where we need one global threshold to decide whether a document belongs to a topic across many queries, the lack of score comparability becomes a significant drawback.

LM Similarities Supported by Elasticsearch

Elasticsearch, a widely used search engine built on top of Lucene, provides built-in language-model-based similarity modules that support both Jelinek Mercer smoothing and Dirichlet Prior smoothing.

Below is a minimal example (tested on Elasticsearch 8.19) showing how to set up an index that applies different scoring strategies to identical text.

The index declares three text fields, each tied to a different similarity:

PUT /my-index
{
  "settings": {
    "index": {
      "similarity": {
        "jm_similarity": {
          "type": "LMJelinekMercer",
          "lambda": "0.1"
        },
        "dirichlet_similarity": {
          "type": "LMDirichlet",
          "mu": 2000
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "bm25": {
        "type": "text"
      },
      "jm": {
        "type": "text",
        "similarity": "jm_similarity"
      },
      "dirichlet": {
        "type": "text",
        "similarity": "dirichlet_similarity"
      }
    }
  }
}

• bm25 uses the default BM25 similarity.
• jm uses Jelinek–Mercer smoothing.
• dirichlet uses Dirichlet prior smoothing.

By indexing the same content into three fields, you can directly compare how each method ranks the same documents for the same query.

Add sample documents:

PUT /my-index/_doc/1
{
  "bm25": "foo bar baz",
  "jm": "foo bar baz",
  "dirichlet": "foo bar baz"
}

PUT /my-index/_doc/2
{
  "bm25": "Lorem ipsum dolor sit amet",
  "jm": "Lorem ipsum dolor sit amet",
  "dirichlet": "Lorem ipsum dolor sit amet"
}

This query searches the dirichlet field for the term “foo”:

GET my-index/_search
{
    "query": {
        "match": {
          "dirichlet": "foo"
        }
    }
}

Document 1 matches, but the key point is the scoring behavior. Even though the underlying text is identical across fields, each similarity computes a different score:

Conclusion

According to Ponte and Croft (1998), the language modeling approach outperformed their baseline tf-idf based term weighting approach. They pointed out that the language modeling approach gives a new way to score how well a query matches a document. The idea is that by using a probabilistic model of language, we can assign better weights to terms and, as a result, improve retrieval performance. The major issue is estimation of the document model, such as choices of how to smooth it effectively [An Introduction to Information Retrieval].

Despite many advances in probabilistic IR, BM25 still remains competitive and often hard to beat. In practice, BM25 and query likelihood models tend to perform similarly. However, BM25 has a clear practical advantage because it lets us directly and intuitively control how term frequency and document length affect the final score.

References

• An Introduction to Information Retrieval by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze
• Text Retrieval and Search Engines by University of Illinois Urbana-Champaign
• Language Models in Elasticsearch - Elastic blog
• Similarity module - Elasticsearch

• Different workloads could not easily share the same hardware.
• Hardware was often idle.
• Testing a new OS or switching OSes required another machine or a reboot.

Virtualization was created to solve these problems. With virtualization, multiple operating systems and applications can run at the same time on the same hardware. Each one acts like it has its own machine, without forcing everyone to use the same operating system.

Virtual Machines and the Hypervisor

Virtual Machines

An early definition of a virtual machine is:

A virtual machine is taken to be an efficient, isolated duplicate of the real machine. –Popek and Goldber, 1974

A virtual machine (VM) is a software-defined machine that shares the physical CPU, memory, and devices. A VM is also called a guest or a domain.

The operating system running inside a VM is called the guest operating system (guest OS). From the guest operating system’s point of view, it is running on a real machine. Therefore, the guest OS does not know (and ideally does not care) that it is sharing physical resources with other VMs.

Hypervisors

To run multiple VMs safely, you need a control layer that manages hardware and keeps VMs apart. This layer is called virtual machine monitor (VMM), or hypervisor.

In most virtualization literature, VMM means virtual machine monitor (the hypervisor). But, some products also use VMM to mean virtual machine manager (a management tool).

A hypervisor is software that sits between the physical hardware and the virtual machines. It is responsible for mainly three goals:

1. Fidelity: The VM should look like a real machine. The guest OS should not need special changes just to boot and run.
2. Performance: The hypervisor should not slow down normal execution.
3. Safety: The hypervisor must control shared resources.

In general, the hypervisor stays out of the way during ordinary execution. It mainly steps in when a guest tries to do something that must be controlled, such as:

• Accessing privileged CPU registers
• Changing page tables
• Performing certain I/O operations

Benefits and Drawbacks

Virtualization is widely used because it has multiple benefits:

• Consolidation: Run many servers on one machine to increase utilization.
• Cost reduction: Fewer machines means less power, cooling, and hardware cost.
• Portability: A VM is easy to copy, move, or back up.
• Fast provisioning: You can create new VMs quickly from templates.
• Maintenance and migration: You can move VMs during hardware maintenance or load changes.
• Fault isolation: A crash in one VM does not have to crash others.
• Testing and debugging: OS experiments are easier when you can snapshot and roll back.
• Legacy support: Old OSes can run without dedicating old hardware.

However, virtualization also has drawbacks:

• Performance overhead compared to running directly on hardware.
• Increased system complexity and management overhead.
• Limited direct resource sharing due to isolation.
• Host failure can impact many VMs unless redundancy is used.
• Licensing and virtualization platform costs may be significant.

Types of VMs

VMM implementations differ widely. Common approaches are:

• Type 1
• Type 2
• Paravirtualization
• Containerization

In some systems like IBM LPARs and Oracle LDOMs, virtualization support is built directly into the hardware or firmware, rather than implemented as a traditional software hypervisor. These systems are often called Type 0 hypervisors.

Type 1: Bare-metal hypervisor

In type 1, the hypervisor runs directly on the hardware, without a host operating system underneath. It has full control over the machine and usually provides better performance and stronger isolation. The VMs run on top of hypervisor.

Type 1 is common in data centers and cloud platforms.

Examples of type 1 virtualization are:

• VMware ESXi
• XenServer
• Linux KVM
• Microsoft Hyper-V

Type 2: Hosted virtualization

In type 2, a normal host OS runs on the hardware first. The virtualization software runs on top of that OS. The host OS then supplies device drivers and many I/O services. And the hypervisor is hosted by the OS.

Type 2 is common on personal computers and for development or testing.

Examples of type 2 virtualization are:

• VMware Fusion and Workstation
• Oracle VirtualBox
• Parallels Desktop

Paravirtualization

Paravirtualization does not try to fully imitate real hardware. Instead, the guest OS is modified to work in cooperation with the VMM to optimize performance.

In a paravirtualized system, the guest kernel replaces some privileged operations with hypercalls. A hypercall is like a system call, but instead of entering the OS kernel, it enters the hypervisor. Examples of privileged operations that are commonly done via hypercalls include:

• Updating page tables
• Configuring interrupts
• Performing device I/O operations

Paravirtualization can be faster because hypercalls avoid replying on traps or instruction rewriting. The downside is that the guest OS must be modified and maintained. It can also be less portable, because it depends on a particular hypervisor interface.

XenServer popularized this approach.

Containerization

Sometimes the goal is not to run a full guest OS, but to isolate and manage just applications. If all applications run on the same OS, full virtualization may be unnecessary. Instead, the system can use containerization (or application containment).

A container is an isolated environment for running applications while sharing the host OS kernel. Unlike a virtual machine, a container does not include a guest OS and does not virtualize hardware. Instead, it bundles only what the application needs to run, such as the application binaries, libraries, and configuration files.

Compared to other virtualization approaches, its main advantage is efficiency. Containers are much lighter than VMs. They use fewer resources and can start and stop quickly—more like processes than full virtual machines.

Containerization is popular in software development and deployment, particularly in cloud systems. It is commonly managed with tools like Docker and Kubernetes.

Building Blocks

Whether virtualization is possible basically depends a lot on the CPU’s features.

To virtualize a system, the hypervisor must provide an exact duplicate of the underlying machine. This is especially difficult on older systems that support only two CPU modes: user mode and kernel mode.

For example, classic x86 defines four privilege levels, called rings:

• Ring 0: most privileged, used by the OS kernel
• (Rings 1 and 2 exist, but they are rarely used.)
• Ring 3: least privileged, used by user programs

On CPUs with only these privilege rings, it’s hard to fit a hypervisor in the right place. The hypervisor needs to run with the highest privileges so it can keep control. But a normal OS is written assuming it also runs with the highest privileges (ring 0). If you push the guest OS down to a less-privileged ring (like ring 3), some operations won’t behave the same way, and the hypervisor may not be able to catch and handle them correctly.

To fix this limitation, modern x86 CPUs add hardware support for virtualization (Intel VT-x and AMD-V). They introduce two new modes:

• Root mode: used by the hypervisor
• Non-root mode: used by guest operating systems

With this setup, the guest kernel can still run in ring 0, but it runs in non-root mode. In other words, it looks like ring 0 to the guest, but the CPU still keeps it on a leash.

When the guest executes certain sensitive instructions, the CPU automatically traps into the hypervisor. Control switches from non-root to root mode, the hypervisor handles the request, and then the CPU returns to the guest.

On top of this hardware foundation, VMMs use techniques such as trap-and-emulate and binary translation to handle privileged behavior efficiently.

Another important concept is the virtual CPU (vCPU). A vCPU does not execute instructions by itself. Instead, it stores what the guest thinks the CPU state is, including things like:

• Register values
• Program counter
• CPU flags

For each VM, the VMM keeps one or more vCPU data structures. When the VMM schedules a guest to run on a real CPU core, it does a simple save/restore cycle:

1. Load: copy the saved CPU state from the vCPU into the real CPU (registers, instruction pointer, flags, etc.).
2. Run: let the guest execute for a time slice, until it traps, blocks, or gets preempted.
3. Save: copy the updated CPU state back into the vCPU so the guest can resume later from the same point.

Trap-and-Emulate

On a typical CPU with only two modes (user and kernel) without extra hardware support, the guest kernel can execute only in user mode. Because the VM must imitate the physical machine, the VMM provides virtual modes:

• Virtual user mode
• Virtual kernel mode

Both virtual modes still execute in physical user mode.

For example, the following actions cause a transfer from user mode to kernel mode on a real machine:

• a system call
• an interrupt
• a privileged instruction

When the same actions occur inside the guest kernel, the VM must also transfer from virtual user mode to virtual kernel mode. This is done using trap-and-emulate:

1. The guest kernel executes a privileged instruction.
2. Because the guest kernel is in physical user mode, it causes a trap to the VMM (in the real machine).
3. The VMM executes (or emulates) the privileged instruction of the guest kernel.
4. It then returns control to the virtual machine.

The downside is overhead. Nonprivileged instructions run at near native speed, but privileged instructions trap to the VMM and must be emulated. Sharing the CPU across multiple VMs can also make performance unpredictable.

However, as hardware has improved, trap-and-emulate has gotten faster, and the system needs it less often. For example, modern CPUs add extra modes for virtualization beyond the typical user and kernel modes. This lets the CPU handle more of the work directly at the hardware level, so the VMM has less to do.

Binary Translation

Trap-and-emulate works only on CPUs where all sensitive instructions trap when executed in unprivileged mode. If some sensitive instructions can run without trapping, trap-and-emulate alone is not sufficient.

Binary translation fixes this by rewriting guest kernel code on the fly. Assume there are some sensitive instructions that do not automatically trap. Call them special instructions. Then:

• If the guest vCPU is in virtual user mode, the guest can run its instructions natively on a physical CPU.
• If the guest vCPU is in virtual kernel mode, the VMM inspects every instruction by the guest:
  • Non-special instructions run natively.
  • Special instructions are translated (rewrited) into safe sequences that produce the same effect

Even though most instructions run natively, special instructions still add overhead because they must be translated. To reduce this cost, the VMM saves translated code in a translation cache. When the same code runs again, the VMM can reuse the cached translation instead of translating it again.

Core Operating-System Components in Virtualization

CPU Scheduling

The effect of virtualization on scheduling varies a lot. This is because virtualization can make even a single-CPU system look like a multiprocessor sytem. For example, even if a host has six physical CPUs, the VMM can present twelve vCPUs across its guests. This implies:

• If there are enough physical CPUs to satisfy each guest’s requested vCPUs, the VMM can treat the allocation as effectively dedicated. It can schedule only a given guest’s threads on that guest’s assigned CPUs. In this situation, the guests behave much like native operating systems running on native hardware.
• If the total number of requested vCPUs exceeds the available physical CPUs (overcommitment), the VMM relies on a standard scheduling algorithm so that guests continue to perceive adequate CPU availability. For instance, with six physical CPUs and twelve vCPUs total, the VMM can distribute CPU time proportionally, so each guest effectively receives about half of the host’s CPU capacity.

Under CPU overcommitment, some performance degradation is expected because vCPUs must time-share physical CPUs and may spend time waiting to be scheduled.

Memory Management

VMMs typically overcommit memory, so the total memory allocated to guests can exceed the amount of physical memory available. This increases memory pressure because there are more guest operating systems and their applications, plus the VMM itself. Therefore, the VMM must preserve the illusion that the guest has that amount of memory, even when physical memory is smaller. To do that:

1. The VMM first evaluates each guest’s maximum memory size so that the guest OSes continue to believe they have what they asked for.
2. The VMM then computes how much memory to actually give it right now based on current load and overcommitment.
3. The VMM then reclaims memory from the guests when the host system is running low on memory.

For reclaiming memory, the most commonly used technique is memory ballooning, developed by VMware. Here, a balloon driver is loaded into the guest as a pseudo-device driver.

A pseudo–device driver uses device-driver interfaces, appearing to the kernel to be a device driver, but does not actually control a device. Rather, it is an easy way to add kernel-mode code without directly modifying the kernel.

The role of a balloon driver is to make the guests aware of the low memory status of the host. It does this by polling the hypervisor and getting a target balloon size. Without this driver, the VM cannot detect host memory pressure because the VM is isolated from the host. When the host is low on memory:

1. The hypervisor sets a target balloon size based on how much memory it wants to reclaim.
2. The balloon driver “inflates” the balloon inside the guest. It allocates and pins that amount of guest physical memory to the balloon. This makes the guest OS treat those pages as in use.
3. The balloon driver reports the pinned page frame numbers to the hypervisor. The hypervisor can then reclaim the host physical pages backing those frames.

When the host has enough memory again, the hypervisor “deflates” the balloon, and the balloon driver releases pages back to the guest.

The key benefit of ballooning is that it lets the guest operating system decide which pages to page out. The hypervisor does not need to be involved in that decision.

Storage Management

In a virtualized system, storage must be handled differently from a native operating system.

The exact approach depends on the type of hypervisor:

• Type 1 hypervisors store each guest’s root disk and configuration data as one or more files in file systems managed by the VMM.
• Type 2 hypervisors store the same information as files in the host operating system’s file systems.

In practice, a guest’s entire root disk is packaged into a single disk image file managed by the VMM. Although this design can introduce performance overhead, it is highly practical. It makes copying, backing up, and moving virtual machines straightforward. For example, if an administrator wants to create a duplicate guest for testing, they can simply copy the disk image file and register the new copy with the VMM. Similarly, moving a virtual machine between systems running the same VMM is easy: shut down the guest, copy the disk image to the new system, and start the guest there.

Many VMMs also support converting systems between physical and virtual systems:

• Physical-to-virtual (P-to-V) conversion: converts a running physical machine into a virtual machine. This is done by copying its disk contents into virtual disk image files.
• Virtual-to-physical (V-to-P) conversion: converts a virtual machine into a physical system. This is useful for debugging a problem where the virtualization layer itself may be contributing to the issue.

I/O Management

I/O virtualization is easier than CPU or memory virtualization because operating systems already handle many kinds of devices through device drivers. A hypervisor uses this by providing specific virtualized devices to guests.

Common ways a VMM handles I/O:

• Pass-through (dedicated device): one real device is directly dedicated to one guest. This can be close to native speed, but other guests cannot use that device.
• Shared device: many guests share one real device. The VMM must sit in the middle for every I/O to keep guests isolated and send data to the right place.
• Split-device: the guest uses a simple driver that talks to the VMM, and the VMM talks to the real hardware. This is often faster and easier to manage than fully emulating real devices.

Direct access can be fast in type 0 hypervisor, but type 1 and type 2 hypervisors usually need hardware support. Without it, I/O slows down because the hypervisor must handle many extra events, especially interrupts.

For networking, each guest must have at least one IP address to communicate with other systems. The VMM then acts like a virtual switch to route the packets to the right guest. Guests can connect using:

• Bridging: the guest appears directly on the external network with its own IP.
• NAT: the guest uses an internal IP, and the VMM translates and forwards traffic.

VMMs often also provide basic firewall rules to control traffic between guests and to the outside network.

I/O Devices

I/O devices cover a wide range of hardware. Common examples are storage devices such as SSDs, network interface cards (NICs), and devices used for human interaction, including displays, keyboards, speakers, and microphones.

Since I/O devices behave very differently from one another, the operating system needs to hide these differences from applications. To make this separation possible, operating systems rely on two key ideas:

• Standard interfaces
• Device drivers

Standard Interfaces

Because I/O devices are very different, applications cannot communicate with them directly. Instead, the operating system defines standard interfaces for broad classes of devices, such as storage devices, network devices, or display devices. These interfaces define what operations a device supports (for example, read, write, send, receive) and how those operations are requested. Applications interact only with these interfaces, not with the hardware itself. This allows the same application code to work with many different devices of the same type.

At a lower level, devices must physically connect to the CPU and memory. In modern systems, this is done using device interconnects, most commonly PCI Express (PCIe).

PCIe provides a high-speed, point-to-point connection between devices and the CPU. Unlike the older shared PCI bus, PCIe uses dedicated links (lanes) for each device. Through PCIe, devices can exchange data with the CPU, memory, and the rest of the system in a standardized way.

Other types of buses are also used even in a PCIe system, depending on the device. For example:

• SATA controllers for certain storage devices
• Simpler peripheral buses (such as USB) for devices like keyboards and mice

The choice of bus affects performance, how devices are discovered at boot time, and how the operating system manages and schedules access to hardware.

Device Drivers

A device driver is software that knows how to control a specific hardware device. Each driver contains all the device-specific code needed to configure the device, send commands to it, move data in and out, and respond to events such as interrupts or errors.

When an application makes an I/O request through a standard interface, the operating system forwards that request to the appropriate driver. This design allows the operating system and applications to remain unchanged even when hardware is replaced, as long as the correct driver is available. From the operating system’s perspective, supporting a new device usually means adding a new driver, not rewriting the rest of the system.

Typically, the operating system defines a standard device driver framework, and hardware manufacturers provide drivers that fit into this framework. This is why users often need to install drivers when adding new hardware, such as printers.

At the hardware level, a device controller contains built-in logic and firmware that directly run the physical device. It receives commands from the device driver, performs the requested actions, transfers data between the device and memory, and signals completion or errors via interrupts.

Devices as Files

Many operating systems represent devices as files. This means applications can interact with devices using familiar operations such as read and write.

On Unix-like systems, devices appear as special files under the /dev directory. For example, running echo "hello, world" > /dev/lp0 sends the text hello, world to the first line printer (lp0). Although this looks like a normal file write, the kernel intercepts the operation and translates it into the appropriate device-specific commands handled by the printer driver.

Unix-like operating systems provide a virtual device known as the null device, exposed as /dev/null. Any data written to this device is discarded by the kernel.

Typical Device Access Flow

Typically, user programs cannot access hardware devices directly. Instead, device access goes through the operating system:

1. When a process needs to use a device—such as sending a network packet or reading a disk file—it makes a system call. This transfers control from the process to the kernel.
2. Inside the kernel, the operating system runs the relevant internal code:
   • Networking requests go through the TCP/IP stack.
   • File requests go through the file system to locate disk blocks.
   • The kernel converts the high-level request into a low-level device operation.
3. The kernel then calls the device driver, which understands the hardware details. The driver:
   • Translates requests into device-specific commands
   • Configures the device by writing to its registers
   • Manages pending requests safely
4. The driver uses mechanisms like programmed I/O or DMA to transfer data. Once configured, the device performs the operation, such as transmitting data or reading from disk.
5. When the operation finishes, the result travels back through the driver and kernel to the user process.

OS Bypass

Some devices support a different model in which applications can access hardware without going through the kernel on every operation. This is called operating system bypass (OS bypass).

In this model, the operating system is involved only during setup. It maps device memory and registers into the process’s address space and establishes permissions. After setup, the data path goes directly between the user process and the device.

Because the kernel is no longer in the data path, applications use a user-level driver, typically a library provided by the device vendor, to issue device commands.

Blocking, Nonblocking, and Asynchronous I/O

When a process issues an I/O request, the device will eventually return some response. What happens to the calling thread depends on whether the I/O is synchronous or asynchronous.

An important design choice in the system-call interface is how I/O operations interact with a running program. The key question is whether the calling thread should wait for I/O to finish or continue running while the I/O happens in the background.

Blocking (synchronous) I/O

In blocking I/O, a system call pauses the calling thread until the I/O operation completes.

Here is what happens step by step:

1. The application issues a blocking system call (for example, read()).
2. The operating system suspends the calling thread.
3. The thread is moved from the run queue to a wait queue.
4. When the I/O finishes, the thread is moved back to the run queue.
5. When the thread runs again, it receives the return values from the system call.

Although I/O devices work asynchronously at the hardware level and may take an unpredictable amount of time, operating systems still offer blocking calls. The reason is simple: blocking code is easier to write and reason about.

Nonblocking I/O

Some operating systems provide nonblocking I/O system calls. These calls do not suspend the thread for a long time.

A nonblocking call:

• Returns immediately.
• Reports how many bytes were transferred.
• May return fewer bytes than requested, or even zero, if no data is available yet.

The thread can then decide whether to try again later or do something else. This avoids wasting CPU time and allows computation to overlap with I/O.

One way to handle this problem is multithreading. An application can create multiple threads:

• Some threads perform blocking I/O.
• Other threads continue running useful work.

This approach works well but increases complexity due to synchronization and shared data management.

Asynchronous I/O

Another option is asynchronous I/O, which goes one step further.

With an asynchronous system call:

• The call returns immediately.
• The operating system guarantees that the entire I/O request will be completed later.
• The thread continues executing without waiting.

When the I/O finishes, the operating system notifies the application using one of several methods:

• Updating a variable in the process’s address space
• Sending a signal or software interrupt
• Invoking a callback function

Asynchronous operations are common inside modern operating systems, even when applications are not directly aware of them.

Block Device Stack

The block device stack describes how an I/O request follows from an application down to the physical storage device.

At the top of the stack are applications.= As explained eariler, applications do not work with disks directly. Instead, they operate on files, which are logical objects provided by the operating system. Programs use system calls such as open, read, and write to access files without knowing where or how the data is stored.

These system calls are handled by the file system. The file system manages files and directories, checks permissions, and translates file-level operations into accesses to disk blocks. Before accessing the disk, the operating system usually checks the page cache. If the requested data is already in memory, it can be returned immediately. If not, the request continues down the stack.

Below the file system and page cache is the generic block layer. This layer provides a common interface for different block devices such as HDDs, SSDs, and USB drives. Even though these devices work differently internally, the block layer hides those differences and schedules read and write requests.

At the bottom of the stack are the device drivers. Drivers communicate directly with the hardware and handle device-specific commands, errors, and interrupts.

A typical storage request flows down the stack as follows:

1. Application calls read or write
2. Operating system enters the file system. If data is present in page cache, return immediately.
3. File system maps the request to disk blocks
4. Block layer schedules the request
5. Device driver accesses the hardware

Virtual File System

Modern operating systems support many different kinds of storage. Files may live on different disks, use different file system formats, or even be stored on remote drives and accessed over a network. Despite this complexity, applications should not need to know where a file is stored or how it is managed internally.

To solve this problem, operating systems such as Linux use a Virtual File System (VFS) layer. The VFS sits between applications and the actual file systems. Applications interact with files using a standard interface, such as the POSIX API, and the VFS takes care of translating these requests to the appropriate underlying file system.

This separation makes it easy to add new file systems or switch storage devices without modifying application code.

To support many file systems in a uniform way, the VFS defines a small set of common abstractions.

Files and File Descriptors

A file is the basic object that the VFS operates on. When a process opens a file, the operating system creates a file descriptor, which is a small integer used to identify that open file.

All file operations—such as reading, writing, locking, or closing—are performed using the file descriptor. The file descriptor exists only while the file is open and is specific to the process that opened it.

Inodes

An inode (short for index node) is the core data structure a file system uses to manage files. Internally, each file is identified by an inode number, not by its file name. The file name is just a human-friendly label used to locate the corresponding inode.

An inode contains a list of disk block numbers that store the file’s actual data. In this sense, the inode acts as an index: it does not store the file contents itself, but instead stores pointers to the blocks that hold the contents. Reading those blocks in the correct order reconstructs the file.

When more data is written to a file and extra space is needed, the file system:

1. Allocates a free disk block
2. Adds its block number to the inode
3. Updates the inode on disk

The inode always reflects the current layout of the file.

Besides block pointers, an inode also stores metadata such as:

• Access permissions
• Ownership
• Locking or status information

This metadata is used to enforce access rules and manage concurrent use.

This inode-based design supports efficient access patterns. Sequential access simply follows the list of blocks in order, while random access directly computes the required block from the file offset.

The main drawback of this simple design is that it limits how large a file can be. Consider this example:

• Inode size: 128 bytes
• Each block pointer: 4 bytes
• Maximum number of block pointers: 128 / 4 = 32
• Block size: 1 KB

With only direct block pointers and no space for metadata, the largest file that can be addressed is 32 blocks * 1 KB = 32 KB.

A common way to overcome the file-size limitation of simple inodes is to use indirect pointers.

An inode still begins with metadata, followed by pointers to data blocks. The first pointers are direct pointers, which point straight to blocks containing file data.

An indirect pointer does not point to data directly. Instead, it points to a block that contains only block addresses. A 1 KB block can store 256 such addresses, allowing a single indirect pointer to reference 256 KB of file data. For larger files, double or even triple indirect pointers are used. A double indirect pointer, for example, points to a block of pointers to other pointer blocks, which then point to data blocks. With 256 pointers per block, double indirect addressing supports up to 64 MB of data.

The trade-off of using indirect pointers is that it increases file size but also increases access cost.

Directories and Dentires

Files are organized into directories. From the VFS perspective, a directory is simply a special kind of file. The difference is that the contents of a directory describe file names and the inodes they refer to.

Dentires are cached in memory in what is known as the dentry cache. This cache avoids repeated disk accesses when the same directories are accessed multiple times. Dentires are not stored on disk. They are temporary objects maintained only in memory.

Superblock

Every file system has a superblock, which describes how the file system is laid out on disk. The superblock contains information such as:

• Where inodes are stored
• Where data blocks are located
• How free space is managed

The superblock acts as a map that allows the operating system to interpret the on-disk data structures correctly.

Many VFS structures, such as file descriptors and dentries, exist only in memory. Others, including file data, inodes, and superblocks, must be stored persistently on disk.

A widely used modern file system in Linux is ext4 (4th extended file system), which is the successor to ext2 and ext3.

Disk Access Optimizations

Disk operations are much slower than memory operations, so file systems use several techniques to reduce disk access and improve I/O performance. These techniques focus on keeping data in memory, reducing disk head movement, and avoiding unnecessary random accesses.

Key techniques include:

• buffer caching: Keep recently accessed file data in memory so most reads and writes avoid disk access; changes are flushed to disk periodically.
• I/O scheduling: Reorder disk requests to minimize disk head movement and favor sequential access over random access.
• prefetching: Load nearby file blocks into memory in advance, increasing cache hits and reducing future access latency.
• journaling: Record updates in a sequential log before writing them to their final disk locations, improving reliability while limiting random disk writes.

This post is a comprehensive walkthrough of deploying a self-managed Elasticsearch cluster using one of the supported installation methods: Docker. It also aims to follow production best practices and recommendations.

Planning Elasticsearch Cluster

By the end of this tutorial, you’ll have the following setup in place:

We will deploy:

• A production cluster with:
  • Three master nodes, as there should normally be an odd number of master-eligible nodes in a cluster.
  • Three data nodes
• A dedicated monitoring cluster with Kibana and Elastic Agent
• An optional load balancer in front of the production cluster.

About Node Roles

Here’s a quick note on node roles in this setup:

• master: Responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes.
• data: Responsible for holding data and performing data related operations such as CRUD, search, and aggregations.

We won’t use coordinating only nodes in this setup, since data nodes can happily serve the same purpose. For more details on node roles, refer to Node roles.

Why Use a Dedicated Monitoring Cluster?

Using a dedicated monitoring cluster is the recommended configuration for two key reasons:

• Production outages do not impact access to monitoring data.
• Monitoring workloads are isolated and cannot degrade production cluster performance.

Elastic also recommends using a separate Kibana instance to view a separate monitoring cluster.

For the sake of operational simplicity, this setup uses a single-node monitoring cluster. When operating at scale, however, you can also configure a more resilient multi-node monitoring cluster by following this tutorial.

Infrastructure Setup

You can skip this part if you already have servers ready to host the Elasticsearch cluster.

This tutorial has been demonstrated using the following setup:

• Elastic Stack version: 8.19.7.
• VPN access: A VPN for secure configuration and access to the Elasticsearch cluster, using the CIDR block 10.0.0.0/8

Create a Security Group

Log in to the AWS Management Console and navigate to EC2 -> Security Groups -> Create security group.

Create a security group named my-elasticsearch-node for the Elasticsearch nodes, and configure the following inbound rules:

Create a Launch Template

Create a Launch Template to make it easier to add multiple nodes to the cluster.

Go to EC2 → Launch Templates → Create launch template, then configure the following:

• Launch template name: my-elasticsearch-node
• Launch template contents:
  • Amazon Machine Image (AMI): Ubuntu Server 24.04 LTS (HVM), SSD Volume Type
  • Architecture: 64-bit (Arm)
• Network settings:
  • Security groups: my-elasticsearch-node
• Storage:
  • Size: 100 GiB

This configuration is just an example. You can adjust the values to match your own requirements. Just make sure to select the security group created in the previous step, that is my-elasticsearch-node.

Launch Instances

1. Go to EC2 -> Instances -> Launch instance from template and configure the following:
   • Source template: my-elasticsearch-node
   • Instance type: (Your choice. For this tutorial, the monitoring node requires at least 8 GiB of RAM.)
   • Key pair (login): Select your key pair.
   • Resource tags:
     • Key: Name (for identifying instances)
     • Value: my-elasticsearch-node
   • Number of instances: 7 (6 for the production cluster and 1 for the monitoring node)
2. Click Launch instance
3. Go to EC2 -> Instances -> Search my-elasticsearch-node and update the Name tags as follows:
   • my-elasticsearch-node-master-1
   • my-elasticsearch-node-master-2
   • my-elasticsearch-node-master-3
   • my-elasticsearch-node-data-1
   • my-elasticsearch-node-data-2
   • my-elasticsearch-node-data-3
   • my-elasticsearch-node-monitoring
4. Retrieve the private IP addresses of all EC2 instances using the AWS CLI:

    aws ec2 describe-instances \
        --filters "Name=tag:Name,Values=my-elasticsearch-node-*" \
        --query "Reservations[].Instances[].[Tags[?Key=='Name'].Value | [0], PrivateIpAddress]" \
        --output text \
        --no-cli-pager
   

   Output:

    my-elasticsearch-node-master-1   10.0.0.1
    my-elasticsearch-node-master-2   10.0.0.2
    my-elasticsearch-node-master-3   10.0.0.3
    my-elasticsearch-node-data-1     10.0.1.1
    my-elasticsearch-node-data-2     10.0.1.2
    my-elasticsearch-node-data-3     10.0.1.3
    my-elasticsearch-node-monitoring        10.0.2.1
   

5. Update SSH config file (usually ~/.ssh/config) to connect to the instances from your local environment:

    Host my-elasticsearch-node-*
        User ubuntu
        IdentityFile your_key.pem
    Host my-elasticsearch-node-master-1
        HostName 10.0.0.1
    Host my-elasticsearch-node-master-2
        HostName 10.0.0.2
    Host my-elasticsearch-node-master-3
        HostName 10.0.0.3
    Host my-elasticsearch-node-data-1
        HostName 10.0.1.1
    Host my-elasticsearch-node-data-2
        HostName 10.0.1.2
    Host my-elasticsearch-node-data-3
        HostName 10.0.1.3
    Host my-elasticsearch-node-monitoring
        HostName 10.0.2.1
   

The IP addresses used in this tutorial are simplified for clarity. Your IP addresses will differ depending on your VPC and subnet configuration.

Deploying an Elasticsearch cluster

Initializing the Cluster with Master Nodes

1. Connect to my-elasticsearch-node-master-1:

    ssh my-elasticsearch-node-master-1
   

2. Install Docker:

    sudo apt-get update
    sudo apt-get install -y ca-certificates curl
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
    echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
        sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo usermod -aG docker ${USER}
   

3. Set vm.max_map_count to at least 262144:

    sudo sysctl -w vm.max_map_count=262144
    # Add `vm.max_map_count` to `sysctl.conf`
    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
   

   • This is to increase the limits on mmap counts to avoid out of memory exceptions:
4. Prepare local directories for storing data and logs through a bind-mount:

    mkdir esdatadir && chmod g+rwx esdatadir && sudo chgrp 0 esdatadir
    mkdir eslogsdir && chmod g+rwx eslogsdir && sudo chgrp 0 eslogsdir
   

5. Prepare a certificate directory:

    mkdir certs && chmod g+rwx certs && sudo chgrp 0 certs
   

6. Generate CA (only if it doesn’t exist - first node should create and share with others):

    sudo apt-get install -y unzip
    sudo docker run --rm \
        -v $HOME/certs:/usr/share/elasticsearch/config/certs \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7 \
        bin/elasticsearch-certutil ca --silent --pem --out /usr/share/elasticsearch/config/certs/ca.zip
    cd $HOME/certs && unzip -o ca.zip && mv ca/* . && rmdir ca && rm ca.zip
   

7. Generate node certificate signed by CA:

    ES_NETWORK_HOST=$(hostname -I | awk '{print $1}')
    sudo docker run --rm \
        -v $HOME/certs:/usr/share/elasticsearch/config/certs \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7 \
        bin/elasticsearch-certutil cert --silent --pem \
        --ca-cert /usr/share/elasticsearch/config/certs/ca.crt \
        --ca-key /usr/share/elasticsearch/config/certs/ca.key \
        --dns master-1,localhost \
        --ip ${ES_NETWORK_HOST},127.0.0.1 \
        --out /usr/share/elasticsearch/config/certs/node.zip
    cd $HOME/certs && unzip -o node.zip && mv instance/* . && rmdir instance && rm node.zip
   

8. Create a custom elasticsearch.yml in your home directory with important settings:

    cluster.name: my-elasticsearch-cluster
    node.roles: master
    node.name: master-1
    network.host: 0.0.0.0
    discovery.seed_hosts: 10.0.0.1,10.0.0.2,10.0.0.3
    cluster.initial_master_nodes: master-1,master-2,master-3
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.key: certs/instance.key
    xpack.security.transport.ssl.certificate: certs/instance.crt
    xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
   

   • The configuration files should contain settings which are node-specific (such as node.name and paths), or settings which a node requires in order to be able to join a cluster (such as cluster.name and network.host), as most settings can be changed on a running cluster using the Cluster update settings API.
   • cluster.name: A node can only join a cluster when it shares its cluster.name with all the other nodes in the cluster. The default name is elasticsearch, but you should change it to an appropriate name that describes the purpose of the cluster.
   • node.roles: We set the role to master to create a dedicated master-eligible node. This is the most reliable way to avoid overloading the master with other tasks.
   • node.name: Elasticsearch uses node.name as a human-readable identifier for a particular instance of Elasticsearch. The node name defaults to the hostname of the machine when Elasticsearch starts.
   • network.host: Sets the address of this node for both HTTP and transport traffic.
   • discovery.seed_hosts: When you want to form a cluster with nodes on other hosts, this setting provides a list of other nodes in the cluster that are master-eligible and likely to be live and contactable to seed the discovery process.
   • cluster.initial_master_nodes: A list of node names of master-eligible nodes whose votes are counted in the first election. Do not configure this setting on master-ineligible nodes or nodes joining an existing cluster.
   • xpack.security.enabled: Set true to install Elastic Agent for stack monitoring.
   • xpack.security.transport.ssl.enabled: Set true as transport SSL must be enabled if security is enabled.
   • For a full list of configuration options, refer to Elasticsearch configuration reference.
9. Run Elasticsearch container:

   sudo docker run -d \
       --restart always \
       --group-add 0 \
       --volume $HOME/esdatadir:/usr/share/elasticsearch/data \
       --volume $HOME/eslogsdir:/usr/share/elasticsearch/logs \
       --volume $HOME/certs:/usr/share/elasticsearch/config/certs:ro \
       --ulimit nofile=65535:65535 \
       --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
       --publish-all \
       --network host \
       --volume $HOME/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
       --env TZ=Asia/Seoul \
       --env ELASTIC_PASSWORD=elasticpassword \
       --name elasticsearch \
       docker.elastic.co/elasticsearch/elasticsearch:8.19.7
   

   • --restart always: to automatically restarts the container if it stops.
   • --group-add 0: to ensure that the user under which Elasticsearch is running has file permissions to esdatadir.
   • --volume $HOME/...: to persist data, logs, and certificates across container restarts.
   • --ulimit nofile=65535:65535: to increase the file descriptor limit required for production workloads.
   • --publish-all: to randomize published ports, which is recommended for production clusters.
   • --network host: to optimize performance. The host network only works on Linux hosts and --publish-all option is ignored when it is enabled.
   • --volume $HOME/...: to bind-mount custom elasticsearch.yml, the preferred approach for production setups.
   • --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1: to disable swapping for performance and node stability.
   • --env TZ=Asia/Seoul: Sets the container timezone (adjust as needed).
   • ELASTIC_PASSWORD: Replace it with a password for your production Elasticsearch cluster.
   • Some setups use ES_JAVA_OPTS, but this is not recommended for production. Use the default Elasticsearch memory settings instead.

At this stage, the cluster isn’t fully formed yet, so you’ll see a warning in the container logs like:

master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [master-1, master-2, master-3] to bootstrap a cluster: ...

You won’t see this error if you’re using only a single master-eligible node.

This happens because we configured three initial master nodes—master-1, master-2, and master-3—but only master-1 has joined the cluster so far. Elasticsearch is still waiting for the remaining master-eligible nodes before it can complete the bootstrap process. To finish forming the cluster, you need to bring up and join the other master-eligible nodes (master-2 and master-3) as well.

Joining Master Nodes to the Cluster

1. Copy certs/ca.crt and certs/ca.key files to your local machine or another secure location. These will be reused when setting up the remaining nodes.
2. Connect to my-elasticsearch-node-master-2:

    ssh my-elasticsearch-node-master-2
   

3. Copy ca.crt and ca.key into the home directory on this node.
4. Install docker, set vm.max_map_count, prepare local directories, and generate node certificate:

    # install docker
    sudo apt-get update
    sudo apt-get install -y ca-certificates curl unzip
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
    echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
        sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo usermod -aG docker ${USER}
   
    # Set `vm.max_map_count` to at least 262144
    sudo sysctl -w vm.max_map_count=262144
    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
   
    # Prepare a local directory for storing data and logs through a bind-mount
    mkdir esdatadir && chmod g+rwx esdatadir && sudo chgrp 0 esdatadir
    mkdir eslogsdir && chmod g+rwx eslogsdir && sudo chgrp 0 eslogsdir
   
    # Prepare certificate directory
    mkdir certs && chmod g+rwx certs && sudo chgrp 0 certs
   
    mv ca.crt certs/
    mv ca.key certs/
   
    # Generate node certificate signed by CA
    ES_NETWORK_HOST=$(hostname -I | awk '{print $1}')
    sudo docker run --rm \
        -v $HOME/certs:/usr/share/elasticsearch/config/certs \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7 \
        bin/elasticsearch-certutil cert --silent --pem \
        --ca-cert /usr/share/elasticsearch/config/certs/ca.crt \
        --ca-key /usr/share/elasticsearch/config/certs/ca.key \
        --dns master-2,localhost \
        --ip ${ES_NETWORK_HOST},127.0.0.1 \
        --out /usr/share/elasticsearch/config/certs/node.zip
   
    cd $HOME/certs && unzip -o node.zip && mv instance/* . && rmdir instance && rm node.zip
   

5. Create a custom elasticsearch.yml in your home directory:

    cluster.name: my-elasticsearch-cluster
    node.roles: master
    node.name: master-2
    network.host: 0.0.0.0
    discovery.seed_hosts: 10.0.0.1,10.0.0.2,10.0.0.3
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.key: certs/instance.key
    xpack.security.transport.ssl.certificate: certs/instance.crt
    xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
   

   • node.name: now set to master-2
6. Run Elasticsearch container:

    sudo docker run -d \
        --restart always \
        --group-add 0 \
        --volume $HOME/esdatadir:/usr/share/elasticsearch/data \
        --volume $HOME/eslogsdir:/usr/share/elasticsearch/logs \
        --volume $HOME/certs:/usr/share/elasticsearch/config/certs:ro \
        --ulimit nofile=65535:65535 \
        --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
        --publish-all \
        --network host \
        --volume $HOME/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
        --env TZ=Asia/Seoul \
        --env ELASTIC_PASSWORD=elasticpassword \
        --name elasticsearch \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7
   

   • --env TZ=Asia/Seoul: Sets the container timezone (adjust as needed).
   • ELASTIC_PASSWORD: Replace it with a password for your production Elasticsearch cluster.
7. Connect to my-elasticsearch-node-master-3 and repeat steps 13-16, making sure to set node.name to master-3 in the elasticsearch.yml file.
8. Verify that the master nodes have formed a cluster:

    curl -u elastic:elasticpassword "http://localhost:9200/_cat/nodes?v&s=name"
   

   Output:

    ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
    10.0.0.1                 40          98   0    0.00    0.05     0.07 m         *      master-1
    10.0.0.2                 26          98   0    0.07    0.17     0.10 m         -      master-2
    10.0.0.3                 24          98   0    0.39    0.31     0.13 m         -      master-3
   

Since there are no data nodes to host shards yet, the cluster health will show up as red:

curl http://elastic:elasticpassword@localhost:9200/_cluster/health

Output:

{
    "cluster_name": "my-elasticsearch-cluster",
    "status": "red",
    "timed_out": false,
    "number_of_nodes": 3,
    "number_of_data_nodes": 0,
    "active_primary_shards": 0,
    "active_shards": 0,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 2,
    "unassigned_primary_shards": 2,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 0.0
}

This is expected. Once the data nodes join the cluster, Elasticsearch will be able to allocate shards and the status should move to yellow or green depending on replica allocation.

Joining Data Nodes to the Cluster

The process for adding a data node is very similar to adding a master node, with a few small changes in the elasticsearch.yml configuration.

1. Connect to my-elasticsearch-node-data-1:

    ssh my-elasticsearch-node-data-1
   

2. Copy ca.crt and ca.key into the home directory on this node.
3. Install docker, set vm.max_map_count, prepare local directories, and generate node certificate:

    # install docker
    sudo apt-get update
    sudo apt-get install -y ca-certificates curl unzip
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
    echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
        sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo usermod -aG docker ${USER}
   
    # Set `vm.max_map_count` to at least 262144
    sudo sysctl -w vm.max_map_count=262144
    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
   
    # Prepare a local directory for storing data and logs through a bind-mount
    mkdir esdatadir && chmod g+rwx esdatadir && sudo chgrp 0 esdatadir
    mkdir eslogsdir && chmod g+rwx eslogsdir && sudo chgrp 0 eslogsdir
   
    # Prepare certificate directory
    mkdir certs && chmod g+rwx certs && sudo chgrp 0 certs
   
    mv ca.crt certs/
    mv ca.key certs/
   
    # Generate node certificate signed by CA
    ES_NETWORK_HOST=$(hostname -I | awk '{print $1}')
    sudo docker run --rm \
        -v $HOME/certs:/usr/share/elasticsearch/config/certs \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7 \
        bin/elasticsearch-certutil cert --silent --pem \
        --ca-cert /usr/share/elasticsearch/config/certs/ca.crt \
        --ca-key /usr/share/elasticsearch/config/certs/ca.key \
        --dns data-1,localhost \
        --ip ${ES_NETWORK_HOST},127.0.0.1 \
        --out /usr/share/elasticsearch/config/certs/node.zip
   
    cd $HOME/certs && unzip -o node.zip && mv instance/* . && rmdir instance && rm node.zip
   

4. Create a custom elasticsearch.yml in your home directory:

    cluster.name: my-elasticsearch-cluster
    node.roles: data
    node.name: data-1
    network.host: 0.0.0.0
    discovery.seed_hosts: 10.0.0.1,10.0.0.2,10.0.0.3
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.key: certs/instance.key
    xpack.security.transport.ssl.certificate: certs/instance.crt
    xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
   

   • node.roles: now set to data.
   • node.name: now set to data-1
5. Run Elasticsearch container:

    sudo docker run -d \
        --restart always \
        --group-add 0 \
        --volume $HOME/esdatadir:/usr/share/elasticsearch/data \
        --volume $HOME/eslogsdir:/usr/share/elasticsearch/logs \
        --volume $HOME/certs:/usr/share/elasticsearch/config/certs:ro \
        --ulimit nofile=65535:65535 \
        --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
        --publish-all \
        --network host \
        --volume $HOME/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
        --env TZ=Asia/Seoul \
        --env ELASTIC_PASSWORD=elasticpassword \
        --name elasticsearch \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7
   

   • --env TZ=Asia/Seoul: Sets the container timezone (adjust as needed).
   • ELASTIC_PASSWORD: Replace it with a password for your production Elasticsearch cluster.
   • Joining the cluster can take a little time. Once a data node joins the cluster, the health status transitions from red to green:

       {
           "cluster_name": "my-elasticsearch-cluster",
           "status": "green",
           "timed_out": false,
           "number_of_nodes": 4,
           "number_of_data_nodes": 1,
           "active_primary_shards": 3,
           "active_shards": 3,
           "relocating_shards": 0,
           "initializing_shards": 0,
           "unassigned_shards": 0,
           "unassigned_primary_shards": 0,
           "delayed_unassigned_shards": 0,
           "number_of_pending_tasks": 0,
           "number_of_in_flight_fetch": 0,
           "task_max_waiting_in_queue_millis": 0,
           "active_shards_percent_as_number": 100.0
       }
     

6. Repeat steps 1-5 for my-elasticsearch-node-data-2 and my-elasticsearch-node-data-3, updating node.name to match each node.
7. From any node, verify that all nodes have successfully joined the cluster:

    curl -u elastic:elasticpassword "http://localhost:9200/_cat/nodes?v&s=name"
   

   Output:

    ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
    10.0.1.1                 41          97   0    0.00    0.06     0.08 d         -      data-1
    10.0.1.2                 36          98   0    0.22    0.33     0.16 d         -      data-2
    10.0.1.3                 33          98   0    0.14    0.24     0.11 d         -      data-3
    10.0.0.1                  8          98   0    0.00    0.00     0.00 m         *      master-1
    10.0.0.2                 44          97   0    0.00    0.00     0.00 m         -      master-2
    10.0.0.3                 42          97   0    0.00    0.00     0.00 m         -      master-3
   

At this point, all master and data nodes should be present, and the cluster should be fully formed and healthy.

Updating custom elasticsearch.yml

Updating a custom elasticsearch.yml in a Docker-based setup is straightforward:

1. Make your changes in elasticsearch.yml in your home directory.
2. Restart the container:

    docker restart elasticsearch
   

Be aware that restarting the container will cause downtime. For production clusters, plan configuration changes carefully and apply them during a maintenance window, especially if the setting you’re changing affects cluster behavior or node roles.

Stack Monitoring

Stack Monitoring lets you collect logs and metrics from Elastic products, most importantly your production Elasticsearch nodes, and also things like Kibana. At a minimum, you need monitoring data for the production Elasticsearch cluster. Once that’s in place, Kibana can show monitoring data for other products in the Stack Monitoring page.

The legacy Elasticsearch Monitoring plugin approach (enabled via xpack.monitoring.collection.enabled) was deprecated in 7.16. These days, Elastic recommends using Elastic Agent or Metricbeat to collect and ship monitoring data to a monitoring cluster.

Next, we’ll deploy a single-node monitoring cluster and use Elastic Agent to monitor our production cluster.

Deploying Elasticsearch

1. Connect to my-elasticsearch-monitoring

    ssh my-elasticsearch-node-monitoring
   

2. Install docker, set vm.max_map_count, and prepare local directories for data persistency:

    # install docker
    sudo apt-get update
    sudo apt-get install -y ca-certificates curl
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
    echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
        sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo usermod -aG docker ${USER}
   
    # Set `vm.max_map_count` to at least 262144
    sudo sysctl -w vm.max_map_count=262144
    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
   
    # Prepare a local directory for storing data and logs through a bind-mount
    mkdir esdatadir && chmod g+rwx esdatadir && sudo chgrp 0 esdatadir
    mkdir eslogsdir && chmod g+rwx eslogsdir && sudo chgrp 0 eslogsdir
   

3. Create a custom elasticsearch.yml in your home directory:

    discovery.type: single-node
    xpack.security.enabled: true
    network.host: 0.0.0.0
   

4. Run Elasticsearch container:

    sudo docker run -d \
        --restart always \
        --group-add 0 \
        --volume $HOME/esdatadir:/usr/share/elasticsearch/data \
        --volume $HOME/eslogsdir:/usr/share/elasticsearch/logs \
        --ulimit nofile=65535:65535 \
        --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
        --publish-all \
        --network host \
        --volume $HOME/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
        --env TZ=Asia/Seoul \
        --env ELASTIC_PASSWORD=elasticpassword \
        --name elasticsearch \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7
   

   • --env TZ=Asia/Seoul: Sets the container timezone (adjust as needed).
   • ELASTIC_PASSWORD: Replace it with a password for your monitoring Elasticsearch node.
5. Verify:

    curl -u elastic:elasticpassword "http://localhost:9200/_cat/nodes?v&s=name"
   

   Output

    ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
    10.0.2.1                 22          97  44    0.92    0.31     0.11 cdfhilmrstw *      ip-10-0-2-1
   

   • Note that the node is assigned every role as it must do everything.

Deploying Kibana

1. Set a password for kibana_system user:

    curl -s -X POST -u "elastic:elasticpassword" -H "Content-Type: application/json" http://localhost:9200/_security/user/kibana_system/_password -d "{\"password\":\"kibanapassword\"}"
   

   • kibanapassword: Replace it with your password.
2. Create custom kibana.yml:

    server.host: "0.0.0.0"
    server.shutdownTimeout: "5s"
    elasticsearch.hosts: ["http://localhost:9200"]
    elasticsearch.username: "kibana_system"
    elasticsearch.password: "kibanapassword"
    xpack.encryptedSavedObjects.encryptionKey: "d7x9s2k5v8y4b3m6n1q0w7e2r5t8y9u1" # Replace this with your own key. You can generate one using `./kibana-encryption-keys generate`.
    server.name: kibana
   

   • elasticsearch.hosts: Use http://localhost:9200 since Elasticsearch is running on the same machine.
3. Run Kibana container:

    sudo docker run -d \
        --restart always \
        --network host \
        --env TZ=Asia/Seoul \
        --volume $HOME/kibana.yml:/usr/share/kibana/config/kibana.yml \
        --name kibana \
        docker.elastic.co/kibana/kibana:8.19.7
   

4. Verify the deployment by opening Kibana (http://10.0.2.1:5601 in this case) in your browser:

   

Installing Elasticsearch Integration

1. Create a user on the production cluster that has the remote_monitoring_collector built-in role.

    curl -X POST "http://localhost:9200/_security/user/elastic_agent?pretty" \
        -u elastic:elasticpassword \
        -H "Content-Type: application/json" \
        -d '{
            "password" : "changeme",
            "roles" : [ "remote_monitoring_collector"]
        }'
   

   • elastic_agent: This is just a username for Elastic Agent. Feel free to use a different one.
   • password: Choose a secure password for this user.
2. Go to Kibana in your browser.
3. Login with:
   • Username: elastic
   • Password: Your ELASTIC_PASSWORD
4. Click Add integrations.

5. In the query bar, search for and select the Elasticsearch integration for Elastic Agent:

   

6. Click Add Elasticsearch.
7. Configure the integration name and optionally add a description. Make sure you configure all required settings:
   • Go to Metrics (Stack monitoring) -> Change defaults:
     • Hosts: Specify data nodes, or a load balancer that routes to master-ineligible nodes:
       • http://10.0.1.1:9200
       • http://10.0.1.2:9200
       • http://10.0.1.3:9200
     • Advanced options:
       • Username: elastic_agent
       • Password: changeme
       • Scope: cluster (If set to node, Elastic Agent only collects metrics from the specified hosts.)
8. Click Save and continue. This may take a minute or two. When it finishes, an agent policy with the Elasticsearch monitoring integration will be created.
9. Click Add Elastic Agent later.

Deploying Elastic Agent

There are two ways to run Elastic Agent:

• Fleet-managed: Elastic Agent is centrally managed through Fleet in Kibana.
• Standalone: Elastic Agent is configured locally using YAML files.

In this tutorial, we’ll use the standalone approach to avoid the extra dependency on Fleet.

1. Go to Kibana for the monitoring cluster in your browser.
2. Go to Fleet -> Agents -> Add Agent -> Run standalone
3. Select Agent Policy 1.
4. Click Create API key to generate an API key for Elastic Agent.
   • By default, the output host is set to http://localhost:9200, which is where Elastic Agent will send monitoring data.
5. Click Copy to clipboard or Download policy
6. Connect to monitoring node:

    ssh my-elasticsearch-node-monitoring
   

7. Create an elastic-agent.yml file and paste the configuration from the clipboard, or copy the downloaded policy from step 5.
8. Run Elastic Agent container:

    docker run \
        -d \
        --restart always \
        --name elastic-agent \
        --network host \
        --user root \
        --volume $HOME/elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml \
        docker.elastic.co/beats/elastic-agent:8.19.7
   

   • --network host: Allows the agent to connect to Elasticsearch via http://localhost:9200.
   • Startup usually takes a few seconds.
9. Go to Kibana for the monitoring cluster in your browser.
10. Click Stack Monitoring in the left menu, then select Remind me later when prompted.

Once everything is set up successfully, Elastic Agent will start collecting and shipping monitoring data from the production Elasticsearch cluster. With this data in place, Kibana can display monitoring information for Elasticsearch and other Elastic Stack components:

Click Nodes for node monitoring:

Click specific node for detailed metrics:

So far, Elastic Agent runs on the same node as the monitoring Elasticsearch and Kibana, but it can be deployed on any other node as well. To do that, update the Outputs in the agent policy:

1. Go to Fleet -> Settings -> Outputs -> Add output
2. Set the monitoring Elasticsearch host to http://10.0.2.1:9200.

Updating the output will result in an elastic-agent.yml similar to the following:

...
outputs:
  default:
    type: elasticsearch
    hosts:
      - http://10.0.2.1:9200
...

When new nodes are added to the production cluster, Elastic Agent will automatically start collecting their metrics without requiring any output changes.

In practice, however, it’s advisable to place a load-balancing proxy in front of master-ineligible nodes. This distributes monitoring traffic evenly and prevents overloading master nodes, which must remain stable.

Creating a Load Balancer

Below is an example of setting up a load balancer with a TLS listener that forwards traffic only to master-ineligible data nodes in your Elasticsearch cluster:

1. Create a target group:
   • Target type: Instances
   • Target group name: my-elasticsearch
   • Protocol: TCP
   • Port: 9200
   • Health check protocol: TCP
   • Health check port: Trafic port (9200 in this case)
   • Targets:
     • my-elasticsearch-node-data-1
     • my-elasticsearch-node-data-2
     • my-elasticsearch-node-data-3
2. Create a networkd load balancer. This is a good choice for Elasticsearch because it has low latency and works well with TCP/TLS passthrough:
   • Load balancer name: my-elasticsearch
   • Scheme: Internal (recommended for private networks)
   • VPC: (Use the same VPC as your Elasticsearch nodes)
   • Security group: (Allow inbound traffic on TCP 443 from trusted sources)
   • Listeners:
     • Protocol: TLS
     • Port: 443
     • Target group: my-elasticsearch
     • Default SSL/TLS server certificate: (Your ACM or imported certificate)
   • With this setup, clients connect over HTTPS on port 443, and the NLB forwards traffic directly to port 9200 on the data nodes.
3. Create a DNS record (for example, in Route 53):
   • Use the hostname covered by your TLS certificate
   • Route to the network load balancer

After this, clients can reach Elasticsearch using a simple HTTPS URL. You can use this URL as the Elasticsearch hosts for Elastic Agent instead of listing the IP addresses of individual data nodes. When you add or remove data nodes, the only thing you need to update is the target group. The client configuration stays the same, and the load balancer automatically handles routing traffic to the available data nodes.

Shell Scripts for Easier Deployment

Often, we need to scale out the cluster to handle more traffic. Manually repeating every step in this guide is daunting.

To make life easier, here are complete scripts you can use:

• For initializing a fresh Elasticsearch cluster: init_cluster.sh
• For joining nodes to an exisiting Elasticsearch cluster: join_cluster.sh
• For initializing a single-node monitoring cluster: init_monitoring_stack.sh

Before running them, make sure you update these values to match your environment:

• VERSION
• Timezone
• ELASTIC_PASSWORD
• ca.crt and ca.key files (refer to REPLACE IT in the join_cluster.sh file)

Zone Awareness

To improve resilience against infrastructure failures, it is highly recommended to deploy an Elasticsearch cluster across multiple availability zones. However, simply placing nodes in multiple availability zones does not automatically make the cluster resilient. The cluster operator must ensure that shards are distributed across zones so that, even if one or more zones fail, the remaining shard copies are still available.

Elasticsearch provides a convenient way to achieve this through shard allocation awareness. The configuration is simple:

1. Add a custom node attribute to each node’s elasticsearch.yml file:

    node.attr.zone: ap-northeast-2a # For nodes in ap-northeast-2a
   

   or

    node.attr.zone: ap-northeast-2b # For nodes in ap-northeast-2b
   

2. Tell Elasticsearch to allocate shards based on that awareness attribute by setting cluster.routing.allocation.awareness.attributes through the cluster update settings API:

    PUT /_cluster/settings
    {
        "persistent": {
            "cluster.routing.allocation.awareness.attributes": "zone"
        }
    }
   

   Once setting is enabled, Elasticsearch starts relocating shards from nodes without the specified awareness attribute to nodes that have it. So, make sure all nodes have already joined the cluster with the appropriate custom attribute before enabling this setting.

Note that for a cluster with three master-eligible nodes, each node should be placed in a different availability zone. If only two availability zones are used, quorum may be lost when the zone containing two master-eligible nodes fails.

Conclusion

Running a self-managed cluster means you own everything. You operate it, you watch it, and you fix it when it breaks. Scaling, monitoring, and compliance are all on you. That work costs time and money, and there’s no way around that.

The upside is what you don’t have. No extra control planes, no operators, no Kubernetes glue code deciding things on your behalf. Systems like ECK (Elastic Cloud on Kubernetes) add layers, and layers add complexity. Complexity creates edge cases, and edge cases could fail in unexpected ways.

If your environment can handle the operational load and you care about clarity, predictability, and knowing exactly how your system behaves, a self-managed cluster can be a good choice.

So, how do we measure it? Since information retrieval is fundamentally an empirical problem, its effectiveness has to be judged by users in some way. One obvious way is to look at what real users do. Metrics like click-through rate (CTR) and conversion rate fall into this category. These are often called online metrics. A/B testing is a popular way to compare different systems using this kind of data.

However, online experiments require deploying systems to actual users, which isn’t always practical or even possible in research settings. This is where systematic approaches based on offline metrics come in.

Cranfield Evaluation Methodology

Cranfield evaluation methodology is a framework for systematically evaluating the effectiveness of information retrieval systems, developed in the 1960s by Cyril Cleverdon at Cranfield University.

This methodology is built upon three key components:

• A collection of documents: a standardized set of documents to search against.
• A set of sample queries: representative queries that simulate user needs.
• Human-judged relevance judgments: a definitive list of which documents are considered relevant for each query.

Note that relevance judgments is by far the most important and the most labor-intensive part of the evaluation since they rely on human assessors. A common strategy to manage this is pooling, where assessors only judge a subset of results pulled together from several existing systems. However, this approach can introduce selection bias. If a new system retrieves relevant documents that never made it into the pool, those documents remain unjudged, and the system ends up being unfairly penalized.

Once we have all three components, the question becomes: how do we quantify the effectiveness?

Precision and Recall

Precision and recall are core metrics to quantify the effectiveness:

• Precision: Of all the documents the system retrieved, how many were actually relevant?
• Recall: Of all the relevant documents, how many did the system retrieve?

These concepts can be illustrated using a confusion matrix:

From this matrix, precision and recall are defined as follows:

• $Precision = \frac{TP}{TP + FP} = \frac{\text{Number of relevant retrieved}}{\text{Total number of retrieved}}$
• $Recall = \frac{TP}{TP + FN} = \frac{\text{Number of relevant retrieved}}{\text{Total number of relevant}}$

Here’s an example.

Suppose we have a small test collection of 10 documents about software:

Our query is “software testing practices”

A human evaluator goes through the entire collection and determines which documents are relevant to the query:

So, there are a total of 5 relevant documents in the collection.

Now, let’s say our retrieval system returns these 4 documents for the query:

• d2: Writing Unit Tests with pytest
• d5: The Practical Test Pyramid
• d9: Principles of Agile Development
• d10: Test-Driven Development Explained

Out of these 4, three are actually relevant (d2, d5, d10), while one (d9) is a false alarm. So:

• Precision = 3 / 4 = 0.75
• Recall = 3 / 5 = 0.6

In Python:

retrieved_docs = ["d2", "d5", "d9", "d10"]
relevant_docs = ["d2", "d5", "d6", "d8", "d10"]


def get_precision(retrieved: list[str], relevant: list[str]) -> float:
    true_positives = len(set(retrieved) & set(relevant))
    false_positives = len(set(retrieved) - set(relevant))
    if (true_positives + false_positives) > 0:
        return true_positives / (true_positives + false_positives)
    return 0


def get_recall(retrieved: list[str], relevant: list[str]) -> float:
    true_positives = len(set(retrieved) & set(relevant))
    false_negatives = len(set(relevant) - set(retrieved))
    if (true_positives + false_negatives) > 0:
        return true_positives / (true_positives + false_negatives)
    return 0

precision = get_precision(retrieved_docs, relevant_docs)
recall = get_recall(retrieved_docs, relevant_docs)

print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")

Output:

Precision: 0.75
Recall: 0.60

In practice, we often use a cutoff at a fixed number of top-ranked results. For example:

• Precision@5 (P@5): The precision when considering only the top 5 retrieved documents.
• Recall@10 (R@10): The recall when considering only the top 10 results.

The ideal system would achieve a precision and recall of 1, meaning it retrieved all relevant documents and no non-relevant ones. However, these two metrics has a classic trade-off: increasing recall by retrieving more documents often leads to a decrease in precision, and vice-versa. To balance the two, we use the F-measure.

F-Measure

F-measure (or F-score) is a way to combine precision and recall into a single score. Instead of just simply averaging them like (P+R)/2, it ensures that both precision and recall matter. The formula is:

Here, the parameter $\beta$ lets you put more weight on recall (if $\beta>1$) or on precision (if $\beta<1$).

The most common variant is the F1-measure, which calculates the harmonic mean of the two:

In Python:

def get_f_score(precision: float, recall: float, beta: float = 1.0) -> float:
    if (precision + recall) > 0:
        return (beta**2 + 1) * precision * recall / (beta**2 * precision + recall)
    return 0

With F1-measure, a system cannot achieve a high F1-score without having both reasonably high precision and high recall. For example,

Notice how the last row shows the penalty in action: even though precision is high (0.9), the low recall (0.1) drags the F1-score way down.

Limitation

The main drawback of F-measure is that it’s set-based so it ignores the order of results. A system that ranks the most relevant document at the top gets the same score as one that hides it at the last, even though the user experience is completely different.

To evaluate a ranked list, we need to consider the order of the results.

Precision-Recall (PR) Curve

The Precision–Recall (PR) curve shows how well a retrieval system balances precision and recall as you move down a ranked list of results. In simple terms, it helps you see whether the relevant documents are concentrated near the top (good) or scattered deeper in the list (not good).

We create the curve by plotting precision against recall at every rank position $k$.

Let’s walk through an example.

Suppose there are 8 relevant documents in total for a query, and our system manages to retrieve 6 of them within the top 10 results ($k$ = 10):

Now, to build the PR curve, we calculate precision and recall at each step:

When plotting the curve, we usually only take the points where a new relevant document appears so that the curve look smoother. That gives us the following (recall on the x-axis, precision on the y-axis):

And visually:

The curve makes it clear how front-loaded the relevant results are. If the curve stays high, the system surfaces relevant docs early. If it drops quickly, it means users would have to dig deeper before finding what they’re looking for.

A system whose curve is higher and more to the right is better. Therefore, an ideal system would have a constant precision of 1.0 for all recall levels.

PR curves are especially useful when comparing multiple retrieval systems. Take the example below:

As System A’s curve is consistently above System B’s, A is clearly superior—it delivers better precision at every level of recall.

However, things get trickier when the curves cross. Take the example below:

In this scenario, system A may have higher precision at low recall levels, while system B has higher precision at high recall levels. The better system depends on the user’s task. For example, a user doing a quick search might prefer System A’s high initial precision, whereas a researcher needing comprehensive results would prefer System B’s high recall.

Mean Average Precision (MAP)

Average Precision (AP)

When PR curves intersect, it’s useful to summarize performance with a single number. Average Precision (AP) does this by averaging the precision values at the ranks where relevant documents appear:

Where:

• $R$ is the total number of relevant documents for the query.
• $N$ is the total number of retrieved documents.
• $P(k)$ is the precision at the rank $k$.
• $Rel(k)$ is whether the document at rank $k$ is relevant (1) or not (0).

$R$ is NOT the number of retrieved relevant documents. If you were to divide by the number of retrieved relevant documents instead, you’d give an unfair advantage to a system that retrieves only a few documents. This would make the score look artificially high even if the system hardly retrieves anything useful.

AP is rank-sensitive: moving a relevant document higher in the list increases the score, while pushing it down lowers it. You can also think of it as a practical approximation of the area under the PR curve.

Let’s compute it for our example. We’ll take the precision at every rank where a relevant document shows up (d1, d2, d3, d5, d6, d8) and average them:

• Precision@d1 = 1/1
• Precision@d2 = 2/2
• Precision@d3 = 3/3
• Precision@d5 = 4/5
• Precision@d6 = 5/6
• Precision@d8 = 6/8

Now average these values:

In Python:

def get_average_precision(relevances: list[int], num_relevant: int) -> float:
    """
    Compute Average Precision (AP) for a single query.
    """
    if num_relevant == 0:
        return 0.0  # edge case: no relevant docs for this query

    precisions = []
    relevant_found = 0
    for k, rel in enumerate(relevances, start=1):  # rank positions start at 1
        if rel == 1:
            relevant_found += 1
            precision_at_k = relevant_found / k
            precisions.append(precision_at_k)

    return sum(precisions) / num_relevant


# Example usage
relevances = [1, 1, 1, 0, 1, 1, 0, 1, 0, 0]  # system output (10 results)
num_relevant = 8  # total relevant docs

AP = get_average_precision(docs, num_relevant)
print("Average Precision:", AP)

Average Precision is great for judging performance on a single query, but it only tells part of the story. To really understand how well a system works, we need to evaluate it across many queries and then aggregate the results. That way, the score reflects overall performance rather than just one example.

Mean Average Precision (MAP)

Mean Average Precision (MAP) is simply the mean of the Average Precision (AP) scores across all queries in the test set:

Where $Q$ is the total number of queries.

It provides a single, comprehensive number that summarizes the system’s ranking quality.

An alternative is the Geometric Mean Average Precision (gMAP), which uses a geometric mean instead of an arithmetic one. The choice between MAP and gMAP depends on the evaluation goal:

• MAP (Arithmetic Mean): The final score is more heavily influenced by high AP scores. This means MAP gives more weight to performance on “easy” queries where the system does well. It is useful if the goal is to measure overall performance, particularly on popular queries that are often easier.
• gMAP (Geometric Mean): This average is more sensitive to low AP scores. Therefore, gMAP emphasizes performance on “difficult” queries where the system struggles. It is useful if the goal is to improve the system’s performance on its weakest queries.

Here’s an example. Suppose we evaluate three queries and get the following AP scores:

• Query 1: 1.0
• Query 2: 0.5
• Query 3: 0.1

Then:

• MAP = $\frac{1.0 + 0.5 + 0.1}{3} \approx 0.53$
• gMAP = $(1.0 \times 0.5 \times 0.1)^{1/3} \approx 0.37$

Here, MAP emphasizes the strong performance on Query 1, while gMAP penalizes the weak score on Query 3 more heavily.

Mean Reciprocal Rank (MRR)

For certain search tasks, there is only one correct or relevant document. This occurs in “known-item searches,” where a user wants to find a single, specific page (e.g., the Youtube homepage), or in some question-answering tasks where there is only one right answer.

In this scenario, the Average Precision calculation simplifies to the Reciprocal Rank:

where $r$ is the rank position of the single relevant documen. For example, if the document is at rank 1, the score is 1. If it’s at rank 2, the score is 0.5, and so on.

The Mean Reciprocal Rank (MRR) is the average of these reciprocal ranks across all queries.

Here’s an example. Suppose we evaluate three queries and the relevant document appears at:

• Query 1: Rank 1 → Reciprocal Rank = 1/1
• Query 2: Rank 3 → Reciprocal Rank = 1/3
• Query 3: Rank 2 → Reciprocal Rank = 1/2

Then

• MRR = $\frac{1/1 + 1/3 + 1/2}{3} \approx 0.61$

Limitation

MAP and MRR are built on the assumption of binary relevance judgments, either a document is relevant (1) or not (0). But in reality, some documents are highly relevant, while others are only marginally relevant.

Let’s revisit our earlier example with the query “software testing practices”:

Here, documents like d2, d5, and d10 are highly relevant — they directly address testing practices. On the other hand, d6 and d8 are only somewhat relevant - they touch on quality practices but don’t focus on testing itself.

To handle this, we need a measure that accounts for graded relevance.

Normalized Discounted Cumulative Gain (nDCG)

Normalized Discounted Cumulative Gain (nDCG) is a metric for evaluating retrieval systems when relevance judgments are multi-level. It measures how good a ranking is by considering both the relevance and position.

In nDCG, relevance is usually graded on a scale from 1 to $r$ (where $r$ > 2). For example, if $r$ = 3, you might define:

• 1 = not relevant
• 2 = somewhat relevant
• 3 = highly relevant

The formula is:

Let’s walk through an example to see how this works.

Gain

The first step is assigning a gain value to each document, which is simply its relevance score:

Where $G_i$ is the gain of the result at position $i$.

Let’s take a simple example using a 3-level relevance scale ($r$ = 3). Suppose our system returns 5 results for a query q1, ranked from to to bottom:

So far, gain is just the raw relevance score.

Cumulative Gain (CG)

Cumulative Gain (CG) is the sum of gains up to rank $k$:

For our example:

So,

The problem with CG is that it ignores the rank order. If you change the top and bottom documents, the CG@5 is still 11. Clearly, that doesn’t reflect user experience because users care much more about the top results.

Discounted Cumulative Gain (DCG)

To make the metric rank-sensitive, the gain at each position is discounted by a logarithmic reduction factor (discount factor). The formula is:

Here, the denominator, $\log_2(i+1)$, reduces the contribution of documents as their rank gets lower. This means a highly relevant document contributes more to the score if it is ranked higher.

For our example:

So:

To compute this with Python, we first define two dictionaries:

qrels_dict = {"q1": {"d1": 3, "d2": 2, "d3": 1, "d4": 2, "d5": 3}}
run_dict = {"q1": {"d1": 0.9, "d2": 0.8, "d3": 0.7, "d4": 0.6, "d5": 0.5}}

• qrels_dict: the relevance judgments for each query and document.
• run_dict: the system’s predicted ranking scores. The exact values don’t matter for DCG as long as they preserve the order since what we ultimately care about is the ranking of documents.

A function to compute DCG:

import math


def get_dcg(qrels_dict: dict, run_dict: dict, k: int, query: str) -> float:
    """
    Compute DCG@k for a specific query.
    """
    # Extract documents for the specific query
    query_qrels = qrels_dict.get(query, {})
    query_run = run_dict.get(query, {})

    # Sort documents by their scores in run_dict (descending order)
    sorted_docs = sorted(query_run.items(), key=lambda x: x[1], reverse=True)

    dcg = 0.0
    for i in range(min(k, len(sorted_docs))):
        doc_id = sorted_docs[i][0]
        relevance = query_qrels.get(doc_id, 0)  # default to 0 if doc not in qrels
        rank = i + 1
        if rank == 1:
            dcg += relevance  # no log discount for rank 1
        else:
            dcg += relevance / (math.log2(rank + 1))  # log base 2
    return dcg


qrels_dict = {"q1": {"d1": 3, "d2": 2, "d3": 1, "d4": 2, "d5": 3}}
run_dict = {"q1": {"d1": 0.9, "d2": 0.8, "d3": 0.7, "d4": 0.6, "d5": 0.5}}

dcg = get_dcg(qrels_dict, run_dict, 5, "q1")
print(f"{dcg:.2f}")

Output:

6.78

Now the ranking matters, but there’s still an issue. DCG scores aren’t directly comparable across queries. Imagine one query has only one relevant document and another has five. Even if both systems ranked perfectly, the DCG for the “easy” query (with lots of relevant docs) will naturally be higher. Without normalization, comparing those scores is unfair.

Normalized Discounted Cumulative Gain (nDCG)

To solve this, we normalize DCG by dividing it by the ideal DCG (IDCG):

Where IDCG is the DCG of an ideal ranking where all the most relevant documents are at the top.

This scales the score between 0 and 1: A system with nDCG = 1 means it produced the ideal ranking. And values below 1 tell you how close (or far) the system is from that ideal.

Suppose the ideal ordering is: [3, 3, 2, 2, 1] for our example:

(Note that the ideal ordering could also be [3, 3, 3, 3, 2] or even [3, 3, 3, 3, 3], depending on how many highly relevant documents exist in the collection.)

So:

Finally:

In Python:

import math


def get_dcg(qrels_dict: dict, run_dict: dict, k: int, query: str) -> float:
    """
    Compute DCG@k for a specific query.
    """
    # Extract documents for the specific query
    query_qrels = qrels_dict.get(query, {})
    query_run = run_dict.get(query, {})

    # Sort documents by their scores in run_dict (descending order)
    sorted_docs = sorted(query_run.items(), key=lambda x: x[1], reverse=True)

    dcg = 0.0
    for i in range(min(k, len(sorted_docs))):
        doc_id = sorted_docs[i][0]
        relevance = query_qrels.get(doc_id, 0)  # default to 0 if doc not in qrels
        rank = i + 1
        if rank == 1:
            dcg += relevance  # no log discount for rank 1
        else:
            dcg += relevance / (math.log2(rank + 1))  # log base 2

    return dcg


def get_ndcg(qrels_dict: dict, run_dict: dict, k: int, query: str) -> float:
    """
    Compute nDCG@k for a specific query.
    """
    dcg = get_dcg(qrels_dict, run_dict, k, query)

    # Compute ideal DCG (iDCG)
    query_qrels = qrels_dict.get(query, {})
    ideal_relevances = sorted(query_qrels.values(), reverse=True)

    idcg = 0.0
    for i in range(min(k, len(ideal_relevances))):
        relevance = ideal_relevances[i]
        rank = i + 1
        if rank == 1:
            idcg += relevance
        else:
            idcg += relevance / (math.log2(rank + 1))

    if idcg == 0:
        return 0.0  # Avoid division by zero; if no relevant documents, nDCG is defined as 0

    ndcg = dcg / idcg

    return ndcg


qrels_dict = {"q1": {"d1": 3, "d2": 2, "d3": 1, "d4": 2, "d5": 3}}
run_dict = {"q1": {"d1": 0.9, "d2": 0.8, "d3": 0.7, "d4": 0.6, "d5": 0.5}}

ndcg = get_ndcg(qrels_dict, run_dict, 5, "q1")
print(f"{ndcg:.2f}")

Output:

0.95

In fact, there’s a Python library ranx that handles evaluation metrics like nDCG out of the box.

Here’s how you can compute nDCG@5 using ranx in just a few lines:

from ranx import Qrels, Run, evaluate

qrels_dict = {"q1": {"d1": 3, "d2": 2, "d3": 1, "d4": 2, "d5": 3}}
run_dict = {"q1": {"d1": 0.9, "d2": 0.8, "d3": 0.7, "d4": 0.6, "d5": 0.5}}

qrels = Qrels(qrels_dict)
run = Run(run_dict)

ndcg = evaluate(qrels, run, metrics="ndcg@5")
print(f"{ndcg:.2f}")

Output:

0.95

Limitation

When comparing two systems, looking only at the average nDCG score (or any single metric) isn’t enough. The question is: “How confident are we that the observed difference isn’t just because of the particular queries we happened to test?”

Imagine two systems:

• System A has an average score of 0.20
• System B has an average score of 0.40

At first glance, System B looks twice as good. But let’s dig deeper:

• Experiment 1 (low variance): System B beats System A on every single query. Here, it’s safe to say B is better.
• Experiment 2 (high variance): Some queries favor A, others favor B, and results swing wildly. The conclusion is much shakier.

This is where statistical significance tests come in.

Statistical Significance Testing

Statistical significance testing answers the question: “Is the observed difference real, or could it have happened by chance?”

Sign Test

One of the simplest approaches is the sign test. Instead of looking at averages, it counts how many times each system wins across queries.

Here’s an example with five queries and scores for two IR systems:

• Wins for System A: 1
• Wins for System B: 4

Here, System B clearly dominates by beating System A in 4 out of 5 queries. That’s a strong sign that B is better than A, at least on this test set.

Limitation

But here’s where things get tricky. Let’s extend the test set with four more queries:

• Wins for System A: 4
• Wins for System B: 5

Now, things don’t look so clear anymore. Even though System B has more wins overall, the margin is just a single query. With such small differences, it’s hard to say whether B is truly the better system or if the result is just due to chance. The sign test doesn’t capture how big the improvements are, only who won more often. To deal with this limitation, researchers often use more advanced significance tests, like the Wilcoxon signed-rank test.

Conclusion

For comparing different IR systems, MAP and nDCG are useful tools. They capture both relevance and ranking quality in a way that’s more informative than simple set-based measures.

But again, information retrieval is fundamentally an empirical problem. What a human assessor marks as “relevant” might not line up with what real users actually care about in the real-world context of search.

The takeaway is that to make reliable decisions, we need to evaluate across many queries. That’s how we avoid shaky averages and get confidence that one system is truly better than another.

References

• Evaluation measures (information retrieval) - Wikipedia
• Discounted cumulative gain - Wikipedia
• Text Retrieval and Search Engines by University of Illinois Urbana-Champaign

Search engines use a ranking function to score document relevance. We write it as $f(q,d)$, where $q$ is the query and $d$ is a document. The higher the $f(q,d)$, the higher the document appears in the search results.

Although various methods exist to estimate document relevance, BM25 (Best Matching 25) is widely regarded as the most practical and effective ranking function for this task. It’s been around since the 90s, and despite newer ranking models, BM25 is still a strong baseline and often tough to beat in real-world applications.

The BM25 formula looks like this:

At its heart, BM25 is just combining three heuristics:

• Term Frequency (TF)
• Inverse Document Frequencey (IDF)
• Document Length Normalization

In this post, I will explain each heuristic and show how they come together to form the BM25 formula.

Term Frequency (TF)

The simplest idea: if a query term appears more often in a document, that document is probably more relevant.

A TF-only ranking function’s formula can be represented as this:

Where:

• $f(q,d)$ is the relevance score of document $d$ for query $q$.
• $\sum_{i=1}^{n}$ is the summation over all terms ($t_1$, $t_2$, …, $t_n$) in the query $q$.
• $TF(t_i,d)$ is the frequency score of query term $t_i$ in $d$.

Then how do we define $TF()$?

A naive TF is just to apply a linear count; $y=x$. Therefore:

Where $c(t,d)$ is the raw count of term t in document d.

Let’s see an example. Suppose we have two documents in our collection:

• d1: The Bank of Korea is expected to lower its benchmark interest rate next month.
• d2: A lower interest rate will be welcomed by indebted households.

Query: “korea interest rate”

Note that this query contains three terms: {$t_1$, $t_2$, $t_3$} = {“Korea”, “interest”, “rate”}

Under this linear application of the raw count, the scores are:

Here, d2 contains two of the query terms—”interest” and “rate”—but misses “Korea”, and d1 matches all three. This ordering makes intuitive sense: d1 fully matches the query, so it gets a higher score.

But there’s a problem: just repeating a term many times can inflate the score too much. This opens the door to what’s called term spamming. Term spamming is when a document unnaturally repeats certain keywords over and over to trick the ranking function into thinking it’s more relevant.

Suppose we add one more document to the collection:

• d3: The interest rate charged on loans is often higher than the interest rate paid on deposits.

With this new collection, the scores are:

Obviously, this is not what we want. Instead, we’d prefer a system where repetition still helps, but each extra occurrence of the same word contributes less and less to the score. In other words, diminishing returns from repetition.

TF Transformation

One simple fix is to apply a sublinear transform like logarithmic transformation; $y=\log(x+1)$. Therefore:

(1 is added to avoid $\log(0)$.)

This way, the impact of each extra occurrence shrinks.

Still, it’s not perfect - TF is unbounded, meaning there’s no upper limit. A spammy document can still rack up arbitrarily high scores if it repeats a term enough times.

To improve it more, here comes a bounded transformation; $y=\frac{x(k+1)}{x+k}$. Therefore:

This function saturates at $(k+1)$, no matter how large $x$ gets. In other words, the function is upper-bounded by $k+1$.

Not only is this transformation capped at $(k+1)$—which stops a single term’s frequency from completely dominating the score—it’s also flexible. The parameter $k$ controls how quickly the curve saturates:

• If $k=0$, it reduces to a simple binary scheme: any occurrence of the term contributes a score of 1, no matter how many times it appears.
• If $k$ is very large, the function behaves almost linearly, getting closer to the linear term frequency.

With this, our TF-based ranking function becomes:

This TF function can be implemented in Python like below:

def tf(term: str, document: str, k: float = 1.2) -> float:
    """
    Calculate TF score using bounded transformation.
    """
    # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
    doc_terms = document.replace(".", " ").lower().split()
    term_count = doc_terms.count(term.lower())
    score = term_count * (k + 1) / (term_count + k)

    return score


def tf_ranking_function(query: str, document: str, k: float = 1.2) -> float:
    """
    Calculate TF score of a document with respect to a query.
    """
    score = sum(tf(term, document, k) for term in query.lower().split())
    return score

To keep things simple, this implementation only applies a couple of basic text normalization steps—removing punctuation and lowercasing. In a real search engine, though, you’d usually go further with more steps such as stemming, lemmatization, and stopword removal to make scoring more robust and accurate.

We can use this code to get the relevance score of a document for a given query. For instance:

score = tf_ranking_function(
    query="korea interest rate",
    document="The Bank of Korea is expected to lower its benchmark interest rate next month.",
)
print(score)

Output:

3.0

The scores for different k values are shown below:

This illustrates the role of $k$: a smaller $k$ dampens the impact of raw term frequency, while a larger $k$ makes the function behave closer to linear counting.

The key takeaway here is that with this transformation, d3 no longer unfairly outranks d1 just because it happens to repeat the query terms.

Limitation

TF alone isn’t enough to reliably measure relevance because not all words are equally informative.

As an example, consider one more document that contains three “interest rate”s:

• d4: The interest rate remains unchanged, but many fear this interest rate keeps loans costly while others welcome a stable interest rate.

Using our TF-only ranking function for the same query “korea interest rate”, repetition wins out:

As you can see, d4 comes out on top even though it never mentions “Korea”, which is likely to be a crucial term. That’s not what we want.

That’s where Inverse Document Frequency (IDF) comes in.

Inverse Document Frequencey (IDF)

Inverse Document Frequency (IDF) solves the problem of treating all words equally. The intuition is that rare words are more informative than common ones. For example, a rarer word like “Korea” can be a strong signal that a document is actually about the topic we care about. On the other hand, common words like “the” or “of” show up in almost every document, so they don’t really help us figure out which document is more relevant.

At its core, IDF penalizes overly common terms that appear everywhere and rewards rarer, more descriptive terms that help distinguish one document from another.

When we combine TF with IDF, we get the TF–IDF ranking function:

Then how do we define $IDF()$?

The standard notation of IDF is this:

Where:

• $N$ is the total number of documents in the collection. (cardinality of documents)
• $df(t)$ is the number of documents containing the term $t$. ($df$ stands for document frequency.)

In words, a word with a low document frequency (a rare word) will have a high IDF score, while a very common word will have a low IDF score.

In practice, however, smoothed IDF variants are often used.

Smoothing

In statistics and NLP, smoothing generally means adjusting counts slightly to avoid extreme cases like zeros or infinities.

One of the most common IDF variants with smoothing is this:

This is also what scikit-learn uses when calculating IDF.

Here’s why the formula has all those extra +1s:

• +1 in the denominator: Without it, if a term never appears in the collection ($df(t) = 0$), you’d end up dividing by zero. The +1 smooths this out so the math always works. You can think of it as “Even if the term is unseen, let’s pretend it appeared at least once.”
• +1 in the numerator: If a term shows up in every document ($df(t) = N$), then $\log\frac{N}{N} = \log(1) = 0$. That would make the term’s weight vanish completely. By adding 1 to the numerator, those super-common terms still get a tiny but non-zero weight.
• +1 outside the log: Even with smoothing inside the fraction, the log value can hover around zero or dip negative when terms are extremely common. Negative IDF values don’t make sense—TF-IDF weights should never imply “anti-importance.” Adding 1 outside guarantees that IDF stays positive, avoiding zeros or negatives.

Graphically:

With this smoothed IDF formula, our TF-IDF ranking function becomes:

In Python:

import math


def tf(term: str, document: str, k: float = 1.2) -> float:
    """
    Calculate TF score using bounded transformation.
    """
    # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
    doc_terms = document.replace(".", " ").lower().split()
    term_count = doc_terms.count(term.lower())
    score = term_count * (k + 1) / (term_count + k)

    return score


def idf(term: str, documents: list[str]) -> float:
    """
    Calculate Inverse Document Frequency (IDF) for a term.
    """
    N = len(documents)  # Total number of documents
    df_t = 0
    for doc in documents:
        # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
        if term.lower() in doc.replace(".", " ").lower().split():
            df_t += 1
    score = math.log((N + 1) / (df_t + 1)) + 1

    return score


def tf_idf_ranking_function(
    query: str, document: str, documents: list[str], k: float = 1.2
) -> float:
    """
    Calculate TF-IDF score
    """
    query_terms = query.lower().split()
    score: float = 0.0
    for term in query_terms:
        tf_score = tf(term, document, k)
        idf_score = idf(term, documents)
        score += tf_score * idf_score

    return score

Let’s revisit the query “korea interest rate” and four documents:

• d1: The Bank of Korea is expected to lower its benchmark interest rate next month.
• d2: A lower interest rate will be welcomed by indebted households.
• d3: The interest rate charged on loans is often higher than the interest rate paid on deposits.
• d4: The interest rate remains unchanged, but many fear this interest rate keeps loans costly while others welcome a stable interest rate.

Using TF-IDF ranking function, the scores are:

The ranking now matches user intent so that d1 rises above d4 because it contains the rarer term “Korea”.

Limitation

The ranking models discussed so far are inherently biased toward long documents. A longer document has a statistically higher chance of matching query terms, even if the matches are coincidental and scattered throughout the text.

Let’s look at one more document:

• d5: In South Korea, the central bank’s decision on the interest rate is closely watched by both businesses and households. Rising interest rate levels have slowed consumer spending, while exporters in Korea argue that a stable interest rate is necessary to remain competitive. Many in Korea believe that future growth depends on how carefully the government manages the interest rate policy.

Notice how d5 is stuffed with the words Korea and interest rate. TF-IDF would likely push this document to the very top, not because it’s the most relevant, but simply because it’s long and keeps repeating the keywords. The scores are:

That’s where Document Length Normalization comes in.

Document Length Normalization

The last heuristic is that longer documents naturally contain more terms, and therefore match queries more often. To counter this, their scores are adjusted relative to the average document length. The goal is simple: balance the bias toward long documents. In short, don’t reward a document just for being long.

Penalizing long documents must be done carefully to avoid over-penalization since a document can be long either because:

• It’s verbose: Some texts are just wordy. They take a simple idea and wrap it in way too many words. In this case, penalization makes sense.
• It’s comprehensive: Other texts are long because they actually cover more ground (like a news archive that bundles many different stories). These shouldn’t be punished too harshly, because their length comes from offering more useful, distinct content.

Pivoted Length Normalization

A robust solution to this problem is Pivoted Length Normalization. This method uses the average document length (avdl) of the entire collection as a “pivot” or reference point.

Here’s how it works in practice:

• Documents with a length equal to the avdl receive no penalty or reward (normalizer = 1).
• Documents longer than the avdl are penalized.
• Documents shorter than the avdl are rewarded.

The formula for the pivoted normalizer is:

Where:

• $|d|$ is the length of the document (number of terms).
• $avdl$ is the average document length across the whole collection.
• $b$ is a tuning parameter between 0 and 1.
  • If $b$ = 0, the normalizer is always 1, and no length normalization occurs.
  • As $b$ increases towards 1, the penalty for long documents and the reward for short documents become more aggressive.

So where does this normalizer actually get applied?

In BM25, it slides into the denominator of the TF term. Here’s the BM25 TF function:

Finally, we arrive at the BM25 (Okapi) ranking function by integrating all three heuristics—TF Transformation, IDF Weighting, and Pivoted Length Normalization:

Where:

• $k$ is the term frequency saturation parameter
• $b$ is the document-length normalization weight (0–1)

In Python:

import math


def tf_with_normalizer(term: str, document: str, k: float = 1.2, normalizer: float = 1.0) -> float:
    """
    Calculate TF score for a term using BM25 with document length normalization.
    """
    # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
    doc_terms = document.replace(".", " ").lower().split()
    term_count = doc_terms.count(term.lower())
    score = term_count * (k + 1) / (term_count + k * normalizer)

    return score


def idf(term: str, documents: list[str]) -> float:
    """
    Calculate Inverse Document Frequency (IDF) for a term.
    """
    N = len(documents)  # Total number of documents
    df_t = 0
    for doc in documents:
        # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
        if term.lower() in doc.replace(".", " ").lower().split():
            df_t += 1
    score = math.log((N + 1) / (df_t + 1)) + 1

    return score


def document_length_normalization(document: str, documents: list[str], b: float = 0.75) -> float:
    """
    Calculate document length normalization factor.
    """
    # Calculate average document length
    total_length = sum(len(doc.split()) for doc in documents)
    avdl = total_length / len(documents)
    doc_length = len(document.split())
    normalizer = 1 - b + b * (doc_length / avdl)

    return normalizer


def bm25_ranking_function(
    query: str, document: str, documents: list[str], k: float = 1.2, b: float = 0.75
) -> float:
    """
    Calculate BM25 score for a document given a query.

    Args:
        query: The query string.
        document: The document string.
        documents: The list of all documents. (collection)
        k: The term frequency saturation parameter.
        b: The length normalization parameter.
    """
    terms = query.lower().split()
    normalizer = document_length_normalization(document, documents, b)
    score: float = 0.0
    for term in terms:
        tf_score = tf_with_normalizer(term, document, k, normalizer)
        idf_score = idf(term, documents)
        score += tf_score * idf_score

    return score

Using the BM25 ranking function, the scores for our example become more acceptable:

What happens here is that even though d5 is long and loaded with repeated mentions of Korea and interest rate, it doesn’t unfairly dominate the ranking anymore.

Conclusion

BM25 is the backbone for many retrieval systems because it’s simple, fast, and surprisingly effective. But here’s one clear limitation: BM25 doesn’t actually understand context. It sees words, counts them, and scores documents based on math, not semantics.

Take these two documents:

• d1: The Bank of Korea is expected to lower its benchmark interest rate next month.
• d6: The interest in street dance across Korea has grown at such a fast rate.

And the query: “korea interest rate”

From BM25’s perspective, both documents look almost identical. The same words appear, with the same frequency, so the scores end up nearly the same. But to a human, the difference is obvious: d1 is clearly more relevant than d6.

That’s why modern search engines rarely rely on BM25 alone. Here are some techniques often used to address BM25’s limitations:

• Relevance feedback: learning from user clicks or pseudo-relevance feedback.
• Semantic embeddings: capturing meaning rather than just word overlap.
• Learning to Rank (LTR): training a machine learning model that combines multiple signals (BM25 scores, embeddings, click data, etc.) to optimize search result ordering.

References

• Okapi BM25 - Wikipedia
• tf–idf - Wikipedia
• Similarity in Elasticsearch by Konrad Beiske
• Text Retrieval and Search Engines by University of Illinois Urbana-Champaign

(YoutTube’s query auto-completion)

This post is a walk-through of building Korean query auto-completion feature based on past searches using Elasticsearch.

What problems does query auto-completion solve?

• Typing can be slow and error-prone. This gets even trickier depending on the user’s environment like mobile. For car navigation, query auto-completion isn’t just nice to have, it’s a must.
• Users sometimes don’t know the exact keywords to search. Query auto-completion helps them find what they want faster, even if they’re not sure what to type.

How is Korean query auto-completion different from English?

The simplest way to build query auto-completion is using just simple prefix matching.

For English, that could be enough with some lowercasing thrown in. For example, if your dataset includes apple, autocomplete, and artificial intelligence, typing just a or A will match all of them.

For Korean, that approach doesn’t quite cut it. Say your dataset includes 가구 (furniture), 간식 (snack), and 겨울 코트 (winter coat). They all start with the consonant ㄱ, but if a user types just ㄱ, none of those will match. That’s because ㄱ, 가, 간, and 겨 are completely different characters. If you want 가구 to show up in the suggestions, you have to type at least 가.

To give users a better experience, you need to break Hangul syllables apart into their building blocks—a process called syllable decomposition. That’s the key to making Korean auto-completion actually work better.

Hangul syllables and Jamo

Hangul (한글) is the Korean alphabet, made up of Hangul Jamos (자모). These Jamos are the Korean consonants and vowels, which are basically the building blocks of Korean syllables. There are 19 consonants and 21 vowels in Jamo.

19 consonants:

• Initial RR: Romanization of consonants at the start of a syllable. These are the sounds you hear starting a Korean syllable.
• Final RR: Romanization of consonants at the end of a syllable. “-“ means these letters don’t appear at the end of syllables.

21 vowels:

Here are some examples of Korean syllables made by combining jamos:

That’s pretty much it. Once you’ve got this down, you can read and write almost any Korean.

Hangul system is remarkably unique in that its letters are made to look like the mouth and tongue shapes used to say the sounds. It is designed so that anyone can easily learn to read and write.

The example in this post uses the following software versions:

• Python 3.13.4 (with pip install elasticsearch==8.13.2 "fastapi[standard]" aiohttp)
• Elasticsearch 8.13.4
• Kibana 8.13.4

Elasticsearch and Kibana are running locally at http://localhost:9200 and http://localhost:5601, respectively.

Context

You’re building an e-commerce application with a search bar. Every time a user types a search query, you’re logging that into an Elasticsearch index called search-logs with the following mappings:

PUT /search-logs
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "query": {
        "type": "keyword"
      },
      "user_id": {
        "type": "keyword"
      }
    }
  }
}

For demo purposes, let’s insert some sample data:

POST /search-logs/_bulk
{"index":{}}
{"@timestamp":"2025-06-21T09:00:00Z","query":"가구","user_id":"user1"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:01Z","query":"가방","user_id":"user2"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:02Z","query":"가방","user_id":"user3"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:03Z","query":"간식","user_id":"user4"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:04Z","query":"겨울 코트","user_id":"user5"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:05Z","query":"사과","user_id":"user6"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:06Z","query":"1234567890","user_id":"user7"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:07Z","query":"airpods 4","user_id":"user8"}

Solution

There are plugins like nori for Korean, but I’m skipping those to build the logic ourselves for full control and to better understand how Korean QAC actually works under the hood.

Another possible solution is using Edge n-gram tokenizer. With this approach, you can set up a custom tokenizer in your index and tweak options like min_gram, max_gram, and token_chars to fine-tune how autocomplete behaves. That said, I won’t be covering this method here.

Instead, I’ll use Elasticsearch’s completion suggester which is a more purpose-built option. I’ll also take a look at its limitations and walk through ways to gradually improve the experience.

Version0: Simple prefix matching

Start simple. No syllable decomposition, just raw keyword prefix matching using Elasticsearch’s completion suggester.

First, create a keyword suggestion index named search-keywords:

PUT /search-keywords
{
  "mappings": {
    "properties": {
      "suggest": { // you can use any name here instead of "suggest"
        "type": "completion",
        "analyzer": "standard"
      }
    }
  }
}

The completion suggester is designed to be as fast as possible because auto-completion should keep pace with your typing, giving you instant, relevant suggestions as you go. This is why auto-completion is also often called as “search-as-you-type” functionality.

We also use the standard analyzer instead of the default simple analyzer to make sure numbers are handled properly. The simple analyzer strips out digits completely, so a query like “1” won’t match “1234567890” in our dataset. The standard analyzer keeps numbers intact—so numeric queries work just like you’d expect.

Now that we have two indices, search-logs and search-keywords, we can read from search-logs and populate search-keywords using this Python script:

# update_keywords0.py
from dataclasses import dataclass

from elasticsearch import Elasticsearch, helpers


def main():
    hosts = ["http://localhost:9200"]
    username = "elastic"
    password = "elasticpassword"
    with Elasticsearch(hosts=hosts, basic_auth=(username, password)) as es_client:
        top_queries = _get_top_queries_from_search_logs(es_client=es_client, size=10000)
        success_count = _update_search_keywords(es_client, keywords=top_queries)
        print("Success count:", success_count)


@dataclass
class Keyword:
    key: str
    count: int


def _get_top_queries_from_search_logs(es_client: Elasticsearch, size: int) -> list[Keyword]:
    """
    Get the top search queries from the search log index.
    We use composite aggregation to handle large datasets efficiently.
    """
    index = "search-logs"

    all_buckets = []
    after_key = None  # for pagination
    while True:
        comp_query = {
            "size": 0,
            "aggs": {
                "top_queries": {
                    "composite": {
                        "size": size,
                        "sources": [{"term": {"terms": {"field": "query"}}}],
                        **({"after": after_key} if after_key else {}),
                    }
                }
            },
        }
        res = es_client.search(index=index, body=comp_query)
        buckets = res["aggregations"]["top_queries"]["buckets"]
        all_buckets.extend(buckets)
        after_key = res["aggregations"]["top_queries"].get("after_key")
        if not after_key:
            break

    # Convert the buckets to a list of Keyword dataclass instances
    keywords: list[Keyword] = []
    for item in all_buckets:
        keywords.append(Keyword(key=item["key"]["term"].strip(), count=item["doc_count"]))

    return keywords


def _update_search_keywords(es_client: Elasticsearch, keywords: list[Keyword]) -> int:
    """
    Update the autocomplete index with new data.
    Return the number of successfully executed actions (including overrides).
    """
    index = "search-keywords"

    actions = []
    for keyword in keywords:
        action = {
            "_op_type": "index",
            "_index": index,
            "_id": keyword.key,
            "_source": {
                "suggest": {
                    "input": [
                        keyword.key,  # Use the keyword itself as input
                    ],
                    "weight": keyword.count,
                }
            },
        }
        actions.append(action)

    res = helpers.bulk(es_client, actions)

    return res[0]


if __name__ == "__main__":
    main()

What it does:

1. Aggregates the top queries from search-logs, using a composite aggregation which efficiently handles large datasets by paginating results. (Without composite aggregation, there is a size limit to the number of buckets.)
2. Bulk-indexes them into search-keywords with:
   • input: the actual query
   • weight: how often it was searched. This way, more frequently searched keywords show up higher in suggestions.

In practice, if you’re dealing with a large volume of logs, you’ll want to add a range filter to speed things up and avoid processing stale data. For example, here’s how to retrieve search logs from the past week:

{
    "size": 0,
    "query": {"bool": {"filter": [{"range": {"@timestamp": {"gte": "now-7d"}}}]}},
    ...
}

Running the code will output:

Success count: 7

In practice, again, you can run this code periodically to keep your dataset fresh and up to date.

Verify the documents by using _search API in Kibana Dev Tools:

GET /search-keywords/_search

Next up, use FastAPI to expose an endpoint /keyword-suggestions:

# app.py
from contextlib import asynccontextmanager
from typing import Annotated

from elasticsearch import AsyncElasticsearch
from fastapi import Depends, FastAPI, Query
from fastapi.middleware.cors import CORSMiddleware

# Create an Elasticsearch client
hosts = ["http://localhost:9200"]
username = "elastic"
password = "elasticpassword"
es_client = AsyncElasticsearch(hosts=hosts, basic_auth=(username, password))


@asynccontextmanager
async def lifespan(app: FastAPI):
    yield
    print("Closing Elasticsearch client...")
    await es_client.close()


# Create FastAPI app with CORS middleware
app = FastAPI(title="Keyword Suggestion API", lifespan=lifespan)
app.add_middleware(CORSMiddleware, allow_origins=["*"])


def get_es_client() -> AsyncElasticsearch:
    return es_client


@app.get(
    "/keyword-suggestions",
    response_model=list[str],
    summary="Suggest keywords for a given query",
    description="Returns a list of keyword suggestions based on the input query.",
)
async def suggest_keywords(
    es_client: Annotated[AsyncElasticsearch, Depends(get_es_client)],
    query: Annotated[str, Query(description="User input query for keyword suggestions")],
    limit: Annotated[int, Query(description="Maximum number of keywords to return", gt=0)] = 10,
):
    body = {
        "suggest": {
            "keyword-suggest": {
                "prefix": query,
                "completion": {
                    "field": "suggest",
                    "size": limit,
                },
            }
        }
    }
    res = await es_client.search(index="search-keywords", body=body)
    results = []
    for option in res["suggest"]["keyword-suggest"][0]["options"]:
        results.append(option["_source"]["suggest"]["input"][0])

    return results

This FastAPI app handles Korean keyword autocomplete using Elasticsearch. Here’s how it works:

1. Set up and manages an Elasticsearch client.
2. Expose a /keyword-suggestions endpoint that takes a query and an optional limit.
3. Query the search-keywords index using Elasticsearch’s completion suggester.
4. Return a list of matching keyword suggestions.

The core of the logic lives in this Elasticsearch query:

body = {
    "suggest": {
        "keyword-suggest": {
            "prefix": query,
            "completion": {
                "field": "suggest",
                "size": limit,
            },
        }
    }
}

Here’s a quick breakdown:

• prefix: the user’s input—we’re trying to autocomplete this.
• completion: tells Elasticsearch to use the completion suggester.
• field: the field in the index we defined to support autocomplete (in this case, suggest).

You can run the app with:

fastapi run

Then head to http://localhost:8000/docs to try it out.

If you hit the endpoint with query=가, you’ll get a response like:

[
  "가방",
  "가구"
]

Note that 가방 comes first because it has a higher weight (i.e., it’s been searched more often). It’s a simple heuristic: the more people search for something, the higher it shows up.

Here are a couple of limitations:

• Typing a single jamo like ㄱ won’t return anything even if some keywords start with that consonant.
• Typing 갑 or 가바 (a typo or maybe just middle of typing 가방) won’t return anything either.

Version1: Syllable decomposition

To support keyword suggestions even when the user types only jamo (like ㄱ, ㄱㄱ, etc.), we need to make a few changes to how we index and search.

We’ll re-create a search-keywords index:

PUT /search-keywords
{
  "mappings": {
    "properties": {
      "suggest": { // you can use any name here instead of "suggest"
        "type": "completion",
        "analyzer": "standard"
      },
      "keyword": {
        "type": "keyword"
      }
    }
  }
}