• Different workloads could not easily share the same hardware.
• Hardware was often idle.
• Testing a new OS or switching OSes required another machine or a reboot.

Virtualization was created to solve these problems. With virtualization, multiple operating systems and applications can run at the same time on the same hardware. Each one acts like it has its own machine, without forcing everyone to use the same operating system.

Virtual Machines and the Hypervisor

Virtual Machines

An early definition of a virtual machine is:

A virtual machine is taken to be an efficient, isolated duplicate of the real machine. –Popek and Goldber, 1974

A virtual machine (VM) is a software-defined machine that shares the physical CPU, memory, and devices. A VM is also called a guest or a domain.

The operating system running inside a VM is called the guest operating system (guest OS). From the guest operating system’s point of view, it is running on a real machine. Therefore, the guest OS does not know (and ideally does not care) that it is sharing physical resources with other VMs.

Hypervisors

To run multiple VMs safely, you need a control layer that manages hardware and keeps VMs apart. This layer is called virtual machine monitor (VMM), or hypervisor.

In most virtualization literature, VMM means virtual machine monitor (the hypervisor). But, some products also use VMM to mean virtual machine manager (a management tool).

A hypervisor is software that sits between the physical hardware and the virtual machines. It is responsible for mainly three goals:

1. Fidelity: The VM should look like a real machine. The guest OS should not need special changes just to boot and run.
2. Performance: The hypervisor should not slow down normal execution.
3. Safety: The hypervisor must control shared resources.

In general, the hypervisor stays out of the way during ordinary execution. It mainly steps in when a guest tries to do something that must be controlled, such as:

• Accessing privileged CPU registers
• Changing page tables
• Performing certain I/O operations

Benefits and Drawbacks

Virtualization is widely used because it has multiple benefits:

• Consolidation: Run many servers on one machine to increase utilization.
• Cost reduction: Fewer machines means less power, cooling, and hardware cost.
• Portability: A VM is easy to copy, move, or back up.
• Fast provisioning: You can create new VMs quickly from templates.
• Maintenance and migration: You can move VMs during hardware maintenance or load changes.
• Fault isolation: A crash in one VM does not have to crash others.
• Testing and debugging: OS experiments are easier when you can snapshot and roll back.
• Legacy support: Old OSes can run without dedicating old hardware.

However, virtualization also has drawbacks:

• Performance overhead compared to running directly on hardware.
• Increased system complexity and management overhead.
• Limited direct resource sharing due to isolation.
• Host failure can impact many VMs unless redundancy is used.
• Licensing and virtualization platform costs may be significant.

Types of VMs

VMM implementations differ widely. Common approaches are:

• Type 1
• Type 2
• Paravirtualization
• Containerization

In some systems like IBM LPARs and Oracle LDOMs, virtualization support is built directly into the hardware or firmware, rather than implemented as a traditional software hypervisor. These systems are often called Type 0 hypervisors.

Type 1: Bare-metal hypervisor

In type 1, the hypervisor runs directly on the hardware, without a host operating system underneath. It has full control over the machine and usually provides better performance and stronger isolation. The VMs run on top of hypervisor.

Type 1 is common in data centers and cloud platforms.

Examples of type 1 virtualization are:

• VMware ESXi
• XenServer
• Linux KVM
• Microsoft Hyper-V

Type 2: Hosted virtualization

In type 2, a normal host OS runs on the hardware first. The virtualization software runs on top of that OS. The host OS then supplies device drivers and many I/O services. And the hypervisor is hosted by the OS.

Type 2 is common on personal computers and for development or testing.

Examples of type 2 virtualization are:

• VMware Fusion and Workstation
• Oracle VirtualBox
• Parallels Desktop

Paravirtualization

Paravirtualization does not try to fully imitate real hardware. Instead, the guest OS is modified to work in cooperation with the VMM to optimize performance.

In a paravirtualized system, the guest kernel replaces some privileged operations with hypercalls. A hypercall is like a system call, but instead of entering the OS kernel, it enters the hypervisor. Examples of privileged operations that are commonly done via hypercalls include:

• Updating page tables
• Configuring interrupts
• Performing device I/O operations

Paravirtualization can be faster because hypercalls avoid replying on traps or instruction rewriting. The downside is that the guest OS must be modified and maintained. It can also be less portable, because it depends on a particular hypervisor interface.

XenServer popularized this approach.

Containerization

Sometimes the goal is not to run a full guest OS, but to isolate and manage just applications. If all applications run on the same OS, full virtualization may be unnecessary. Instead, the system can use containerization (or application containment).

A container is an isolated environment for running applications while sharing the host OS kernel. Unlike a virtual machine, a container does not include a guest OS and does not virtualize hardware. Instead, it bundles only what the application needs to run, such as the application binaries, libraries, and configuration files.

Compared to other virtualization approaches, its main advantage is efficiency. Containers are much lighter than VMs. They use fewer resources and can start and stop quickly—more like processes than full virtual machines.

Containerization is popular in software development and deployment, particularly in cloud systems. It is commonly managed with tools like Docker and Kubernetes.

Building Blocks

Whether virtualization is possible basically depends a lot on the CPU’s features.

To virtualize a system, the hypervisor must provide an exact duplicate of the underlying machine. This is especially difficult on older systems that support only two CPU modes: user mode and kernel mode.

For example, classic x86 defines four privilege levels, called rings:

• Ring 0: most privileged, used by the OS kernel
• (Rings 1 and 2 exist, but they are rarely used.)
• Ring 3: least privileged, used by user programs

On CPUs with only these privilege rings, it’s hard to fit a hypervisor in the right place. The hypervisor needs to run with the highest privileges so it can keep control. But a normal OS is written assuming it also runs with the highest privileges (ring 0). If you push the guest OS down to a less-privileged ring (like ring 3), some operations won’t behave the same way, and the hypervisor may not be able to catch and handle them correctly.

To fix this limitation, modern x86 CPUs add hardware support for virtualization (Intel VT-x and AMD-V). They introduce two new modes:

• Root mode: used by the hypervisor
• Non-root mode: used by guest operating systems

With this setup, the guest kernel can still run in ring 0, but it runs in non-root mode. In other words, it looks like ring 0 to the guest, but the CPU still keeps it on a leash.

When the guest executes certain sensitive instructions, the CPU automatically traps into the hypervisor. Control switches from non-root to root mode, the hypervisor handles the request, and then the CPU returns to the guest.

On top of this hardware foundation, VMMs use techniques such as trap-and-emulate and binary translation to handle privileged behavior efficiently.

Another important concept is the virtual CPU (vCPU). A vCPU does not execute instructions by itself. Instead, it stores what the guest thinks the CPU state is, including things like:

• Register values
• Program counter
• CPU flags

For each VM, the VMM keeps one or more vCPU data structures. When the VMM schedules a guest to run on a real CPU core, it does a simple save/restore cycle:

1. Load: copy the saved CPU state from the vCPU into the real CPU (registers, instruction pointer, flags, etc.).
2. Run: let the guest execute for a time slice, until it traps, blocks, or gets preempted.
3. Save: copy the updated CPU state back into the vCPU so the guest can resume later from the same point.

Trap-and-Emulate

On a typical CPU with only two modes (user and kernel) without extra hardware support, the guest kernel can execute only in user mode. Because the VM must imitate the physical machine, the VMM provides virtual modes:

• Virtual user mode
• Virtual kernel mode

Both virtual modes still execute in physical user mode.

For example, the following actions cause a transfer from user mode to kernel mode on a real machine:

• a system call
• an interrupt
• a privileged instruction

When the same actions occur inside the guest kernel, the VM must also transfer from virtual user mode to virtual kernel mode. This is done using trap-and-emulate:

1. The guest kernel executes a privileged instruction.
2. Because the guest kernel is in physical user mode, it causes a trap to the VMM (in the real machine).
3. The VMM executes (or emulates) the privileged instruction of the guest kernel.
4. It then returns control to the virtual machine.

The downside is overhead. Nonprivileged instructions run at near native speed, but privileged instructions trap to the VMM and must be emulated. Sharing the CPU across multiple VMs can also make performance unpredictable.

However, as hardware has improved, trap-and-emulate has gotten faster, and the system needs it less often. For example, modern CPUs add extra modes for virtualization beyond the typical user and kernel modes. This lets the CPU handle more of the work directly at the hardware level, so the VMM has less to do.

Binary Translation

Trap-and-emulate works only on CPUs where all sensitive instructions trap when executed in unprivileged mode. If some sensitive instructions can run without trapping, trap-and-emulate alone is not sufficient.

Binary translation fixes this by rewriting guest kernel code on the fly. Assume there are some sensitive instructions that do not automatically trap. Call them special instructions. Then:

• If the guest vCPU is in virtual user mode, the guest can run its instructions natively on a physical CPU.
• If the guest vCPU is in virtual kernel mode, the VMM inspects every instruction by the guest:
  • Non-special instructions run natively.
  • Special instructions are translated (rewrited) into safe sequences that produce the same effect

Even though most instructions run natively, special instructions still add overhead because they must be translated. To reduce this cost, the VMM saves translated code in a translation cache. When the same code runs again, the VMM can reuse the cached translation instead of translating it again.

Core Operating-System Components in Virtualization

CPU Scheduling

The effect of virtualization on scheduling varies a lot. This is because virtualization can make even a single-CPU system look like a multiprocessor sytem. For example, even if a host has six physical CPUs, the VMM can present twelve vCPUs across its guests. This implies:

• If there are enough physical CPUs to satisfy each guest’s requested vCPUs, the VMM can treat the allocation as effectively dedicated. It can schedule only a given guest’s threads on that guest’s assigned CPUs. In this situation, the guests behave much like native operating systems running on native hardware.
• If the total number of requested vCPUs exceeds the available physical CPUs (overcommitment), the VMM relies on a standard scheduling algorithm so that guests continue to perceive adequate CPU availability. For instance, with six physical CPUs and twelve vCPUs total, the VMM can distribute CPU time proportionally, so each guest effectively receives about half of the host’s CPU capacity.

Under CPU overcommitment, some performance degradation is expected because vCPUs must time-share physical CPUs and may spend time waiting to be scheduled.

Memory Management

VMMs typically overcommit memory, so the total memory allocated to guests can exceed the amount of physical memory available. This increases memory pressure because there are more guest operating systems and their applications, plus the VMM itself. Therefore, the VMM must preserve the illusion that the guest has that amount of memory, even when physical memory is smaller. To do that:

1. The VMM first evaluates each guest’s maximum memory size so that the guest OSes continue to believe they have what they asked for.
2. The VMM then computes how much memory to actually give it right now based on current load and overcommitment.
3. The VMM then reclaims memory from the guests when the host system is running low on memory.

For reclaiming memory, the most commonly used technique is memory ballooning, developed by VMware. Here, a balloon driver is loaded into the guest as a pseudo-device driver.

A pseudo–device driver uses device-driver interfaces, appearing to the kernel to be a device driver, but does not actually control a device. Rather, it is an easy way to add kernel-mode code without directly modifying the kernel.

The role of a balloon driver is to make the guests aware of the low memory status of the host. It does this by polling the hypervisor and getting a target balloon size. Without this driver, the VM cannot detect host memory pressure because the VM is isolated from the host. When the host is low on memory:

1. The hypervisor sets a target balloon size based on how much memory it wants to reclaim.
2. The balloon driver “inflates” the balloon inside the guest. It allocates and pins that amount of guest physical memory to the balloon. This makes the guest OS treat those pages as in use.
3. The balloon driver reports the pinned page frame numbers to the hypervisor. The hypervisor can then reclaim the host physical pages backing those frames.

When the host has enough memory again, the hypervisor “deflates” the balloon, and the balloon driver releases pages back to the guest.

The key benefit of ballooning is that it lets the guest operating system decide which pages to page out. The hypervisor does not need to be involved in that decision.

Storage Management

In a virtualized system, storage must be handled differently from a native operating system.

The exact approach depends on the type of hypervisor:

• Type 1 hypervisors store each guest’s root disk and configuration data as one or more files in file systems managed by the VMM.
• Type 2 hypervisors store the same information as files in the host operating system’s file systems.

In practice, a guest’s entire root disk is packaged into a single disk image file managed by the VMM. Although this design can introduce performance overhead, it is highly practical. It makes copying, backing up, and moving virtual machines straightforward. For example, if an administrator wants to create a duplicate guest for testing, they can simply copy the disk image file and register the new copy with the VMM. Similarly, moving a virtual machine between systems running the same VMM is easy: shut down the guest, copy the disk image to the new system, and start the guest there.

Many VMMs also support converting systems between physical and virtual systems:

• Physical-to-virtual (P-to-V) conversion: converts a running physical machine into a virtual machine. This is done by copying its disk contents into virtual disk image files.
• Virtual-to-physical (V-to-P) conversion: converts a virtual machine into a physical system. This is useful for debugging a problem where the virtualization layer itself may be contributing to the issue.

I/O Management

I/O virtualization is easier than CPU or memory virtualization because operating systems already handle many kinds of devices through device drivers. A hypervisor uses this by providing specific virtualized devices to guests.

Common ways a VMM handles I/O:

• Pass-through (dedicated device): one real device is directly dedicated to one guest. This can be close to native speed, but other guests cannot use that device.
• Shared device: many guests share one real device. The VMM must sit in the middle for every I/O to keep guests isolated and send data to the right place.
• Split-device: the guest uses a simple driver that talks to the VMM, and the VMM talks to the real hardware. This is often faster and easier to manage than fully emulating real devices.

Direct access can be fast in type 0 hypervisor, but type 1 and type 2 hypervisors usually need hardware support. Without it, I/O slows down because the hypervisor must handle many extra events, especially interrupts.

For networking, each guest must have at least one IP address to communicate with other systems. The VMM then acts like a virtual switch to route the packets to the right guest. Guests can connect using:

• Bridging: the guest appears directly on the external network with its own IP.
• NAT: the guest uses an internal IP, and the VMM translates and forwards traffic.

VMMs often also provide basic firewall rules to control traffic between guests and to the outside network.

I/O Devices

I/O devices cover a wide range of hardware. Common examples are storage devices such as SSDs, network interface cards (NICs), and devices used for human interaction, including displays, keyboards, speakers, and microphones.

Since I/O devices behave very differently from one another, the operating system needs to hide these differences from applications. To make this separation possible, operating systems rely on two key ideas:

• Standard interfaces
• Device drivers

Standard Interfaces

Because I/O devices are very different, applications cannot communicate with them directly. Instead, the operating system defines standard interfaces for broad classes of devices, such as storage devices, network devices, or display devices. These interfaces define what operations a device supports (for example, read, write, send, receive) and how those operations are requested. Applications interact only with these interfaces, not with the hardware itself. This allows the same application code to work with many different devices of the same type.

At a lower level, devices must physically connect to the CPU and memory. In modern systems, this is done using device interconnects, most commonly PCI Express (PCIe).

PCIe provides a high-speed, point-to-point connection between devices and the CPU. Unlike the older shared PCI bus, PCIe uses dedicated links (lanes) for each device. Through PCIe, devices can exchange data with the CPU, memory, and the rest of the system in a standardized way.

Other types of buses are also used even in a PCIe system, depending on the device. For example:

• SATA controllers for certain storage devices
• Simpler peripheral buses (such as USB) for devices like keyboards and mice

The choice of bus affects performance, how devices are discovered at boot time, and how the operating system manages and schedules access to hardware.

Device Drivers

A device driver is software that knows how to control a specific hardware device. Each driver contains all the device-specific code needed to configure the device, send commands to it, move data in and out, and respond to events such as interrupts or errors.

When an application makes an I/O request through a standard interface, the operating system forwards that request to the appropriate driver. This design allows the operating system and applications to remain unchanged even when hardware is replaced, as long as the correct driver is available. From the operating system’s perspective, supporting a new device usually means adding a new driver, not rewriting the rest of the system.

Typically, the operating system defines a standard device driver framework, and hardware manufacturers provide drivers that fit into this framework. This is why users often need to install drivers when adding new hardware, such as printers.

At the hardware level, a device controller contains built-in logic and firmware that directly run the physical device. It receives commands from the device driver, performs the requested actions, transfers data between the device and memory, and signals completion or errors via interrupts.

Devices as Files

Many operating systems represent devices as files. This means applications can interact with devices using familiar operations such as read and write.

On Unix-like systems, devices appear as special files under the /dev directory. For example, running echo "hello, world" > /dev/lp0 sends the text hello, world to the first line printer (lp0). Although this looks like a normal file write, the kernel intercepts the operation and translates it into the appropriate device-specific commands handled by the printer driver.

Unix-like operating systems provide a virtual device known as the null device, exposed as /dev/null. Any data written to this device is discarded by the kernel.

Typical Device Access Flow

Typically, user programs cannot access hardware devices directly. Instead, device access goes through the operating system:

1. When a process needs to use a device—such as sending a network packet or reading a disk file—it makes a system call. This transfers control from the process to the kernel.
2. Inside the kernel, the operating system runs the relevant internal code:
   • Networking requests go through the TCP/IP stack.
   • File requests go through the file system to locate disk blocks.
   • The kernel converts the high-level request into a low-level device operation.
3. The kernel then calls the device driver, which understands the hardware details. The driver:
   • Translates requests into device-specific commands
   • Configures the device by writing to its registers
   • Manages pending requests safely
4. The driver uses mechanisms like programmed I/O or DMA to transfer data. Once configured, the device performs the operation, such as transmitting data or reading from disk.
5. When the operation finishes, the result travels back through the driver and kernel to the user process.

OS Bypass

Some devices support a different model in which applications can access hardware without going through the kernel on every operation. This is called operating system bypass (OS bypass).

In this model, the operating system is involved only during setup. It maps device memory and registers into the process’s address space and establishes permissions. After setup, the data path goes directly between the user process and the device.

Because the kernel is no longer in the data path, applications use a user-level driver, typically a library provided by the device vendor, to issue device commands.

Blocking, Nonblocking, and Asynchronous I/O

When a process issues an I/O request, the device will eventually return some response. What happens to the calling thread depends on whether the I/O is synchronous or asynchronous.

An important design choice in the system-call interface is how I/O operations interact with a running program. The key question is whether the calling thread should wait for I/O to finish or continue running while the I/O happens in the background.

Blocking (synchronous) I/O

In blocking I/O, a system call pauses the calling thread until the I/O operation completes.

Here is what happens step by step:

1. The application issues a blocking system call (for example, read()).
2. The operating system suspends the calling thread.
3. The thread is moved from the run queue to a wait queue.
4. When the I/O finishes, the thread is moved back to the run queue.
5. When the thread runs again, it receives the return values from the system call.

Although I/O devices work asynchronously at the hardware level and may take an unpredictable amount of time, operating systems still offer blocking calls. The reason is simple: blocking code is easier to write and reason about.

Nonblocking I/O

Some operating systems provide nonblocking I/O system calls. These calls do not suspend the thread for a long time.

A nonblocking call:

• Returns immediately.
• Reports how many bytes were transferred.
• May return fewer bytes than requested, or even zero, if no data is available yet.

The thread can then decide whether to try again later or do something else. This avoids wasting CPU time and allows computation to overlap with I/O.

One way to handle this problem is multithreading. An application can create multiple threads:

• Some threads perform blocking I/O.
• Other threads continue running useful work.

This approach works well but increases complexity due to synchronization and shared data management.

Asynchronous I/O

Another option is asynchronous I/O, which goes one step further.

With an asynchronous system call:

• The call returns immediately.
• The operating system guarantees that the entire I/O request will be completed later.
• The thread continues executing without waiting.

When the I/O finishes, the operating system notifies the application using one of several methods:

• Updating a variable in the process’s address space
• Sending a signal or software interrupt
• Invoking a callback function

Asynchronous operations are common inside modern operating systems, even when applications are not directly aware of them.

Block Device Stack

The block device stack describes how an I/O request follows from an application down to the physical storage device.

At the top of the stack are applications.= As explained eariler, applications do not work with disks directly. Instead, they operate on files, which are logical objects provided by the operating system. Programs use system calls such as open, read, and write to access files without knowing where or how the data is stored.

These system calls are handled by the file system. The file system manages files and directories, checks permissions, and translates file-level operations into accesses to disk blocks. Before accessing the disk, the operating system usually checks the page cache. If the requested data is already in memory, it can be returned immediately. If not, the request continues down the stack.

Below the file system and page cache is the generic block layer. This layer provides a common interface for different block devices such as HDDs, SSDs, and USB drives. Even though these devices work differently internally, the block layer hides those differences and schedules read and write requests.

At the bottom of the stack are the device drivers. Drivers communicate directly with the hardware and handle device-specific commands, errors, and interrupts.

A typical storage request flows down the stack as follows:

1. Application calls read or write
2. Operating system enters the file system. If data is present in page cache, return immediately.
3. File system maps the request to disk blocks
4. Block layer schedules the request
5. Device driver accesses the hardware

Virtual File System

Modern operating systems support many different kinds of storage. Files may live on different disks, use different file system formats, or even be stored on remote drives and accessed over a network. Despite this complexity, applications should not need to know where a file is stored or how it is managed internally.

To solve this problem, operating systems such as Linux use a Virtual File System (VFS) layer. The VFS sits between applications and the actual file systems. Applications interact with files using a standard interface, such as the POSIX API, and the VFS takes care of translating these requests to the appropriate underlying file system.

This separation makes it easy to add new file systems or switch storage devices without modifying application code.

To support many file systems in a uniform way, the VFS defines a small set of common abstractions.

Files and File Descriptors

A file is the basic object that the VFS operates on. When a process opens a file, the operating system creates a file descriptor, which is a small integer used to identify that open file.

All file operations—such as reading, writing, locking, or closing—are performed using the file descriptor. The file descriptor exists only while the file is open and is specific to the process that opened it.

Inodes

An inode (short for index node) is the core data structure a file system uses to manage files. Internally, each file is identified by an inode number, not by its file name. The file name is just a human-friendly label used to locate the corresponding inode.

An inode contains a list of disk block numbers that store the file’s actual data. In this sense, the inode acts as an index: it does not store the file contents itself, but instead stores pointers to the blocks that hold the contents. Reading those blocks in the correct order reconstructs the file.

When more data is written to a file and extra space is needed, the file system:

1. Allocates a free disk block
2. Adds its block number to the inode
3. Updates the inode on disk

The inode always reflects the current layout of the file.

Besides block pointers, an inode also stores metadata such as:

• Access permissions
• Ownership
• Locking or status information

This metadata is used to enforce access rules and manage concurrent use.

This inode-based design supports efficient access patterns. Sequential access simply follows the list of blocks in order, while random access directly computes the required block from the file offset.

The main drawback of this simple design is that it limits how large a file can be. Consider this example:

• Inode size: 128 bytes
• Each block pointer: 4 bytes
• Maximum number of block pointers: 128 / 4 = 32
• Block size: 1 KB

With only direct block pointers and no space for metadata, the largest file that can be addressed is 32 blocks * 1 KB = 32 KB.

A common way to overcome the file-size limitation of simple inodes is to use indirect pointers.

An inode still begins with metadata, followed by pointers to data blocks. The first pointers are direct pointers, which point straight to blocks containing file data.

An indirect pointer does not point to data directly. Instead, it points to a block that contains only block addresses. A 1 KB block can store 256 such addresses, allowing a single indirect pointer to reference 256 KB of file data. For larger files, double or even triple indirect pointers are used. A double indirect pointer, for example, points to a block of pointers to other pointer blocks, which then point to data blocks. With 256 pointers per block, double indirect addressing supports up to 64 MB of data.

The trade-off of using indirect pointers is that it increases file size but also increases access cost.

Directories and Dentires

Files are organized into directories. From the VFS perspective, a directory is simply a special kind of file. The difference is that the contents of a directory describe file names and the inodes they refer to.

Dentires are cached in memory in what is known as the dentry cache. This cache avoids repeated disk accesses when the same directories are accessed multiple times. Dentires are not stored on disk. They are temporary objects maintained only in memory.

Superblock

Every file system has a superblock, which describes how the file system is laid out on disk. The superblock contains information such as:

• Where inodes are stored
• Where data blocks are located
• How free space is managed

The superblock acts as a map that allows the operating system to interpret the on-disk data structures correctly.

Many VFS structures, such as file descriptors and dentries, exist only in memory. Others, including file data, inodes, and superblocks, must be stored persistently on disk.

A widely used modern file system in Linux is ext4 (4th extended file system), which is the successor to ext2 and ext3.

Disk Access Optimizations

Disk operations are much slower than memory operations, so file systems use several techniques to reduce disk access and improve I/O performance. These techniques focus on keeping data in memory, reducing disk head movement, and avoiding unnecessary random accesses.

Key techniques include:

• buffer caching: Keep recently accessed file data in memory so most reads and writes avoid disk access; changes are flushed to disk periodically.
• I/O scheduling: Reorder disk requests to minimize disk head movement and favor sequential access over random access.
• prefetching: Load nearby file blocks into memory in advance, increasing cache hits and reducing future access latency.
• journaling: Record updates in a sequential log before writing them to their final disk locations, improving reliability while limiting random disk writes.

This post is a comprehensive walkthrough of deploying a self-managed Elasticsearch cluster using one of the supported installation methods: Docker. It also aims to follow production best practices and recommendations.

Planning Elasticsearch Cluster

By the end of this tutorial, you’ll have the following setup in place:

We will deploy:

• A production cluster with:
  • Three master nodes, as there should normally be an odd number of master-eligible nodes in a cluster.
  • Three data nodes
• A dedicated monitoring cluster with Kibana and Elastic Agent
• An optional load balancer in front of the production cluster.

About Node Roles

Here’s a quick note on node roles in this setup:

• master: Responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes.
• data: Responsible for holding data and performing data related operations such as CRUD, search, and aggregations.

We won’t use coordinating only nodes in this setup, since data nodes can happily serve the same purpose. For more details on node roles, refer to Node roles.

Why Use a Dedicated Monitoring Cluster?

Using a dedicated monitoring cluster is the recommended configuration for two key reasons:

• Production outages do not impact access to monitoring data.
• Monitoring workloads are isolated and cannot degrade production cluster performance.

Elastic also recommends using a separate Kibana instance to view a separate monitoring cluster.

For the sake of operational simplicity, this setup uses a single-node monitoring cluster. When operating at scale, however, you can also configure a more resilient multi-node monitoring cluster by following this tutorial.

Infrastructure Setup

You can skip this part if you already have servers ready to host the Elasticsearch cluster.

This tutorial has been demonstrated using the following setup:

• Elastic Stack version: 8.19.7.
• VPN access: A VPN for secure configuration and access to the Elasticsearch cluster, using the CIDR block 10.0.0.0/8

Create a Security Group

Log in to the AWS Management Console and navigate to EC2 -> Security Groups -> Create security group.

Create a security group named my-elasticsearch-node for the Elasticsearch nodes, and configure the following inbound rules:

Create a Launch Template

Create a Launch Template to make it easier to add multiple nodes to the cluster.

Go to EC2 → Launch Templates → Create launch template, then configure the following:

• Launch template name: my-elasticsearch-node
• Launch template contents:
  • Amazon Machine Image (AMI): Ubuntu Server 24.04 LTS (HVM), SSD Volume Type
  • Architecture: 64-bit (Arm)
• Network settings:
  • Security groups: my-elasticsearch-node
• Storage:
  • Size: 100 GiB

This configuration is just an example. You can adjust the values to match your own requirements. Just make sure to select the security group created in the previous step, that is my-elasticsearch-node.

Launch Instances

1. Go to EC2 -> Instances -> Launch instance from template and configure the following:
   • Source template: my-elasticsearch-node
   • Instance type: (Your choice. For this tutorial, the monitoring node requires at least 8 GiB of RAM.)
   • Key pair (login): Select your key pair.
   • Resource tags:
     • Key: Name (for identifying instances)
     • Value: my-elasticsearch-node
   • Number of instances: 7 (6 for the production cluster and 1 for the monitoring node)
2. Click Launch instance
3. Go to EC2 -> Instances -> Search my-elasticsearch-node and update the Name tags as follows:
   • my-elasticsearch-node-master-1
   • my-elasticsearch-node-master-2
   • my-elasticsearch-node-master-3
   • my-elasticsearch-node-data-1
   • my-elasticsearch-node-data-2
   • my-elasticsearch-node-data-3
   • my-elasticsearch-node-monitoring
4. Retrieve the private IP addresses of all EC2 instances using the AWS CLI:

    aws ec2 describe-instances \
        --filters "Name=tag:Name,Values=my-elasticsearch-node-*" \
        --query "Reservations[].Instances[].[Tags[?Key=='Name'].Value | [0], PrivateIpAddress]" \
        --output text \
        --no-cli-pager
   

   Output:

    my-elasticsearch-node-master-1   10.0.0.1
    my-elasticsearch-node-master-2   10.0.0.2
    my-elasticsearch-node-master-3   10.0.0.3
    my-elasticsearch-node-data-1     10.0.1.1
    my-elasticsearch-node-data-2     10.0.1.2
    my-elasticsearch-node-data-3     10.0.1.3
    my-elasticsearch-node-monitoring        10.0.2.1
   

5. Update SSH config file (usually ~/.ssh/config) to connect to the instances from your local environment:

    Host my-elasticsearch-node-*
        User ubuntu
        IdentityFile your_key.pem
    Host my-elasticsearch-node-master-1
        HostName 10.0.0.1
    Host my-elasticsearch-node-master-2
        HostName 10.0.0.2
    Host my-elasticsearch-node-master-3
        HostName 10.0.0.3
    Host my-elasticsearch-node-data-1
        HostName 10.0.1.1
    Host my-elasticsearch-node-data-2
        HostName 10.0.1.2
    Host my-elasticsearch-node-data-3
        HostName 10.0.1.3
    Host my-elasticsearch-node-monitoring
        HostName 10.0.2.1
   

The IP addresses used in this tutorial are simplified for clarity. Your IP addresses will differ depending on your VPC and subnet configuration.

Deploying an Elasticsearch cluster

Initializing the Cluster with Master Nodes

1. Connect to my-elasticsearch-node-master-1:

    ssh my-elasticsearch-node-master-1
   

2. Install Docker 7:

    sudo apt-get update
    sudo apt-get install -y ca-certificates curl
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
    echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
        sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo usermod -aG docker ${USER}
   

3. Set vm.max_map_count to at least 262144:

    sudo sysctl -w vm.max_map_count=262144
    # Add `vm.max_map_count` to `sysctl.conf`
    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
   

   • This is to increase the limits on mmap counts to avoid out of memory exceptions:
4. Prepare local directories for storing data and logs through a bind-mount:

    mkdir esdatadir && chmod g+rwx esdatadir && sudo chgrp 0 esdatadir
    mkdir eslogsdir && chmod g+rwx eslogsdir && sudo chgrp 0 eslogsdir
   

5. Prepare a certificate directory:

    mkdir certs && chmod g+rwx certs && sudo chgrp 0 certs
   

6. Generate CA (only if it doesn’t exist - first node should create and share with others):

    sudo apt-get install -y unzip
    sudo docker run --rm \
        -v $HOME/certs:/usr/share/elasticsearch/config/certs \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7 \
        bin/elasticsearch-certutil ca --silent --pem --out /usr/share/elasticsearch/config/certs/ca.zip
    cd $HOME/certs && unzip -o ca.zip && mv ca/* . && rmdir ca && rm ca.zip
   

7. Generate node certificate signed by CA:

    ES_NETWORK_HOST=$(hostname -I | awk '{print $1}')
    sudo docker run --rm \
        -v $HOME/certs:/usr/share/elasticsearch/config/certs \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7 \
        bin/elasticsearch-certutil cert --silent --pem \
        --ca-cert /usr/share/elasticsearch/config/certs/ca.crt \
        --ca-key /usr/share/elasticsearch/config/certs/ca.key \
        --dns master-1,localhost \
        --ip ${ES_NETWORK_HOST},127.0.0.1 \
        --out /usr/share/elasticsearch/config/certs/node.zip
    cd $HOME/certs && unzip -o node.zip && mv instance/* . && rmdir instance && rm node.zip
   

8. Create a custom elasticsearch.yml in your home directory with important settings:

    cluster.name: my-elasticsearch-cluster
    node.roles: master
    node.name: master-1
    network.host: 0.0.0.0
    discovery.seed_hosts: 10.0.0.1,10.0.0.2,10.0.0.3
    cluster.initial_master_nodes: master-1,master-2,master-3
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.key: certs/instance.key
    xpack.security.transport.ssl.certificate: certs/instance.crt
    xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
   

   • The configuration files should contain settings which are node-specific (such as node.name and paths), or settings which a node requires in order to be able to join a cluster (such as cluster.name and network.host), as most settings can be changed on a running cluster using the Cluster update settings API.
   • cluster.name: A node can only join a cluster when it shares its cluster.name with all the other nodes in the cluster. The default name is elasticsearch, but you should change it to an appropriate name that describes the purpose of the cluster.
   • node.roles: We set the role to master to create a dedicated master-eligible node. This is the most reliable way to avoid overloading the master with other tasks.
   • node.name: Elasticsearch uses node.name as a human-readable identifier for a particular instance of Elasticsearch. The node name defaults to the hostname of the machine when Elasticsearch starts.
   • network.host: Sets the address of this node for both HTTP and transport traffic.
   • discovery.seed_hosts: When you want to form a cluster with nodes on other hosts, this setting provides a list of other nodes in the cluster that are master-eligible and likely to be live and contactable to seed the discovery process.
   • cluster.initial_master_nodes: A list of node names of master-eligible nodes whose votes are counted in the first election. Do not configure this setting on master-ineligible nodes or nodes joining an existing cluster.
   • xpack.security.enabled: Set true to install Elastic Agent for stack monitoring.
   • xpack.security.transport.ssl.enabled: Set true as transport SSL must be enabled if security is enabled.
   • For a full list of configuration options, refer to Elasticsearch configuration reference.
9. Run Elasticsearch container:

   sudo docker run -d \
       --restart always \
       --group-add 0 \
       --volume $HOME/esdatadir:/usr/share/elasticsearch/data \
       --volume $HOME/eslogsdir:/usr/share/elasticsearch/logs \
       --volume $HOME/certs:/usr/share/elasticsearch/config/certs:ro \
       --ulimit nofile=65535:65535 \
       --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
       --publish-all \
       --network host \
       --volume $HOME/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
       --env TZ=Asia/Seoul \
       --env ELASTIC_PASSWORD=elasticpassword \
       --name elasticsearch \
       docker.elastic.co/elasticsearch/elasticsearch:8.19.7
   

   • --restart always: to automatically restarts the container if it stops.
   • --group-add 0: to ensure that the user under which Elasticsearch is running has file permissions to esdatadir.
   • --volume $HOME/...: to persist data, logs, and certificates across container restarts.
   • --ulimit nofile=65535:65535: to increase the file descriptor limit required for production workloads.
   • --publish-all: to randomize published ports, which is recommended for production clusters.
   • --network host: to optimize performance. The host network only works on Linux hosts and --publish-all option is ignored when it is enabled.
   • --volume $HOME/...: to bind-mount custom elasticsearch.yml, the preferred approach for production setups.
   • --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1: to disable swapping for performance and node stability.
   • --env TZ=Asia/Seoul: Sets the container timezone (adjust as needed).
   • ELASTIC_PASSWORD: Replace it with a password for your production Elasticsearch cluster.
   • Some setups use ES_JAVA_OPTS, but this is not recommended for production. Use the default Elasticsearch memory settings instead.

At this stage, the cluster isn’t fully formed yet, so you’ll see a warning in the container logs like:

master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [master-1, master-2, master-3] to bootstrap a cluster: ...

You won’t see this error if you’re using only a single master-eligible node.

This happens because we configured three initial master nodes—master-1, master-2, and master-3—but only master-1 has joined the cluster so far. Elasticsearch is still waiting for the remaining master-eligible nodes before it can complete the bootstrap process. To finish forming the cluster, you need to bring up and join the other master-eligible nodes (master-2 and master-3) as well.

Joining Master Nodes to the Cluster

1. Copy certs/ca.crt and certs/ca.key files to your local machine or another secure location. These will be reused when setting up the remaining nodes.
2. Connect to my-elasticsearch-node-master-2:

    ssh my-elasticsearch-node-master-2
   

3. Copy ca.crt and ca.key into the home directory on this node.
4. Install docker, set vm.max_map_count, prepare local directories, and generate node certificate:

    # install docker
    sudo apt-get update
    sudo apt-get install -y ca-certificates curl unzip
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
    echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
        sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo usermod -aG docker ${USER}
   
    # Set `vm.max_map_count` to at least 262144
    sudo sysctl -w vm.max_map_count=262144
    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
   
    # Prepare a local directory for storing data and logs through a bind-mount
    mkdir esdatadir && chmod g+rwx esdatadir && sudo chgrp 0 esdatadir
    mkdir eslogsdir && chmod g+rwx eslogsdir && sudo chgrp 0 eslogsdir
   
    # Prepare certificate directory
    mkdir certs && chmod g+rwx certs && sudo chgrp 0 certs
   
    mv ca.crt certs/
    mv ca.key certs/
   
    # Generate node certificate signed by CA
    ES_NETWORK_HOST=$(hostname -I | awk '{print $1}')
    sudo docker run --rm \
        -v $HOME/certs:/usr/share/elasticsearch/config/certs \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7 \
        bin/elasticsearch-certutil cert --silent --pem \
        --ca-cert /usr/share/elasticsearch/config/certs/ca.crt \
        --ca-key /usr/share/elasticsearch/config/certs/ca.key \
        --dns master-2,localhost \
        --ip ${ES_NETWORK_HOST},127.0.0.1 \
        --out /usr/share/elasticsearch/config/certs/node.zip
   
    cd $HOME/certs && unzip -o node.zip && mv instance/* . && rmdir instance && rm node.zip
   

5. Create a custom elasticsearch.yml in your home directory:

    cluster.name: my-elasticsearch-cluster
    node.roles: master
    node.name: master-2
    network.host: 0.0.0.0
    discovery.seed_hosts: 10.0.0.1,10.0.0.2,10.0.0.3
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.key: certs/instance.key
    xpack.security.transport.ssl.certificate: certs/instance.crt
    xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
   

   • node.name: now set to master-2
6. Run Elasticsearch container:

    sudo docker run -d \
        --restart always \
        --group-add 0 \
        --volume $HOME/esdatadir:/usr/share/elasticsearch/data \
        --volume $HOME/eslogsdir:/usr/share/elasticsearch/logs \
        --volume $HOME/certs:/usr/share/elasticsearch/config/certs:ro \
        --ulimit nofile=65535:65535 \
        --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
        --publish-all \
        --network host \
        --volume $HOME/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
        --env TZ=Asia/Seoul \
        --env ELASTIC_PASSWORD=elasticpassword \
        --name elasticsearch \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7
   

   • --env TZ=Asia/Seoul: Sets the container timezone (adjust as needed).
   • ELASTIC_PASSWORD: Replace it with a password for your production Elasticsearch cluster.
7. Connect to my-elasticsearch-node-master-3 and repeat steps 13-16, making sure to set node.name to master-3 in the elasticsearch.yml file.
8. Verify that the master nodes have formed a cluster:

    curl -u elastic:elasticpassword "http://localhost:9200/_cat/nodes?v&s=name"
   

   Output:

    ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
    10.0.0.1                 40          98   0    0.00    0.05     0.07 m         *      master-1
    10.0.0.2                 26          98   0    0.07    0.17     0.10 m         -      master-2
    10.0.0.3                 24          98   0    0.39    0.31     0.13 m         -      master-3
   

Since there are no data nodes to host shards yet, the cluster health will show up as red:

curl http://elastic:elasticpassword@localhost:9200/_cluster/health

Output:

{
    "cluster_name": "my-elasticsearch-cluster",
    "status": "red",
    "timed_out": false,
    "number_of_nodes": 3,
    "number_of_data_nodes": 0,
    "active_primary_shards": 0,
    "active_shards": 0,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 2,
    "unassigned_primary_shards": 2,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 0.0
}

This is expected. Once the data nodes join the cluster, Elasticsearch will be able to allocate shards and the status should move to yellow or green depending on replica allocation.

Joining Data Nodes to the Cluster

The process for adding a data node is very similar to adding a master node, with a few small changes in the elasticsearch.yml configuration.

1. Connect to my-elasticsearch-node-data-1:

    ssh my-elasticsearch-node-data-1
   

2. Copy ca.crt and ca.key into the home directory on this node.
3. Install docker, set vm.max_map_count, prepare local directories, and generate node certificate:

    # install docker
    sudo apt-get update
    sudo apt-get install -y ca-certificates curl unzip
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
    echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
        sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo usermod -aG docker ${USER}
   
    # Set `vm.max_map_count` to at least 262144
    sudo sysctl -w vm.max_map_count=262144
    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
   
    # Prepare a local directory for storing data and logs through a bind-mount
    mkdir esdatadir && chmod g+rwx esdatadir && sudo chgrp 0 esdatadir
    mkdir eslogsdir && chmod g+rwx eslogsdir && sudo chgrp 0 eslogsdir
   
    # Prepare certificate directory
    mkdir certs && chmod g+rwx certs && sudo chgrp 0 certs
   
    mv ca.crt certs/
    mv ca.key certs/
   
    # Generate node certificate signed by CA
    ES_NETWORK_HOST=$(hostname -I | awk '{print $1}')
    sudo docker run --rm \
        -v $HOME/certs:/usr/share/elasticsearch/config/certs \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7 \
        bin/elasticsearch-certutil cert --silent --pem \
        --ca-cert /usr/share/elasticsearch/config/certs/ca.crt \
        --ca-key /usr/share/elasticsearch/config/certs/ca.key \
        --dns data-1,localhost \
        --ip ${ES_NETWORK_HOST},127.0.0.1 \
        --out /usr/share/elasticsearch/config/certs/node.zip
   
    cd $HOME/certs && unzip -o node.zip && mv instance/* . && rmdir instance && rm node.zip
   

4. Create a custom elasticsearch.yml in your home directory:

    cluster.name: my-elasticsearch-cluster
    node.roles: data
    node.name: data-1
    network.host: 0.0.0.0
    discovery.seed_hosts: 10.0.0.1,10.0.0.2,10.0.0.3
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.key: certs/instance.key
    xpack.security.transport.ssl.certificate: certs/instance.crt
    xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
   

   • node.roles: now set to data.
   • node.name: now set to data-1
5. Run Elasticsearch container:

    sudo docker run -d \
        --restart always \
        --group-add 0 \
        --volume $HOME/esdatadir:/usr/share/elasticsearch/data \
        --volume $HOME/eslogsdir:/usr/share/elasticsearch/logs \
        --volume $HOME/certs:/usr/share/elasticsearch/config/certs:ro \
        --ulimit nofile=65535:65535 \
        --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
        --publish-all \
        --network host \
        --volume $HOME/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
        --env TZ=Asia/Seoul \
        --env ELASTIC_PASSWORD=elasticpassword \
        --name elasticsearch \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7
   

   • --env TZ=Asia/Seoul: Sets the container timezone (adjust as needed).
   • ELASTIC_PASSWORD: Replace it with a password for your production Elasticsearch cluster.
   • Joining the cluster can take a little time. Once a data node joins the cluster, the health status transitions from red to green:

       {
           "cluster_name": "my-elasticsearch-cluster",
           "status": "green",
           "timed_out": false,
           "number_of_nodes": 4,
           "number_of_data_nodes": 1,
           "active_primary_shards": 3,
           "active_shards": 3,
           "relocating_shards": 0,
           "initializing_shards": 0,
           "unassigned_shards": 0,
           "unassigned_primary_shards": 0,
           "delayed_unassigned_shards": 0,
           "number_of_pending_tasks": 0,
           "number_of_in_flight_fetch": 0,
           "task_max_waiting_in_queue_millis": 0,
           "active_shards_percent_as_number": 100.0
       }
     

6. Repeat steps 1-5 for my-elasticsearch-node-data-2 and my-elasticsearch-node-data-3, updating node.name to match each node.
7. From any node, verify that all nodes have successfully joined the cluster:

    curl -u elastic:elasticpassword "http://localhost:9200/_cat/nodes?v&s=name"
   

   Output:

    ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
    10.0.1.1                 41          97   0    0.00    0.06     0.08 d         -      data-1
    10.0.1.2                 36          98   0    0.22    0.33     0.16 d         -      data-2
    10.0.1.3                 33          98   0    0.14    0.24     0.11 d         -      data-3
    10.0.0.1                  8          98   0    0.00    0.00     0.00 m         *      master-1
    10.0.0.2                 44          97   0    0.00    0.00     0.00 m         -      master-2
    10.0.0.3                 42          97   0    0.00    0.00     0.00 m         -      master-3
   

At this point, all master and data nodes should be present, and the cluster should be fully formed and healthy.

Updating custom elasticsearch.yml

Updating a custom elasticsearch.yml in a Docker-based setup is straightforward:

1. Make your changes in elasticsearch.yml in your home directory.
2. Restart the container:

    docker restart elasticsearch.yml
   

Be aware that restarting the container will cause downtime. For production clusters, plan configuration changes carefully and apply them during a maintenance window, especially if the setting you’re changing affects cluster behavior or node roles.

Stack Monitoring

Stack Monitoring lets you collect logs and metrics from Elastic products, most importantly your production Elasticsearch nodes, and also things like Kibana. At a minimum, you need monitoring data for the production Elasticsearch cluster. Once that’s in place, Kibana can show monitoring data for other products in the Stack Monitoring page.

The legacy Elasticsearch Monitoring plugin approach (enabled via xpack.monitoring.collection.enabled) was deprecated in 7.16. These days, Elastic recommends using Elastic Agent or Metricbeat to collect and ship monitoring data to a monitoring cluster.

Next, we’ll deploy a single-node monitoring cluster and use Elastic Agent to monitor our production cluster.

Deploying Elasticsearch

1. Connect to my-elasticsearch-monitoring

    ssh my-elasticsearch-node-monitoring
   

2. Install docker, set vm.max_map_count, and prepare local directories for data persistency:

    # install docker
    sudo apt-get update
    sudo apt-get install -y ca-certificates curl
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
    echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
        sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo usermod -aG docker ${USER}
   
    # Set `vm.max_map_count` to at least 262144
    sudo sysctl -w vm.max_map_count=262144
    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
   
    # Prepare a local directory for storing data and logs through a bind-mount
    mkdir esdatadir && chmod g+rwx esdatadir && sudo chgrp 0 esdatadir
    mkdir eslogsdir && chmod g+rwx eslogsdir && sudo chgrp 0 eslogsdir
   

3. Create a custom elasticsearch.yml in your home directory:

    discovery.type: single-node
    xpack.security.enabled: true
    network.host: 0.0.0.0
   

4. Run Elasticsearch container:

    sudo docker run -d \
        --restart always \
        --group-add 0 \
        --volume $HOME/esdatadir:/usr/share/elasticsearch/data \
        --volume $HOME/eslogsdir:/usr/share/elasticsearch/logs \
        --ulimit nofile=65535:65535 \
        --env "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
        --publish-all \
        --network host \
        --volume $HOME/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
        --env TZ=Asia/Seoul \
        --env ELASTIC_PASSWORD=elasticpassword \
        --name elasticsearch \
        docker.elastic.co/elasticsearch/elasticsearch:8.19.7
   

   • --env TZ=Asia/Seoul: Sets the container timezone (adjust as needed).
   • ELASTIC_PASSWORD: Replace it with a password for your monitoring Elasticsearch node.
5. Verify:

    curl -u elastic:elasticpassword "http://localhost:9200/_cat/nodes?v&s=name"
   

   Output

    ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
    10.0.2.1                 22          97  44    0.92    0.31     0.11 cdfhilmrstw *      ip-10-0-2-1
   

   • Note that the node is assigned every role as it must do everything.

Deploying Kibana

1. Set a password for kibana_system user:

    curl -s -X POST -u "elastic:elasticpassword" -H "Content-Type: application/json" http://localhost:9200/_security/user/kibana_system/_password -d "{\"password\":\"kibanapassword\"}"
   

   • kibanapassword: Replace it with your password.
2. Create custom kibana.yml:

    server.host: "0.0.0.0"
    server.shutdownTimeout: "5s"
    elasticsearch.hosts: ["http://localhost:9200"]
    elasticsearch.username: "kibana_system"
    elasticsearch.password: "kibanapassword"
    xpack.encryptedSavedObjects.encryptionKey: "d7x9s2k5v8y4b3m6n1q0w7e2r5t8y9u1" # Replace this with your own key. You can generate one using `./kibana-encryption-keys generate`.
    server.name: kibana
   

   • elasticsearch.hosts: Use http://localhost:9200 since Elasticsearch is running on the same machine.
3. Run Kibana container:

    sudo docker run -d \
        --restart always \
        --network host \
        --env TZ=Asia/Seoul \
        --volume $HOME/kibana.yml:/usr/share/kibana/config/kibana.yml \
        --name kibana \
        docker.elastic.co/kibana/kibana:8.19.7
   

4. Verify the deployment by opening Kibana (http://10.0.2.1:5601 in this case) in your browser:

   

Installing Elasticsearch Integration

1. Create a user on the production cluster that has the remote_monitoring_collector built-in role.

    curl -X POST "http://localhost:9200/_security/user/elastic_agent?pretty" \
        -u elastic:elasticpassword \
        -H "Content-Type: application/json" \
        -d '{
            "password" : "changeme",
            "roles" : [ "remote_monitoring_collector"]
        }'
   

   • elastic_agent: This is just a username for Elastic Agent. Feel free to use a different one.
   • password: Choose a secure password for this user.
2. Go to Kibana in your browser.
3. Login with:
   • Username: elastic
   • Password: Your ELASTIC_PASSWORD
4. Click Add integrations.

5. In the query bar, search for and select the Elasticsearch integration for Elastic Agent:

   

6. Click Add Elasticsearch.
7. Configure the integration name and optionally add a description. Make sure you configure all required settings:
   • Go to Metrics (Stack monitoring) -> Change defaults:
     • Hosts: Specify data nodes, or a load balancer that routes to master-ineligible nodes:
       • http://10.0.1.1:9200
       • http://10.0.1.2:9200
       • http://10.0.1.3:9200
     • Advanced options:
       • Username: elastic_agent
       • Password: changeme
       • Scope: cluster (If set to node, Elastic Agent only collects metrics from the specified hosts.)
8. Click Save and continue. This may take a minute or two. When it finishes, an agent policy with the Elasticsearch monitoring integration will be created.
9. Click Add Elastic Agent later.

Deploying Elastic Agent

There are two ways to run Elastic Agent:

• Fleet-managed: Elastic Agent is centrally managed through Fleet in Kibana.
• Standalone: Elastic Agent is configured locally using YAML files.

In this tutorial, we’ll use the standalone approach to avoid the extra dependency on Fleet.

1. Go to Kibana for the monitoring cluster in your browser.
2. Go to Fleet -> Agents -> Add Agent -> Run standalone
3. Select Agent Policy 1.
4. Click Create API key to generate an API key for Elastic Agent.
   • By default, the output host is set to http://localhost:9200, which is where Elastic Agent will send monitoring data.
5. Click Copy to clipboard or Download policy
6. Connect to monitoring node:

    ssh my-elasticsearch-node-monitoring
   

7. Create an elastic-agent.yml file and paste the configuration from the clipboard, or copy the downloaded policy from step 5.
8. Run Elastic Agent container:

    docker run \
        -d \
        --restart always \
        --name elastic-agent \
        --network host \
        --user root \
        --volume $HOME/elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml \
        docker.elastic.co/beats/elastic-agent:8.19.7
   

   • --network host: Allows the agent to connect to Elasticsearch via http://localhost:9200.
   • Startup usually takes a few seconds.
9. Go to Kibana for the monitoring cluster in your browser.
10. Click Stack Monitoring in the left menu, then select Remind me later when prompted.

Once everything is set up successfully, Elastic Agent will start collecting and shipping monitoring data from the production Elasticsearch cluster. With this data in place, Kibana can display monitoring information for Elasticsearch and other Elastic Stack components:

Click Nodes for node monitoring:

Click specific node for detailed metrics:

So far, Elastic Agent runs on the same node as the monitoring Elasticsearch and Kibana, but it can be deployed on any other node as well. To do that, update the Outputs in the agent policy:

1. Go to Fleet -> Settings -> Outputs -> Add output
2. Set the monitoring Elasticsearch host to http://10.0.2.1:9200.

Updating the output will result in an elastic-agent.yml similar to the following:

...
outputs:
  default:
    type: elasticsearch
    hosts:
      - http://10.0.2.1:9200
...

When new nodes are added to the production cluster, Elastic Agent will automatically start collecting their metrics without requiring any output changes.

In practice, however, it’s advisable to place a load-balancing proxy in front of master-ineligible nodes. This distributes monitoring traffic evenly and prevents overloading master nodes, which must remain stable.

Creating a Load Balancer

Below is an example of setting up a load balancer with a TLS listener that forwards traffic only to master-ineligible data nodes in your Elasticsearch cluster:

1. Create a target group:
   • Target type: Instances
   • Target group name: my-elasticsearch
   • Protocol: TCP
   • Port: 9200
   • Health check protocol: TCP
   • Health check port: Trafic port (9200 in this case)
   • Targets:
     • my-elasticsearch-node-data-1
     • my-elasticsearch-node-data-2
     • my-elasticsearch-node-data-3
2. Create a networkd load balancer. This is a good choice for Elasticsearch because it has low latency and works well with TCP/TLS passthrough:
   • Load balancer name: my-elasticsearch
   • Scheme: Internal (recommended for private networks)
   • VPC: (Use the same VPC as your Elasticsearch nodes)
   • Security group: (Allow inbound traffic on TCP 443 from trusted sources)
   • Listeners:
     • Protocol: TLS
     • Port: 443
     • Target group: my-elasticsearch
     • Default SSL/TLS server certificate: (Your ACM or imported certificate)
   • With this setup, clients connect over HTTPS on port 443, and the NLB forwards traffic directly to port 9200 on the data nodes.
3. Create a DNS record (for example, in Route 53):
   • Use the hostname covered by your TLS certificate
   • Route to the network load balancer

After this, clients can reach Elasticsearch using a simple HTTPS URL. You can use this URL as the Elasticsearch hosts for Elastic Agent instead of listing the IP addresses of individual data nodes. When you add or remove data nodes, the only thing you need to update is the target group. The client configuration stays the same, and the load balancer automatically handles routing traffic to the available data nodes.

Shell Scripts for Easier Deployment

Often, we need to scale out the cluster to handle more traffic. Manually repeating every step in this guide is daunting.

To make life easier, here are complete scripts you can use:

• For initializing a fresh Elasticsearch cluster: init_cluster.sh
• For joining nodes to an exisiting Elasticsearch cluster: join_cluster.sh
• For initializing a single-node monitoring cluster: init_monitoring_stack.sh

Before running them, make sure you update these values to match your environment:

• VERSION
• Timezone
• ELASTIC_PASSWORD
• ca.crt and ca.key files (refer to REPLACE IT in the join_cluster.sh file)

Conclusion

Running a self-managed cluster means you own everything. You operate it, you watch it, and you fix it when it breaks. Scaling, monitoring, and compliance are all on you. That work costs time and money, and there’s no way around that.

The upside is what you don’t have. No extra control planes, no operators, no Kubernetes glue code deciding things on your behalf. Systems like ECK (Elastic Cloud on Kubernetes) add layers, and layers add complexity. Complexity creates edge cases, and edge cases could fail in unexpected ways.

If your environment can handle the operational load and you care about clarity, predictability, and knowing exactly how your system behaves, a self-managed cluster can be a good choice.

So, how do we measure it? Since information retrieval is fundamentally an empirical problem, its effectiveness has to be judged by users in some way. One obvious way is to look at what real users do. Metrics like click-through rate (CTR) and conversion rate fall into this category. These are often called online metrics. A/B testing is a popular way to compare different systems using this kind of data.

However, online experiments require deploying systems to actual users, which isn’t always practical or even possible in research settings. This is where systematic approaches based on offline metrics come in.

Cranfield Evaluation Methodology

Cranfield evaluation methodology is a framework for systematically evaluating the effectiveness of information retrieval systems, developed in the 1960s by Cyril Cleverdon at Cranfield University.

This methodology is built upon three key components:

• A collection of documents: a standardized set of documents to search against.
• A set of sample queries: representative queries that simulate user needs.
• Human-judged relevance judgments: a definitive list of which documents are considered relevant for each query.

Note that relevance judgments is by far the most important and the most labor-intensive part of the evaluation since they rely on human assessors. A common strategy to manage this is pooling, where assessors only judge a subset of results pulled together from several existing systems. However, this approach can introduce selection bias. If a new system retrieves relevant documents that never made it into the pool, those documents remain unjudged, and the system ends up being unfairly penalized.

Once we have all three components, the question becomes: how do we quantify the effectiveness?

Precision and Recall

Precision and recall are core metrics to quantify the effectiveness:

• Precision: Of all the documents the system retrieved, how many were actually relevant?
• Recall: Of all the relevant documents, how many did the system retrieve?

These concepts can be illustrated using a confusion matrix:

From this matrix, precision and recall are defined as follows:

• $Precision = \frac{TP}{TP + FP} = \frac{\text{Number of relevant retrieved}}{\text{Total number of retrieved}}$
• $Recall = \frac{TP}{TP + FN} = \frac{\text{Number of relevant retrieved}}{\text{Total number of relevant}}$

Here’s an example.

Suppose we have a small test collection of 10 documents about software:

Our query is “software testing practices”

A human evaluator goes through the entire collection and determines which documents are relevant to the query:

So, there are a total of 5 relevant documents in the collection.

Now, let’s say our retrieval system returns these 4 documents for the query:

• d2: Writing Unit Tests with pytest
• d5: The Practical Test Pyramid
• d9: Principles of Agile Development
• d10: Test-Driven Development Explained

Out of these 4, three are actually relevant (d2, d5, d10), while one (d9) is a false alarm. So:

• Precision = 3 / 4 = 0.75
• Recall = 3 / 5 = 0.6

In Python:

retrieved_docs = ["d2", "d5", "d9", "d10"]
relevant_docs = ["d2", "d5", "d6", "d8", "d10"]


def get_precision(retrieved: list[str], relevant: list[str]) -> float:
    true_positives = len(set(retrieved) & set(relevant))
    false_positives = len(set(retrieved) - set(relevant))
    if (true_positives + false_positives) > 0:
        return true_positives / (true_positives + false_positives)
    return 0


def get_recall(retrieved: list[str], relevant: list[str]) -> float:
    true_positives = len(set(retrieved) & set(relevant))
    false_negatives = len(set(relevant) - set(retrieved))
    if (true_positives + false_negatives) > 0:
        return true_positives / (true_positives + false_negatives)
    return 0

precision = get_precision(retrieved_docs, relevant_docs)
recall = get_recall(retrieved_docs, relevant_docs)

print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")

Output:

Precision: 0.75
Recall: 0.60

In practice, we often use a cutoff at a fixed number of top-ranked results. For example:

• Precision@5 (P@5): The precision when considering only the top 5 retrieved documents.
• Recall@10 (R@10): The recall when considering only the top 10 results.

The ideal system would achieve a precision and recall of 1, meaning it retrieved all relevant documents and no non-relevant ones. However, these two metrics has a classic trade-off: increasing recall by retrieving more documents often leads to a decrease in precision, and vice-versa. To balance the two, we use the F-measure.

F-Measure

F-measure (or F-score) is a way to combine precision and recall into a single score. Instead of just simply averaging them like (P+R)/2, it ensures that both precision and recall matter. The formula is:

Here, the parameter $\beta$ lets you put more weight on recall (if $\beta>1$) or on precision (if $\beta<1$).

The most common variant is the F1-measure, which calculates the harmonic mean of the two:

In Python:

def get_f_score(precision: float, recall: float, beta: float = 1.0) -> float:
    if (precision + recall) > 0:
        return (beta**2 + 1) * precision * recall / (beta**2 * precision + recall)
    return 0

With F1-measure, a system cannot achieve a high F1-score without having both reasonably high precision and high recall. For example,

Notice how the last row shows the penalty in action: even though precision is high (0.9), the low recall (0.1) drags the F1-score way down.

Limitation

The main drawback of F-measure is that it’s set-based so it ignores the order of results. A system that ranks the most relevant document at the top gets the same score as one that hides it at the last, even though the user experience is completely different.

To evaluate a ranked list, we need to consider the order of the results.

Precision-Recall (PR) Curve

The Precision–Recall (PR) curve shows how well a retrieval system balances precision and recall as you move down a ranked list of results. In simple terms, it helps you see whether the relevant documents are concentrated near the top (good) or scattered deeper in the list (not good).

We create the curve by plotting precision against recall at every rank position $k$.

Let’s walk through an example.

Suppose there are 8 relevant documents in total for a query, and our system manages to retrieve 6 of them within the top 10 results ($k$ = 10):

Now, to build the PR curve, we calculate precision and recall at each step:

When plotting the curve, we usually only take the points where a new relevant document appears so that the curve look smoother. That gives us the following (recall on the x-axis, precision on the y-axis):

And visually:

The curve makes it clear how front-loaded the relevant results are. If the curve stays high, the system surfaces relevant docs early. If it drops quickly, it means users would have to dig deeper before finding what they’re looking for.

A system whose curve is higher and more to the right is better. Therefore, an ideal system would have a constant precision of 1.0 for all recall levels.

PR curves are especially useful when comparing multiple retrieval systems. Take the example below:

As System A’s curve is consistently above System B’s, A is clearly superior—it delivers better precision at every level of recall.

However, things get trickier when the curves cross. Take the example below:

In this scenario, system A may have higher precision at low recall levels, while system B has higher precision at high recall levels. The better system depends on the user’s task. For example, a user doing a quick search might prefer System A’s high initial precision, whereas a researcher needing comprehensive results would prefer System B’s high recall.

Mean Average Precision (MAP)

Average Precision (AP)

When PR curves intersect, it’s useful to summarize performance with a single number. Average Precision (AP) does this by averaging the precision values at the ranks where relevant documents appear:

Where:

• $R$ is the total number of relevant documents for the query.
• $N$ is the total number of retrieved documents.
• $P(k)$ is the precision at the rank $k$.
• $Rel(k)$ is whether the document at rank $k$ is relevant (1) or not (0).

$R$ is NOT the number of retrieved relevant documents. If you were to divide by the number of retrieved relevant documents instead, you’d give an unfair advantage to a system that retrieves only a few documents. This would make the score look artificially high even if the system hardly retrieves anything useful.

AP is rank-sensitive: moving a relevant document higher in the list increases the score, while pushing it down lowers it. You can also think of it as a practical approximation of the area under the PR curve.

Let’s compute it for our example. We’ll take the precision at every rank where a relevant document shows up (d1, d2, d3, d5, d6, d8) and average them:

• Precision@d1 = 1/1
• Precision@d2 = 2/2
• Precision@d3 = 3/3
• Precision@d5 = 4/5
• Precision@d6 = 5/6
• Precision@d8 = 6/8

Now average these values:

In Python:

def get_average_precision(relevances: list[int], num_relevant: int) -> float:
    """
    Compute Average Precision (AP) for a single query.
    """
    if num_relevant == 0:
        return 0.0  # edge case: no relevant docs for this query

    precisions = []
    relevant_found = 0
    for k, rel in enumerate(relevances, start=1):  # rank positions start at 1
        if rel == 1:
            relevant_found += 1
            precision_at_k = relevant_found / k
            precisions.append(precision_at_k)

    return sum(precisions) / num_relevant


# Example usage
relevances = [1, 1, 1, 0, 1, 1, 0, 1, 0, 0]  # system output (10 results)
num_relevant = 8  # total relevant docs

AP = get_average_precision(docs, num_relevant)
print("Average Precision:", AP)

Average Precision is great for judging performance on a single query, but it only tells part of the story. To really understand how well a system works, we need to evaluate it across many queries and then aggregate the results. That way, the score reflects overall performance rather than just one example.

Mean Average Precision (MAP)

Mean Average Precision (MAP) is simply the mean of the Average Precision (AP) scores across all queries in the test set:

Where $Q$ is the total number of queries.

It provides a single, comprehensive number that summarizes the system’s ranking quality.

An alternative is the Geometric Mean Average Precision (gMAP), which uses a geometric mean instead of an arithmetic one. The choice between MAP and gMAP depends on the evaluation goal:

• MAP (Arithmetic Mean): The final score is more heavily influenced by high AP scores. This means MAP gives more weight to performance on “easy” queries where the system does well. It is useful if the goal is to measure overall performance, particularly on popular queries that are often easier.
• gMAP (Geometric Mean): This average is more sensitive to low AP scores. Therefore, gMAP emphasizes performance on “difficult” queries where the system struggles. It is useful if the goal is to improve the system’s performance on its weakest queries.

Here’s an example. Suppose we evaluate three queries and get the following AP scores:

• Query 1: 1.0
• Query 2: 0.5
• Query 3: 0.1

Then:

• MAP = $\frac{1.0 + 0.5 + 0.1}{3} \approx 0.53$
• gMAP = $(1.0 \times 0.5 \times 0.1)^{1/3} \approx 0.37$

Here, MAP emphasizes the strong performance on Query 1, while gMAP penalizes the weak score on Query 3 more heavily.

Mean Reciprocal Rank (MRR)

For certain search tasks, there is only one correct or relevant document. This occurs in “known-item searches,” where a user wants to find a single, specific page (e.g., the Youtube homepage), or in some question-answering tasks where there is only one right answer.

In this scenario, the Average Precision calculation simplifies to the Reciprocal Rank:

where $r$ is the rank position of the single relevant documen. For example, if the document is at rank 1, the score is 1. If it’s at rank 2, the score is 0.5, and so on.

The Mean Reciprocal Rank (MRR) is the average of these reciprocal ranks across all queries.

Here’s an example. Suppose we evaluate three queries and the relevant document appears at:

• Query 1: Rank 1 → Reciprocal Rank = 1/1
• Query 2: Rank 3 → Reciprocal Rank = 1/3
• Query 3: Rank 2 → Reciprocal Rank = 1/2

Then

• MRR = $\frac{1/1 + 1/3 + 1/2}{3} \approx 0.61$

Limitation

MAP and MRR are built on the assumption of binary relevance judgments, either a document is relevant (1) or not (0). But in reality, some documents are highly relevant, while others are only marginally relevant.

Let’s revisit our earlier example with the query “software testing practices”:

Here, documents like d2, d5, and d10 are highly relevant — they directly address testing practices. On the other hand, d6 and d8 are only somewhat relevant - they touch on quality practices but don’t focus on testing itself.

To handle this, we need a measure that accounts for graded relevance.

Normalized Discounted Cumulative Gain (nDCG)

Normalized Discounted Cumulative Gain (nDCG) is a metric for evaluating retrieval systems when relevance judgments are multi-level. It measures how good a ranking is by considering both the relevance and position.

In nDCG, relevance is usually graded on a scale from 1 to $r$ (where $r$ > 2). For example, if $r$ = 3, you might define:

• 1 = not relevant
• 2 = somewhat relevant
• 3 = highly relevant

The formula is:

Let’s walk through an example to see how this works.

Gain

The first step is assigning a gain value to each document, which is simply its relevance score:

Where $G_i$ is the gain of the result at position $i$.

Let’s take a simple example using a 3-level relevance scale ($r$ = 3). Suppose our system returns 5 results for a query q1, ranked from to to bottom:

So far, gain is just the raw relevance score.

Cumulative Gain (CG)

Cumulative Gain (CG) is the sum of gains up to rank $k$:

For our example:

So,

The problem with CG is that it ignores the rank order. If you change the top and bottom documents, the CG@5 is still 11. Clearly, that doesn’t reflect user experience because users care much more about the top results.

Discounted Cumulative Gain (DCG)

To make the metric rank-sensitive, the gain at each position is discounted by a logarithmic reduction factor (discount factor). The formula is:

Here, the denominator, $\log_2(i+1)$, reduces the contribution of documents as their rank gets lower. This means a highly relevant document contributes more to the score if it is ranked higher.

For our example:

So:

To compute this with Python, we first define two dictionaries:

qrels_dict = {"q1": {"d1": 3, "d2": 2, "d3": 1, "d4": 2, "d5": 3}}
run_dict = {"q1": {"d1": 0.9, "d2": 0.8, "d3": 0.7, "d4": 0.6, "d5": 0.5}}

• qrels_dict: the relevance judgments for each query and document.
• run_dict: the system’s predicted ranking scores. The exact values don’t matter for DCG as long as they preserve the order since what we ultimately care about is the ranking of documents.

A function to compute DCG:

import math


def get_dcg(qrels_dict: dict, run_dict: dict, k: int, query: str) -> float:
    """
    Compute DCG@k for a specific query.
    """
    # Extract documents for the specific query
    query_qrels = qrels_dict.get(query, {})
    query_run = run_dict.get(query, {})

    # Sort documents by their scores in run_dict (descending order)
    sorted_docs = sorted(query_run.items(), key=lambda x: x[1], reverse=True)

    dcg = 0.0
    for i in range(min(k, len(sorted_docs))):
        doc_id = sorted_docs[i][0]
        relevance = query_qrels.get(doc_id, 0)  # default to 0 if doc not in qrels
        rank = i + 1
        if rank == 1:
            dcg += relevance  # no log discount for rank 1
        else:
            dcg += relevance / (math.log2(rank + 1))  # log base 2
    return dcg


qrels_dict = {"q1": {"d1": 3, "d2": 2, "d3": 1, "d4": 2, "d5": 3}}
run_dict = {"q1": {"d1": 0.9, "d2": 0.8, "d3": 0.7, "d4": 0.6, "d5": 0.5}}

dcg = get_dcg(qrels_dict, run_dict, 5, "q1")
print(f"{dcg:.2f}")

Output:

6.78

Now the ranking matters, but there’s still an issue. DCG scores aren’t directly comparable across queries. Imagine one query has only one relevant document and another has five. Even if both systems ranked perfectly, the DCG for the “easy” query (with lots of relevant docs) will naturally be higher. Without normalization, comparing those scores is unfair.

Normalized Discounted Cumulative Gain (nDCG)

To solve this, we normalize DCG by dividing it by the ideal DCG (IDCG):

Where IDCG is the DCG of an ideal ranking where all the most relevant documents are at the top.

This scales the score between 0 and 1: A system with nDCG = 1 means it produced the ideal ranking. And values below 1 tell you how close (or far) the system is from that ideal.

Suppose the ideal ordering is: [3, 3, 2, 2, 1] for our example:

(Note that the ideal ordering could also be [3, 3, 3, 3, 2] or even [3, 3, 3, 3, 3], depending on how many highly relevant documents exist in the collection.)

So:

Finally:

In Python:

import math


def get_dcg(qrels_dict: dict, run_dict: dict, k: int, query: str) -> float:
    """
    Compute DCG@k for a specific query.
    """
    # Extract documents for the specific query
    query_qrels = qrels_dict.get(query, {})
    query_run = run_dict.get(query, {})

    # Sort documents by their scores in run_dict (descending order)
    sorted_docs = sorted(query_run.items(), key=lambda x: x[1], reverse=True)

    dcg = 0.0
    for i in range(min(k, len(sorted_docs))):
        doc_id = sorted_docs[i][0]
        relevance = query_qrels.get(doc_id, 0)  # default to 0 if doc not in qrels
        rank = i + 1
        if rank == 1:
            dcg += relevance  # no log discount for rank 1
        else:
            dcg += relevance / (math.log2(rank + 1))  # log base 2

    return dcg


def get_ndcg(qrels_dict: dict, run_dict: dict, k: int, query: str) -> float:
    """
    Compute nDCG@k for a specific query.
    """
    dcg = get_dcg(qrels_dict, run_dict, k, query)

    # Compute ideal DCG (iDCG)
    query_qrels = qrels_dict.get(query, {})
    ideal_relevances = sorted(query_qrels.values(), reverse=True)

    idcg = 0.0
    for i in range(min(k, len(ideal_relevances))):
        relevance = ideal_relevances[i]
        rank = i + 1
        if rank == 1:
            idcg += relevance
        else:
            idcg += relevance / (math.log2(rank + 1))

    if idcg == 0:
        return 0.0  # Avoid division by zero; if no relevant documents, nDCG is defined as 0

    ndcg = dcg / idcg

    return ndcg


qrels_dict = {"q1": {"d1": 3, "d2": 2, "d3": 1, "d4": 2, "d5": 3}}
run_dict = {"q1": {"d1": 0.9, "d2": 0.8, "d3": 0.7, "d4": 0.6, "d5": 0.5}}

ndcg = get_ndcg(qrels_dict, run_dict, 5, "q1")
print(f"{ndcg:.2f}")

Output:

0.95

In fact, there’s a Python library ranx that handles evaluation metrics like nDCG out of the box.

Here’s how you can compute nDCG@5 using ranx in just a few lines:

from ranx import Qrels, Run, evaluate

qrels_dict = {"q1": {"d1": 3, "d2": 2, "d3": 1, "d4": 2, "d5": 3}}
run_dict = {"q1": {"d1": 0.9, "d2": 0.8, "d3": 0.7, "d4": 0.6, "d5": 0.5}}

qrels = Qrels(qrels_dict)
run = Run(run_dict)

ndcg = evaluate(qrels, run, metrics="ndcg@5")
print(f"{ndcg:.2f}")

Output:

0.95

Limitation

When comparing two systems, looking only at the average nDCG score (or any single metric) isn’t enough. The question is: “How confident are we that the observed difference isn’t just because of the particular queries we happened to test?”

Imagine two systems:

• System A has an average score of 0.20
• System B has an average score of 0.40

At first glance, System B looks twice as good. But let’s dig deeper:

• Experiment 1 (low variance): System B beats System A on every single query. Here, it’s safe to say B is better.
• Experiment 2 (high variance): Some queries favor A, others favor B, and results swing wildly. The conclusion is much shakier.

This is where statistical significance tests come in.

Statistical Significance Testing

Statistical significance testing answers the question: “Is the observed difference real, or could it have happened by chance?”

Sign Test

One of the simplest approaches is the sign test. Instead of looking at averages, it counts how many times each system wins across queries.

Here’s an example with five queries and scores for two IR systems:

• Wins for System A: 1
• Wins for System B: 4

Here, System B clearly dominates by beating System A in 4 out of 5 queries. That’s a strong sign that B is better than A, at least on this test set.

Limitation

But here’s where things get tricky. Let’s extend the test set with four more queries:

• Wins for System A: 4
• Wins for System B: 5

Now, things don’t look so clear anymore. Even though System B has more wins overall, the margin is just a single query. With such small differences, it’s hard to say whether B is truly the better system or if the result is just due to chance. The sign test doesn’t capture how big the improvements are, only who won more often. To deal with this limitation, researchers often use more advanced significance tests, like the Wilcoxon signed-rank test.

Conclusion

For comparing different IR systems, MAP and nDCG are useful tools. They capture both relevance and ranking quality in a way that’s more informative than simple set-based measures.

But again, information retrieval is fundamentally an empirical problem. What a human assessor marks as “relevant” might not line up with what real users actually care about in the real-world context of search.

The takeaway is that to make reliable decisions, we need to evaluate across many queries. That’s how we avoid shaky averages and get confidence that one system is truly better than another.

References

• Evaluation measures (information retrieval) - Wikipedia
• Discounted cumulative gain - Wikipedia
• Text Retrieval and Search Engines by University of Illinois Urbana-Champaign

Search engines use a ranking function to score document relevance. We write it as $f(q,d)$, where $q$ is the query and $d$ is a document. The higher the $f(q,d)$, the higher the document appears in the search results.

Although various methods exist to estimate document relevance, BM25 (Best Matching 25) is widely regarded as the most practical and effective ranking function for this task. It’s been around since the 90s, and despite newer ranking models, BM25 is still a strong baseline and often tough to beat in real-world applications.

The BM25 formula looks like this:

At its heart, BM25 is just combining three heuristics:

• Term Frequency (TF)
• Inverse Document Frequencey (IDF)
• Document Length Normalization

In this post, I will explain each heuristic and show how they come together to form the BM25 formula.

Term Frequency (TF)

The simplest idea: if a query term appears more often in a document, that document is probably more relevant.

A TF-only ranking function’s formula can be represented as this:

Where:

• $f(q,d)$ is the relevance score of document $d$ for query $q$.
• $\sum_{i=1}^{n}$ is the summation over all terms ($t_1$, $t_2$, …, $t_n$) in the query $q$.
• $TF(t_i,d)$ is the frequency score of query term $t_i$ in $d$.

Then how do we define $TF()$?

A naive TF is just to apply a linear count; $y=x$. Therefore:

Where $c(t,d)$ is the raw count of term t in document d.

Let’s see an example. Suppose we have two documents in our collection:

• d1: The Bank of Korea is expected to lower its benchmark interest rate next month.
• d2: A lower interest rate will be welcomed by indebted households.

Query: “korea interest rate”

Note that this query contains three terms: {$t_1$, $t_2$, $t_3$} = {“Korea”, “interest”, “rate”}

Under this linear application of the raw count, the scores are:

Here, d2 contains two of the query terms—”interest” and “rate”—but misses “Korea”, and d1 matches all three. This ordering makes intuitive sense: d1 fully matches the query, so it gets a higher score.

But there’s a problem: just repeating a term many times can inflate the score too much. This opens the door to what’s called term spamming. Term spamming is when a document unnaturally repeats certain keywords over and over to trick the ranking function into thinking it’s more relevant.

Suppose we add one more document to the collection:

• d3: The interest rate charged on loans is often higher than the interest rate paid on deposits.

With this new collection, the scores are:

Obviously, this is not what we want. Instead, we’d prefer a system where repetition still helps, but each extra occurrence of the same word contributes less and less to the score. In other words, diminishing returns from repetition.

TF Transformation

One simple fix is to apply a sublinear transform like logarithmic transformation; $y=\log(x+1)$. Therefore:

(1 is added to avoid $\log(0)$.)

This way, the impact of each extra occurrence shrinks.

Still, it’s not perfect - TF is unbounded, meaning there’s no upper limit. A spammy document can still rack up arbitrarily high scores if it repeats a term enough times.

To improve it more, here comes a bounded transformation; $y=\frac{x(k+1)}{x+k}$. Therefore:

This function saturates at $(k+1)$, no matter how large $x$ gets. In other words, the function is upper-bounded by $k+1$.

Not only is this transformation capped at $(k+1)$—which stops a single term’s frequency from completely dominating the score—it’s also flexible. The parameter $k$ controls how quickly the curve saturates:

• If $k=0$, it reduces to a simple binary scheme: any occurrence of the term contributes a score of 1, no matter how many times it appears.
• If $k$ is very large, the function behaves almost linearly, getting closer to the linear term frequency.

With this, our TF-based ranking function becomes:

This TF function can be implemented in Python like below:

def tf(term: str, document: str, k: float = 1.2) -> float:
    """
    Calculate TF score using bounded transformation.
    """
    # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
    doc_terms = document.replace(".", " ").lower().split()
    term_count = doc_terms.count(term.lower())
    score = term_count * (k + 1) / (term_count + k)

    return score


def tf_ranking_function(query: str, document: str, k: float = 1.2) -> float:
    """
    Calculate TF score of a document with respect to a query.
    """
    score = sum(tf(term, document, k) for term in query.lower().split())
    return score

To keep things simple, this implementation only applies a couple of basic text normalization steps—removing punctuation and lowercasing. In a real search engine, though, you’d usually go further with more steps such as stemming, lemmatization, and stopword removal to make scoring more robust and accurate.

We can use this code to get the relevance score of a document for a given query. For instance:

score = tf_ranking_function(
    query="korea interest rate",
    document="The Bank of Korea is expected to lower its benchmark interest rate next month.",
)
print(score)

Output:

3.0

The scores for different k values are shown below:

This illustrates the role of $k$: a smaller $k$ dampens the impact of raw term frequency, while a larger $k$ makes the function behave closer to linear counting.

The key takeaway here is that with this transformation, d3 no longer unfairly outranks d1 just because it happens to repeat the query terms.

Limitation

TF alone isn’t enough to reliably measure relevance because not all words are equally informative.

As an example, consider one more document that contains three “interest rate”s:

• d4: The interest rate remains unchanged, but many fear this interest rate keeps loans costly while others welcome a stable interest rate.

Using our TF-only ranking function for the same query “korea interest rate”, repetition wins out:

As you can see, d4 comes out on top even though it never mentions “Korea”, which is likely to be a crucial term. That’s not what we want.

That’s where Inverse Document Frequency (IDF) comes in.

Inverse Document Frequencey (IDF)

Inverse Document Frequency (IDF) solves the problem of treating all words equally. The intuition is that rare words are more informative than common ones. For example, a rarer word like “Korea” can be a strong signal that a document is actually about the topic we care about. On the other hand, common words like “the” or “of” show up in almost every document, so they don’t really help us figure out which document is more relevant.

At its core, IDF penalizes overly common terms that appear everywhere and rewards rarer, more descriptive terms that help distinguish one document from another.

When we combine TF with IDF, we get the TF–IDF ranking function:

Then how do we define $IDF()$?

The standard notation of IDF is this:

Where:

• $N$ is the total number of documents in the collection. (cardinality of documents)
• $df(t)$ is the number of documents containing the term $t$. ($df$ stands for document frequency.)

In words, a word with a low document frequency (a rare word) will have a high IDF score, while a very common word will have a low IDF score.

In practice, however, smoothed IDF variants are often used.

Smoothing

In statistics and NLP, smoothing generally means adjusting counts slightly to avoid extreme cases like zeros or infinities.

One of the most common IDF variants with smoothing is this:

This is also what scikit-learn uses when calculating IDF.

Here’s why the formula has all those extra +1s:

• +1 in the denominator: Without it, if a term never appears in the collection ($df(t) = 0$), you’d end up dividing by zero. The +1 smooths this out so the math always works. You can think of it as “Even if the term is unseen, let’s pretend it appeared at least once.”
• +1 in the numerator: If a term shows up in every document ($df(t) = N$), then $\log\frac{N}{N} = \log(1) = 0$. That would make the term’s weight vanish completely. By adding 1 to the numerator, those super-common terms still get a tiny but non-zero weight.
• +1 outside the log: Even with smoothing inside the fraction, the log value can hover around zero or dip negative when terms are extremely common. Negative IDF values don’t make sense—TF-IDF weights should never imply “anti-importance.” Adding 1 outside guarantees that IDF stays positive, avoiding zeros or negatives.

Graphically:

With this smoothed IDF formula, our TF-IDF ranking function becomes:

In Python:

import math


def tf(term: str, document: str, k: float = 1.2) -> float:
    """
    Calculate TF score using bounded transformation.
    """
    # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
    doc_terms = document.replace(".", " ").lower().split()
    term_count = doc_terms.count(term.lower())
    score = term_count * (k + 1) / (term_count + k)

    return score


def idf(term: str, documents: list[str]) -> float:
    """
    Calculate Inverse Document Frequency (IDF) for a term.
    """
    N = len(documents)  # Total number of documents
    df_t = 0
    for doc in documents:
        # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
        if term.lower() in doc.replace(".", " ").lower().split():
            df_t += 1
    score = math.log((N + 1) / (df_t + 1)) + 1

    return score


def tf_idf_ranking_function(
    query: str, document: str, documents: list[str], k: float = 1.2
) -> float:
    """
    Calculate TF-IDF score
    """
    query_terms = query.lower().split()
    score: float = 0.0
    for term in query_terms:
        tf_score = tf(term, document, k)
        idf_score = idf(term, documents)
        score += tf_score * idf_score

    return score

Let’s revisit the query “korea interest rate” and four documents:

• d1: The Bank of Korea is expected to lower its benchmark interest rate next month.
• d2: A lower interest rate will be welcomed by indebted households.
• d3: The interest rate charged on loans is often higher than the interest rate paid on deposits.
• d4: The interest rate remains unchanged, but many fear this interest rate keeps loans costly while others welcome a stable interest rate.

Using TF-IDF ranking function, the scores are:

The ranking now matches user intent so that d1 rises above d4 because it contains the rarer term “Korea”.

Limitation

The ranking models discussed so far are inherently biased toward long documents. A longer document has a statistically higher chance of matching query terms, even if the matches are coincidental and scattered throughout the text.

Let’s look at one more document:

• d5: In South Korea, the central bank’s decision on the interest rate is closely watched by both businesses and households. Rising interest rate levels have slowed consumer spending, while exporters in Korea argue that a stable interest rate is necessary to remain competitive. Many in Korea believe that future growth depends on how carefully the government manages the interest rate policy.

Notice how d5 is stuffed with the words Korea and interest rate. TF-IDF would likely push this document to the very top, not because it’s the most relevant, but simply because it’s long and keeps repeating the keywords. The scores are:

That’s where Document Length Normalization comes in.

Document Length Normalization

The last heuristic is that longer documents naturally contain more terms, and therefore match queries more often. To counter this, their scores are adjusted relative to the average document length. The goal is simple: balance the bias toward long documents. In short, don’t reward a document just for being long.

Penalizing long documents must be done carefully to avoid over-penalization since a document can be long either because:

• It’s verbose: Some texts are just wordy. They take a simple idea and wrap it in way too many words. In this case, penalization makes sense.
• It’s comprehensive: Other texts are long because they actually cover more ground (like a news archive that bundles many different stories). These shouldn’t be punished too harshly, because their length comes from offering more useful, distinct content.

Pivoted Length Normalization

A robust solution to this problem is Pivoted Length Normalization. This method uses the average document length (avdl) of the entire collection as a “pivot” or reference point.

Here’s how it works in practice:

• Documents with a length equal to the avdl receive no penalty or reward (normalizer = 1).
• Documents longer than the avdl are penalized.
• Documents shorter than the avdl are rewarded.

The formula for the pivoted normalizer is:

Where:

• $|d|$ is the length of the document (number of terms).
• $avdl$ is the average document length across the whole collection.
• $b$ is a tuning parameter between 0 and 1.
  • If $b$ = 0, the normalizer is always 1, and no length normalization occurs.
  • As $b$ increases towards 1, the penalty for long documents and the reward for short documents become more aggressive.

So where does this normalizer actually get applied?

In BM25, it slides into the denominator of the TF term. Here’s the BM25 TF function:

Finally, we arrive at the BM25 (Okapi) ranking function by integrating all three heuristics—TF Transformation, IDF Weighting, and Pivoted Length Normalization:

Where:

• $k$ is the term frequency saturation parameter
• $b$ is the document-length normalization weight (0–1)

In Python:

import math


def tf_with_normalizer(term: str, document: str, k: float = 1.2, normalizer: float = 1.0) -> float:
    """
    Calculate TF score for a term using BM25 with document length normalization.
    """
    # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
    doc_terms = document.replace(".", " ").lower().split()
    term_count = doc_terms.count(term.lower())
    score = term_count * (k + 1) / (term_count + k * normalizer)

    return score


def idf(term: str, documents: list[str]) -> float:
    """
    Calculate Inverse Document Frequency (IDF) for a term.
    """
    N = len(documents)  # Total number of documents
    df_t = 0
    for doc in documents:
        # Apply some basic text normalization: 1. punctuation handling 2. lowercasing
        if term.lower() in doc.replace(".", " ").lower().split():
            df_t += 1
    score = math.log((N + 1) / (df_t + 1)) + 1

    return score


def document_length_normalization(document: str, documents: list[str], b: float = 0.75) -> float:
    """
    Calculate document length normalization factor.
    """
    # Calculate average document length
    total_length = sum(len(doc.split()) for doc in documents)
    avdl = total_length / len(documents)
    doc_length = len(document.split())
    normalizer = 1 - b + b * (doc_length / avdl)

    return normalizer


def bm25_ranking_function(
    query: str, document: str, documents: list[str], k: float = 1.2, b: float = 0.75
) -> float:
    """
    Calculate BM25 score for a document given a query.

    Args:
        query: The query string.
        document: The document string.
        documents: The list of all documents. (collection)
        k: The term frequency saturation parameter.
        b: The length normalization parameter.
    """
    terms = query.lower().split()
    normalizer = document_length_normalization(document, documents, b)
    score: float = 0.0
    for term in terms:
        tf_score = tf_with_normalizer(term, document, k, normalizer)
        idf_score = idf(term, documents)
        score += tf_score * idf_score

    return score

Using the BM25 ranking function, the scores for our example become more acceptable:

What happens here is that even though d5 is long and loaded with repeated mentions of Korea and interest rate, it doesn’t unfairly dominate the ranking anymore.

Conclusion

BM25 is the backbone for many retrieval systems because it’s simple, fast, and surprisingly effective. But here’s one clear limitation: BM25 doesn’t actually understand context. It sees words, counts them, and scores documents based on math, not semantics.

Take these two documents:

• d1: The Bank of Korea is expected to lower its benchmark interest rate next month.
• d6: The interest in street dance across Korea has grown at such a fast rate.

And the query: “korea interest rate”

From BM25’s perspective, both documents look almost identical. The same words appear, with the same frequency, so the scores end up nearly the same. But to a human, the difference is obvious: d1 is clearly more relevant than d6.

That’s why modern search engines rarely rely on BM25 alone. Here are some techniques often used to address BM25’s limitations:

• Relevance feedback: learning from user clicks or pseudo-relevance feedback.
• Semantic embeddings: capturing meaning rather than just word overlap.
• Learning to Rank (LTR): training a machine learning model that combines multiple signals (BM25 scores, embeddings, click data, etc.) to optimize search result ordering.

References

• Okapi BM25 - Wikipedia
• tf–idf - Wikipedia
• Similarity in Elasticsearch by Konrad Beiske
• Text Retrieval and Search Engines by University of Illinois Urbana-Champaign

(YoutTube’s query auto-completion)

This post is a walk-through of building Korean query auto-completion feature based on past searches using Elasticsearch.

What problems does query auto-completion solve?

• Typing can be slow and error-prone. This gets even trickier depending on the user’s environment like mobile. For car navigation, query auto-completion isn’t just nice to have, it’s a must.
• Users sometimes don’t know the exact keywords to search. Query auto-completion helps them find what they want faster, even if they’re not sure what to type.

How is Korean query auto-completion different from English?

The simplest way to build query auto-completion is using just simple prefix matching.

For English, that could be enough with some lowercasing thrown in. For example, if your dataset includes apple, autocomplete, and artificial intelligence, typing just a or A will match all of them.

For Korean, that approach doesn’t quite cut it. Say your dataset includes 가구 (furniture), 간식 (snack), and 겨울 코트 (winter coat). They all start with the consonant ㄱ, but if a user types just ㄱ, none of those will match. That’s because ㄱ, 가, 간, and 겨 are completely different characters. If you want 가구 to show up in the suggestions, you have to type at least 가.

To give users a better experience, you need to break Hangul syllables apart into their building blocks—a process called syllable decomposition. That’s the key to making Korean auto-completion actually work better.

Hangul syllables and Jamo

Hangul (한글) is the Korean alphabet, made up of Hangul Jamos (자모). These Jamos are the Korean consonants and vowels, which are basically the building blocks of Korean syllables. There are 19 consonants and 21 vowels in Jamo.

19 consonants:

• Initial RR: Romanization of consonants at the start of a syllable. These are the sounds you hear starting a Korean syllable.
• Final RR: Romanization of consonants at the end of a syllable. “-“ means these letters don’t appear at the end of syllables.

21 vowels:

Here are some examples of Korean syllables made by combining jamos:

That’s pretty much it. Once you’ve got this down, you can read and write almost any Korean.

Hangul system is remarkably unique in that its letters are made to look like the mouth and tongue shapes used to say the sounds. It is designed so that anyone can easily learn to read and write.

The example in this post uses the following software versions:

• Python 3.13.4 (with pip install elasticsearch==8.13.2 "fastapi[standard]" aiohttp)
• Elasticsearch 8.13.4
• Kibana 8.13.4

Elasticsearch and Kibana are running locally at http://localhost:9200 and http://localhost:5601, respectively.

Context

You’re building an e-commerce application with a search bar. Every time a user types a search query, you’re logging that into an Elasticsearch index called search-logs with the following mappings:

PUT /search-logs
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "query": {
        "type": "keyword"
      },
      "user_id": {
        "type": "keyword"
      }
    }
  }
}

For demo purposes, let’s insert some sample data:

POST /search-logs/_bulk
{"index":{}}
{"@timestamp":"2025-06-21T09:00:00Z","query":"가구","user_id":"user1"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:01Z","query":"가방","user_id":"user2"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:02Z","query":"가방","user_id":"user3"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:03Z","query":"간식","user_id":"user4"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:04Z","query":"겨울 코트","user_id":"user5"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:05Z","query":"사과","user_id":"user6"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:06Z","query":"1234567890","user_id":"user7"}
{"index":{}}
{"@timestamp":"2025-06-21T09:00:07Z","query":"airpods 4","user_id":"user8"}

Solution

There are plugins like nori for Korean, but I’m skipping those to build the logic ourselves for full control and to better understand how Korean QAC actually works under the hood.

Another possible solution is using Edge n-gram tokenizer. With this approach, you can set up a custom tokenizer in your index and tweak options like min_gram, max_gram, and token_chars to fine-tune how autocomplete behaves. That said, I won’t be covering this method here.

Instead, I’ll use Elasticsearch’s completion suggester which is a more purpose-built option. I’ll also take a look at its limitations and walk through ways to gradually improve the experience.

Version0: Simple prefix matching

Start simple. No syllable decomposition, just raw keyword prefix matching using Elasticsearch’s completion suggester.

First, create a keyword suggestion index named search-keywords:

PUT /search-keywords
{
  "mappings": {
    "properties": {
      "suggest": { // you can use any name here instead of "suggest"
        "type": "completion",
        "analyzer": "standard"
      }
    }
  }
}

The completion suggester is designed to be as fast as possible because auto-completion should keep pace with your typing, giving you instant, relevant suggestions as you go. This is why auto-completion is also often called as “search-as-you-type” functionality.

We also use the standard analyzer instead of the default simple analyzer to make sure numbers are handled properly. The simple analyzer strips out digits completely, so a query like “1” won’t match “1234567890” in our dataset. The standard analyzer keeps numbers intact—so numeric queries work just like you’d expect.

Now that we have two indices, search-logs and search-keywords, we can read from search-logs and populate search-keywords using this Python script:

# update_keywords0.py
from dataclasses import dataclass

from elasticsearch import Elasticsearch, helpers


def main():
    hosts = ["http://localhost:9200"]
    username = "elastic"
    password = "elasticpassword"
    with Elasticsearch(hosts=hosts, basic_auth=(username, password)) as es_client:
        top_queries = _get_top_queries_from_search_logs(es_client=es_client, size=10000)
        success_count = _update_search_keywords(es_client, keywords=top_queries)
        print("Success count:", success_count)


@dataclass
class Keyword:
    key: str
    count: int


def _get_top_queries_from_search_logs(es_client: Elasticsearch, size: int) -> list[Keyword]:
    """
    Get the top search queries from the search log index.
    We use composite aggregation to handle large datasets efficiently.
    """
    index = "search-logs"

    all_buckets = []
    after_key = None  # for pagination
    while True:
        comp_query = {
            "size": 0,
            "aggs": {
                "top_queries": {
                    "composite": {
                        "size": size,
                        "sources": [{"term": {"terms": {"field": "query"}}}],
                        **({"after": after_key} if after_key else {}),
                    }
                }
            },
        }
        res = es_client.search(index=index, body=comp_query)
        buckets = res["aggregations"]["top_queries"]["buckets"]
        all_buckets.extend(buckets)
        after_key = res["aggregations"]["top_queries"].get("after_key")
        if not after_key:
            break

    # Convert the buckets to a list of Keyword dataclass instances
    keywords: list[Keyword] = []
    for item in all_buckets:
        keywords.append(Keyword(key=item["key"]["term"].strip(), count=item["doc_count"]))

    return keywords


def _update_search_keywords(es_client: Elasticsearch, keywords: list[Keyword]) -> int:
    """
    Update the autocomplete index with new data.
    Return the number of successfully executed actions (including overrides).
    """
    index = "search-keywords"

    actions = []
    for keyword in keywords:
        action = {
            "_op_type": "index",
            "_index": index,
            "_id": keyword.key,
            "_source": {
                "suggest": {
                    "input": [
                        keyword.key,  # Use the keyword itself as input
                    ],
                    "weight": keyword.count,
                }
            },
        }
        actions.append(action)

    res = helpers.bulk(es_client, actions)

    return res[0]


if __name__ == "__main__":
    main()

What it does:

1. Aggregates the top queries from search-logs, using a composite aggregation which efficiently handles large datasets by paginating results. (Without composite aggregation, there is a size limit to the number of buckets.)
2. Bulk-indexes them into search-keywords with:
   • input: the actual query
   • weight: how often it was searched. This way, more frequently searched keywords show up higher in suggestions.

In practice, if you’re dealing with a large volume of logs, you’ll want to add a range filter to speed things up and avoid processing stale data. For example, here’s how to retrieve search logs from the past week:

{
    "size": 0,
    "query": {"bool": {"filter": [{"range": {"@timestamp": {"gte": "now-7d"}}}]}},
    ...
}

Running the code will output:

Success count: 7

In practice, again, you can run this code periodically to keep your dataset fresh and up to date.

Verify the documents by using _search API in Kibana Dev Tools:

GET /search-keywords/_search

Next up, use FastAPI to expose an endpoint /keyword-suggestions:

# app.py
from contextlib import asynccontextmanager
from typing import Annotated

from elasticsearch import AsyncElasticsearch
from fastapi import Depends, FastAPI, Query
from fastapi.middleware.cors import CORSMiddleware

# Create an Elasticsearch client
hosts = ["http://localhost:9200"]
username = "elastic"
password = "elasticpassword"
es_client = AsyncElasticsearch(hosts=hosts, basic_auth=(username, password))


@asynccontextmanager
async def lifespan(app: FastAPI):
    yield
    print("Closing Elasticsearch client...")
    await es_client.close()


# Create FastAPI app with CORS middleware
app = FastAPI(title="Keyword Suggestion API", lifespan=lifespan)
app.add_middleware(CORSMiddleware, allow_origins=["*"])


def get_es_client() -> AsyncElasticsearch:
    return es_client


@app.get(
    "/keyword-suggestions",
    response_model=list[str],
    summary="Suggest keywords for a given query",
    description="Returns a list of keyword suggestions based on the input query.",
)
async def suggest_keywords(
    es_client: Annotated[AsyncElasticsearch, Depends(get_es_client)],
    query: Annotated[str, Query(description="User input query for keyword suggestions")],
    limit: Annotated[int, Query(description="Maximum number of keywords to return", gt=0)] = 10,
):
    body = {
        "suggest": {
            "keyword-suggest": {
                "prefix": query,
                "completion": {
                    "field": "suggest",
                    "size": limit,
                },
            }
        }
    }
    res = await es_client.search(index="search-keywords", body=body)
    results = []
    for option in res["suggest"]["keyword-suggest"][0]["options"]:
        results.append(option["_source"]["suggest"]["input"][0])

    return results

This FastAPI app handles Korean keyword autocomplete using Elasticsearch. Here’s how it works:

1. Set up and manages an Elasticsearch client.
2. Expose a /keyword-suggestions endpoint that takes a query and an optional limit.
3. Query the search-keywords index using Elasticsearch’s completion suggester.
4. Return a list of matching keyword suggestions.

The core of the logic lives in this Elasticsearch query:

body = {
    "suggest": {
        "keyword-suggest": {
            "prefix": query,
            "completion": {
                "field": "suggest",
                "size": limit,
            },
        }
    }
}

Here’s a quick breakdown:

• prefix: the user’s input—we’re trying to autocomplete this.
• completion: tells Elasticsearch to use the completion suggester.
• field: the field in the index we defined to support autocomplete (in this case, suggest).

You can run the app with:

fastapi run

Then head to http://localhost:8000/docs to try it out.

If you hit the endpoint with query=가, you’ll get a response like:

[
  "가방",
  "가구"
]

Note that 가방 comes first because it has a higher weight (i.e., it’s been searched more often). It’s a simple heuristic: the more people search for something, the higher it shows up.

Here are a couple of limitations:

• Typing a single jamo like ㄱ won’t return anything even if some keywords start with that consonant.
• Typing 갑 or 가바 (a typo or maybe just middle of typing 가방) won’t return anything either.

Version1: Syllable decomposition

To support keyword suggestions even when the user types only jamo (like ㄱ, ㄱㄱ, etc.), we need to make a few changes to how we index and search.

We’ll re-create a search-keywords index:

PUT /search-keywords
{
  "mappings": {
    "properties": {
      "suggest": { // you can use any name here instead of "suggest"
        "type": "completion",
        "analyzer": "standard"
      },
      "keyword": {
        "type": "keyword"
      }
    }
  }
}

Now, we’ve separated responsibilities by defining two fields:

• suggest: used for querying suggestions.
• keyword: stores the original keyword to display to users.

Next, we need to break each Korean syllable into its jamos (choseong, jungseong, and jongseong) so the completion suggester can match partial inputs.

Here’s a utility function to do that:

# Hangul.py
"""
Hangul (Korean alphabet) processing utilities.

This module provides functions for working with Korean text, including:
- Extracting initial consonants (choseong)
- Splitting Korean syllables into individual jamo (consonants and vowels)
- Converting Korean text to romanized form based on keyboard layout

The module uses Unicode code points to manipulate Korean characters.
"""

# For long lists, use `extend` in favor of readability

# 19 initial consonants
CHOSEONGS = []
CHOSEONGS.extend(["ㄱ", "ㄲ", "ㄴ", "ㄷ"])
CHOSEONGS.extend(["ㄸ", "ㄹ", "ㅁ", "ㅂ"])
CHOSEONGS.extend(["ㅃ", "ㅅ", "ㅆ", "ㅇ"])
CHOSEONGS.extend(["ㅈ", "ㅉ", "ㅊ", "ㅋ"])
CHOSEONGS.extend(["ㅌ", "ㅍ", "ㅎ"])

# 21 vowels
JUNGSEONGS = []
JUNGSEONGS.extend(["ㅏ", "ㅐ", "ㅑ", "ㅒ"])
JUNGSEONGS.extend(["ㅓ", "ㅔ", "ㅕ", "ㅖ"])
JUNGSEONGS.extend(["ㅗ", "ㅘ", "ㅙ", "ㅚ"])
JUNGSEONGS.extend(["ㅛ", "ㅜ", "ㅝ", "ㅞ"])
JUNGSEONGS.extend(["ㅟ", "ㅠ", "ㅡ", "ㅢ"])
JUNGSEONGS.extend(["ㅣ"])

# 28 final consonants (including no-final)
JONGSEONGS = []
JONGSEONGS.extend([" ", "ㄱ", "ㄲ", "ㄳ"])
JONGSEONGS.extend(["ㄴ", "ㄵ", "ㄶ", "ㄷ"])
JONGSEONGS.extend(["ㄹ", "ㄺ", "ㄻ", "ㄼ"])
JONGSEONGS.extend(["ㄽ", "ㄾ", "ㄿ", "ㅀ"])
JONGSEONGS.extend(["ㅁ", "ㅂ", "ㅄ", "ㅅ"])
JONGSEONGS.extend(["ㅆ", "ㅇ", "ㅈ", "ㅊ"])
JONGSEONGS.extend(["ㅋ", "ㅌ", "ㅍ", "ㅎ"])


def decompose_syllables(text: str) -> list[str]:
    """
    Decompose Hangul syllables into individual jamo (consonants and vowels).

    Example:
        "사과" -> ["ㅅ", "ㅏ", "ㄱ", "ㅘ"]
    """
    array = []
    for character in list(text.replace(" ", "").strip()):
        if "가" <= character <= "힣":  # For Hangul syllables
            offset = ord(character) - ord("가")
            choseong = offset // 588
            jungseong = (offset - (588 * choseong)) // 28
            jongseong = offset - (588 * choseong) - (28 * jungseong)
            array.append([CHOSEONGS[choseong], JUNGSEONGS[jungseong], JONGSEONGS[jongseong]])
        else:
            array.append([character])

    return [char for sublist in array for char in sublist if not char == " "]

To explain in detail, there are 19 initial consonants * 21 vowels * 28 final consonants (including no-final) = 11,172 possible combinations for modern Hangul syllables. And they occupy consecutive Unicode code points staring at U+AC00 (“가”, 44032) and ending at U+D7A3 (“힣”, 55203). So, the check if "가" <= character <= "힣": ensures we only apply the decomposition math to Hangul syllables.

The Unicode code points for Hangul syllables advance first by final consonant, then by medial vowel, and last by initial consonant. For example:

• by 28 final consonant:
  • 가: 44032
  • 각: 44033 (advanced by 1)
  • 갂: 44034
  • …
• by 21 medial vowel:
  • 가: 44032
  • 개: 44060 (advanced by 28)
  • 갸: 44088
  • …
• by 19 initial consonant:
  • 가: 44032
  • 까: 44620 (advanced by 21 * 28)
  • 나: 45208
  • …
  • 힣: 55203

Thus, every Hangul syllable code point can be thought of as:

code_point = 0xAC00 + (initial_index * 21 * 28) + (medial_index * 28) + final_index

where initial_index, medial_index, and final_index are zero-based positions in the respective jamo lists.

Therefore:

offset = ord(character) - ord("가")  # ord("가") == 0xAC00 == 44032
choseong = offset // 588  # initial_index (0–18)
jungseong = (offset - (588 * choseong)) // 28  # medial_index (0–20)
jongseong = offset - (588 * choseong) - (28 * jungseong)  # final_index (0–27)

(588 here is just the precomputed value for 21 * 28.)

For example:

Once you have those three indices, you simply look up the corresponding jamo in the three lists:

CHOSEONGS[choseong], JUNGSEONGS[jungseong], JONGSEONGS[jongseong]

As a result, the decompose_syllables takes a string like 사과 and turns it into ["ㅅ", "ㅏ", "ㄱ", "ㅘ"].

This utility function is used to decompose the syllables and feed that into the suggest field while the original keyword goes into keyword field:

# update_keywords1.py
from Hangul import decompose_syllables

...

def _update_search_keywords(es_client: Elasticsearch, keywords: list[Keyword]) -> int:
    """
    Update the autocomplete index with new data.
    Return the number of successfully executed actions (including overrides).
    """
    index = "search-keywords"

    actions = []
    for keyword in keywords:
        action = {
            "_op_type": "index",
            "_index": index,
            "_id": keyword.key,
            "_source": {
                "suggest": {
                    "input": [
                        "".join(decompose_syllables(keyword.key)),  # changed
                    ],
                    "weight": keyword.count,
                },
                "keyword": keyword.key,
            },
        }
        actions.append(action)

    res = helpers.bulk(es_client, actions)

    return res[0]

Running this code will output the same as the previous one:

Success count: 7

Next, modify the FastAPI endpoint to:

• Decompose the user query.
• Use the decomposed form for prefix search.
• Return the original keywords (from keyword field) as the suggestions.

# app.py
from Hangul import decompose_syllables

...

@app.get(
    "/keyword-suggestions",
    response_model=list[str],
    summary="Suggest keywords for a given query",
    description="Returns a list of keyword suggestions based on the input query.",
)
async def suggest_keywords(
    es_client: Annotated[AsyncElasticsearch, Depends(get_es_client)],
    query: Annotated[str, Query(description="User input query for keyword suggestions")],
    limit: Annotated[int, Query(description="Maximum number of keywords to return", gt=0)] = 10,
):
    body = {
        "suggest": {
            "keyword-suggest": {
                "prefix": "".join(decompose_syllables(query)),  # changed
                "completion": {
                    "field": "suggest",
                    "size": limit,
                },
            }
        }
    }
    res = await es_client.search(index="search-keywords", body=body)
    results = []
    for option in res["suggest"]["keyword-suggest"][0]["options"]:
        results.append(option["_source"]["keyword"])  # changed

    return results

With these changes, autocomplet becomes more responsive, like this:

One limitation of this improvement is that people sometimes search with first consonants, like ㄱㄱ for 가구 and ㅅㄱ for 사과.

Version2: Initial consonant search

Initial consonant (초성) search is one of the unique traits of the Korean language. Instead of typing full syllables, people often just type the initial sounds. For example, to search for “맥도날드” (McDonald’s), you might just type ㅁㄷㄴㄷ.

To make this work, add this function to Hangul.py:

# Hangul.py

...

def extract_choseongs(text: str) -> list[str]:
    """
    Extracts the initial consonants (choseong) from Korean text.

    Example:
        "사과" -> ["ㅅ", "ㄱ"]
    """
    result = []
    for character in text.replace(" ", "").strip():
        if "가" <= character <= "힣":  # For Hangul syllables
            code = ord(character) - ord("가")
            choseong = code // (21 * 28)
            result.append(CHOSEONGS[choseong])
        else:
            result.append(character)

    return result

This function simply extracts first consonant (choseong) from each Hangul syllable and returns as a list of them. For example:

Now we update our keyword indexing logic to include initial consonants in the suggest.input array:

# update_keywords2.py
from Hangul import decompose_syllables, extract_choseongs

...

def _update_search_keywords(es_client: Elasticsearch, keywords: list[Keyword]) -> int:
    """
    Update the autocomplete index with new data.
    Return the number of successfully executed actions (including overrides).
    """
    index = "search-keywords"

    actions = []
    for keyword in keywords:
        action = {
            "_op_type": "index",
            "_index": index,
            "_id": keyword.key,
            "_source": {
                "suggest": {
                    "input": list(
                        set(  # Remove duplicate items if keyword is not Korean
                            [
                                "".join(decompose_syllables(keyword.key)),
                                "".join(extract_choseongs(keyword.key)),  # new line
                            ]
                        )
                    ),
                    "weight": keyword.count,
                },
                "keyword": keyword.key,
            },
        }
        actions.append(action)

    res = helpers.bulk(es_client, actions)

    return res[0]

This way, our autocomplete can handle initial consonant input. For example:

Our query auto-completion feature now feels a lot better now, but there’s still room to improve. What’s the common problem we haven’t catched yet? It’s that users sometimes type Korean words on an English keyboard — like tkrhk when they meant 사과.

Version3: Latin-to-Jamo mapping

To type in Korean, users need to switch their keyboard to the Korean layout. But it’s super common for people to forget and just start typing in English mode whether they’re on a phone or a PC.

For example, someone trying to search for 가구 might accidentally type rkrn—because that’s what shows up when you type 가구 on a standard Korean keyboard while in English mode.

To automatically correct such typos, we map Korean syllables to their corresponding Latin keyboard inputs so the autocomplete system can recognize and suggest the intended Korean keywords even when users accidentally type using the English keyboard layout.

Here’s a utility function that maps Hangul jamo to their Latin keyboard equivalents. Add this to Hangul.py:

# Hangul.py

...

ENG_TO_KOR = {}
# Consonants
ENG_TO_KOR.update({"r": "ㄱ", "R": "ㄲ", "rt": "ㄳ", "s": "ㄴ"})
ENG_TO_KOR.update({"sw": "ㄵ", "sg": "ㄶ", "e": "ㄷ", "E": "ㄸ"})
ENG_TO_KOR.update({"f": "ㄹ", "fr": "ㄺ", "fa": "ㄻ", "fq": "ㄼ"})
ENG_TO_KOR.update({"ft": "ㄽ", "fx": "ㄾ", "fv": "ㄿ", "fg": "ㅀ"})
ENG_TO_KOR.update({"a": "ㅁ", "q": "ㅂ", "Q": "ㅃ", "qt": "ㅄ"})
ENG_TO_KOR.update({"t": "ㅅ", "T": "ㅆ", "d": "ㅇ", "w": "ㅈ"})
ENG_TO_KOR.update({"W": "ㅉ", "c": "ㅊ", "z": "ㅋ", "x": "ㅌ"})
ENG_TO_KOR.update({"v": "ㅍ", "g": "ㅎ"})
# Vowels
ENG_TO_KOR.update({"k": "ㅏ", "o": "ㅐ", "i": "ㅑ", "O": "ㅒ"})
ENG_TO_KOR.update({"j": "ㅓ", "p": "ㅔ", "u": "ㅕ", "P": "ㅖ"})
ENG_TO_KOR.update({"h": "ㅗ", "hk": "ㅘ", "ho": "ㅙ", "hl": "ㅚ"})
ENG_TO_KOR.update({"y": "ㅛ", "n": "ㅜ", "nj": "ㅝ", "np": "ㅞ"})
ENG_TO_KOR.update({"nl": "ㅟ", "b": "ㅠ", "m": "ㅡ", "ml": "ㅢ"})
ENG_TO_KOR.update({"l": "ㅣ"})

# Create a reverse mapping from Korean to English
KOR_TO_ENG = {v: k for k, v in ENG_TO_KOR.items()}

def convert_jamo_to_latin(text: str) -> str:
    """
    Convert Hangul jamo (consonants and vowels) to a romanized form based on the standard Korean keyboard layout.

    Example:
        "사과" -> "tkrhk"
    """
    jamos = decompose_syllables(text=text)
    result = []
    for jamo in jamos:
        if jamo in KOR_TO_ENG:
            result.append(KOR_TO_ENG[jamo])
        else:
            result.append(jamo)
    return "".join(result)

What this does:

• Decompose Korean syllables into jamos.
• Convert each jamo to its QWERTY key equivalent.
• Join the result back into a string.

For example:

After that, update _update_search_keywords function to include these typo-prone Latin inputs in the suggest.input array:

# update_keywords3.py
from Hangul import convert_jamo_to_latin, decompose_syllables, extract_choseongs

...

def _update_search_keywords(es_client: Elasticsearch, keywords: list[Keyword]) -> int:
    """
    Update the autocomplete index with new data.
    Return the number of successfully executed actions (including overrides).
    """
    index = "search-keywords"

    actions = []
    for keyword in keywords:
        action = {
            "_op_type": "index",
            "_index": index,
            "_id": keyword.key,
            "_source": {
                "suggest": {
                    "input": list(
                        set(  # Remove duplicate items if keyword is not Korean
                            [
                                "".join(decompose_syllables(keyword.key)),
                                "".join(extract_choseongs(keyword.key)),
                                "".join(convert_jamo_to_latin(keyword.key)),  # new line
                            ]
                        )
                    ),
                    "weight": keyword.count,
                },
                "keyword": keyword.key,
            },
        }
        actions.append(action)

    res = helpers.bulk(es_client, actions)

    return res[0]

After running the keyword update script, here’s what the document for “사과” might look like:

{
  "_index": "search-keywords",
  "_id": "사과",
  "_score": 1,
  "_source": {
    "suggest": {
      "input": [
        "ㅅㅏㄱㅘ",
        "ㅅㄱ",
        "tkrhk"
      ],
      "weight": 1
    },
    "keyword": "사과"
  }
}

Note that each input value is the result of the following utility functions:

• "".join(decompose_syllables("사과")) -> ㅅㅏㄱㅘ
• "".join(extract_choseongs("사과")) -> ㅅㄱ
• "".join(convert_jamo_to_latin("사과")) -> tkrhk

Even if the user mistakenly types in English mode, your autocomplete will still just work.

Meanwhile, the English keyword airpods 4 stays mostly intact because we removed duplicate items when indexing input values:

{
  "_index": "search-keywords",
  "_id": "airpods 4",
  "_score": 1,
  "_source": {
    "suggest": {
      "input": [
        "airpods4"
      ],
      "weight": 1
    },
    "keyword": "airpods 4"
  }
}