When a server runs out of memory, Linux does not stop entirely: the kernel picks a process and kills it. This mechanism is called the OOM killer (OOM stands for out of memory). From the outside, it looks like a database crashing suddenly, a 502 error on a website, or a container restarting for no apparent reason. Below is how to confirm that memory is the cause, find which process was killed and why, and what to do so that it does not happen again.

Tested on Ubuntu 24.04 (systemd 255) and Ubuntu 26.04 (systemd 259); the container examples were run on Docker 29. All kernel and systemd messages in this article are real output from our tests.

Signs that a server is running out of memory

  • A service restarted suddenly, and its log shows no error, only code=killed, status=9/KILL.
  • A Docker container stopped with exit code 137.
  • The PostgreSQL log shows client backend (PID 83) was terminated by signal 9: Killed, after which the database drops all connections and restarts.
  • A command in the terminal stopped with the word Killed.
  • The server responds slowly, SSH login takes tens of seconds, and the disk is constantly busy.

Signal 9 (SIGKILL) and exit code 137 (128 + 9) do not prove a memory shortage by themselves: a process killed with kill -9 ends the same way. The proof is an entry in the kernel log, covered below.

How to read free -h output

free -h

Output on our test machine (4 GB of RAM, 1 GB of swap) at a moment when memory was short:

               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       2.7Gi        63Mi        87Mi       1.3Gi       1.1Gi
Swap:          1.0Gi       1.0Gi        32Ki
  • total is all memory; used is what programs occupy.
  • free is memory not used at all. A small value here is normal: Linux uses free memory for the file cache.
  • buff/cache is the file cache and kernel buffers. The kernel gives most of the cache back to programs on demand.
  • available is the main column: how much memory programs can get, counting the cache that can be freed.
  • Swap is a disk area where the kernel moves rarely used memory.

By free, 63 MB looks alarming, but available is 1.1 GB, so there is memory right now. The warning sign in this output is a different one: swap is completely full, meaning memory ran short recently and the kernel moved everything it could to disk.

If available regularly drops below 10% of total during peak hours, the server is short on memory.

vmstat: the si and so columns show swap activity

free shows a single moment, while vmstat shows what happens every second:

vmstat 1 6

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 0  0 1048400 161836   4260 1270584    6    8 13674   919 4579   10  5  4 91  0  0  0
 1  0 1048372  68376   4680 1318176   68   60 99328    60 6749 7141 40  9 50  1  0  0
 1  0 1048352 140808   4040 1287604   20    0 45132   240 3529 2531 16  6 78  1  0  0
 1  0 1048372  65872   3572 1327492   48  220 157552  7804 9445 9595 19 15 56 10  0  0
 2  0 1048364  95968   4608 1319124    4    0 14680    60 3748 2234 16 11 69  4  0  0
 7  0 1048352  69220   3312 1232392   12    0 129016     0 8178 9361 20 14 64  2  0  0

The first line is the average since boot; look at the lines after it.

  • si / so are how many kilobytes per second the kernel reads from swap and writes to it. If both columns stay nonzero, the memory of running programs no longer fits in RAM, and the server spends time moving pages between memory and disk.
  • bi is disk reads. Large values with a small cache mean the kernel lacks memory even for the file cache and keeps rereading files from disk.
  • wa is the share of time the CPU waits for the disk.
  • b is the number of processes waiting for I/O.

In the output above, so reaches 220 KB/s and bi 157 MB/s: the machine is at its limit.

Another indicator is memory pressure (PSI), the share of time during which processes were waiting for memory:

cat /proc/pressure/memory

some avg10=1.27 avg60=5.41 avg300=21.02 total=5374333492
full avg10=1.22 avg60=4.57 avg300=15.49 total=3809828370

some avg300=21.02 means that over the last 5 minutes, at least one process was waiting for memory 21% of the time. On a healthy server, these values are zero or a fraction of a percent.

Where to find the OOM killer: dmesg and journalctl -k

The kernel records every OOM killer event in its log. On Ubuntu, reading the kernel log requires sudo:

sudo journalctl -k -g "out of memory" --since "2 days ago"
sudo dmesg -T | grep -i -E "out of memory|killed process"

Write the journalctl -g pattern in lowercase: then the search is case-insensitive and finds both forms of the message, Out of memory and Memory cgroup out of memory. With a capital letter, the second form is not found.

You will find two kinds of lines.

The whole server ran out of memory. This is how the kernel killed MySQL on our test machine when other processes took all the memory:

containerd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-500
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/83e3731abda6…,task=mysqld,pid=56630,uid=999
Out of memory: Killed process 56630 (mysqld) total-vm:2177364kB, anon-rss:332220kB, file-rss:1184kB, shmem-rss:0kB, UID:999 pgtables:1264kB oom_score_adj:0

A process hit the limit of its service or container (how to set one is covered below):

python invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
memory: usage 102400kB, limit 102400kB, failcnt 108
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=b3ef78fd1125…,mems_allowed=0,oom_memcg=/docker/b3ef78fd1125…,task_memcg=/docker/b3ef78fd1125…,task=python,pid=42515,uid=0
Memory cgroup out of memory: Killed process 42515 (python) total-vm:116168kB, anon-rss:101572kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:264kB oom_score_adj:0

How to read them:

  • X invoked oom-killer is the process that could not get memory at that moment. It is not necessarily the culprit: in the first example, containerd requested memory, but mysqld was killed.
  • constraint=CONSTRAINT_NONE and global_oom mean the whole server ran out of memory. CONSTRAINT_MEMCG means a process group (a container or a systemd service) hit its limit, and the rest of the server was not affected.
  • Killed process 56630 (mysqld) is the process that was killed.
  • anon-rss is how much memory the process used when it was killed: 332 MB for MySQL.
  • Before these lines, the kernel prints a table of all processes with their memory (Tasks state), which shows who else was using memory.

systemd service

If the kernel killed a service's process, systemd records it in the service log:

systemctl status myapp
journalctl -u myapp -o cat

Active: failed (Result: oom-kill) since Sun 2026-10-11 11:19:28 UTC; 6s ago

myapp.service: A process of this unit has been killed by the OOM killer.
myapp.service: Main process exited, code=killed, status=9/KILL
myapp.service: Failed with result 'oom-kill'.

This is what systemd 255 on Ubuntu 24.04 writes. On Ubuntu 26.04 (systemd 259), the first line reads differently: The kernel OOM killer killed some processes in this unit. Search for oom-kill: that string is the same in both versions.

Docker container

A container whose process was killed for lack of memory stops with exit code 137, and Docker records this in its state:

docker ps -a                                                  # Exited (137)
docker inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}}' myapp   # true 137
docker events --filter event=oom                              # watch for new cases

OOMKilled becomes true both when the container hits its own limit and when the whole server runs out of memory, as with MySQL in the example above.

How the OOM killer picks a process: oom_score and oom_score_adj

The kernel computes a score for each process, mostly based on how much memory it uses, and kills the process with the highest score. That is why databases and large applications are hit first.

The score and its adjustment are in /proc:

cat /proc/1234/oom_score       # current score; the higher it is, the more likely the process is killed
cat /proc/1234/oom_score_adj   # adjustment from -1000 to 1000, 0 by default

The processes with the highest scores:

for p in /proc/[0-9]*; do printf "%s %s %s\n" "$(cat $p/oom_score)" "${p#/proc/}" "$(cat $p/comm)"; done 2>/dev/null | sort -rn | head

The adjustment is changed to protect an important process or, conversely, to make a less important one the target. -1000 prevents the process from being killed at all. To change the adjustment of a running process, use choom:

sudo choom -p 1494 -n 300

pid 1494's OOM score adjust value changed from -500 to 300

To keep the adjustment across restarts, set it in the service settings (OOMScoreAdjust=-500, see the next section) or for the container (docker run --oom-score-adj=-500).

Do not protect everything from the OOM killer: if a process with -1000 takes all the memory itself, the kernel will kill everything else, including SSH.

systemd-oomd on Ubuntu Server

systemd-oomd is a service that kills processes before the kernel does when memory pressure (PSI) stays high. On Ubuntu, it is part of the desktop edition: the systemd-oomd package is installed as a dependency of ubuntu-desktop and ubuntu-desktop-minimal.

Ubuntu Server 24.04 and 26.04 do not have it: the package is not in the cloud image package list, and the ubuntu-server package does not depend on it. To check on your server:

systemctl status systemd-oomd

Unit systemd-oomd.service could not be found.

This means only the kernel's OOM killer runs on the server, and you should look for its traces in the kernel log, as described above.

How to limit a service's memory: systemd MemoryMax

A limit does not add memory, but it changes who suffers: instead of a random large process (often the database), the kernel kills the service that went over its limit. Everything else keeps running.

The limit is set in a drop-in configuration file, so you do not edit the main unit file. For the myapp service:

sudo mkdir -p /etc/systemd/system/myapp.service.d
sudo tee /etc/systemd/system/myapp.service.d/memory.conf > /dev/null <<'EOF'
[Service]
MemoryMax=300M
OOMScoreAdjust=300
Restart=on-failure
EOF
sudo systemctl daemon-reload
sudo systemctl restart myapp
  • MemoryMax is a hard limit: if it is exceeded, the kernel kills the service's process.
  • OOMScoreAdjust is the score adjustment for this service.
  • Restart=on-failure makes systemd start the service again after it is killed.

If the server has swap, a service over MemoryMax can use swap as well. To prevent that, add the line MemorySwapMax=0.

Check: systemctl status myapp shows a line with the limit, and /proc/<PID>/oom_score_adj shows the adjustment:

Memory: 90.6M (max: 300.0M available: 209.3M peak: 90.6M)

What happens when the service exceeds the limit (journalctl -u myapp -o cat, Ubuntu 24.04):

Started myapp.service - Test app.
myapp.service: A process of this unit has been killed by the OOM killer.
myapp.service: Main process exited, code=killed, status=9/KILL
myapp.service: Failed with result 'oom-kill'.
myapp.service: Scheduled restart job, restart counter is at 1.
Started myapp.service - Test app.

There is also a soft threshold, MemoryHigh. When it is exceeded, the kernel does not kill processes but throttles them and reclaims the service's cache. In our test, a service with MemoryHigh=40M and no swap hit this threshold and nearly stopped working, but was not killed. Use MemoryHigh together with MemoryMax, slightly below it, as a warning signal rather than the only limit.

How to limit a Docker container's memory

docker run -d --name myapp --memory=300m --memory-swap=300m myimage

If you set only --memory, the container can use the same amount again in swap: by default, --memory-swap is twice --memory. Equal values deny the container any swap.

To see what this looks like, test a container with a 100 MB limit in which Python allocates memory in 10 MB chunks:

docker run --name oomtest --memory=100m --memory-swap=100m python:3-slim python -c '
import time
chunks = []
for i in range(1, 1000):
    chunks.append(bytearray(10 * 1024 * 1024))
    print(f"allocated {i * 10} MB", flush=True)
    time.sleep(0.1)
'
echo "exit code: $?"
docker inspect oomtest --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}'
docker rm oomtest

allocated 80 MB
allocated 90 MB
exit code: 137
OOMKilled=true ExitCode=137

The kernel log then shows the Memory cgroup out of memory lines we went through above.

The same in Docker Compose:

services:
  app:
    image: myimage
    deploy:
      resources:
        limits:
          memory: 300M
    memswap_limit: 300M
    restart: unless-stopped

Check: docker inspect -f '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}' <container> shows 314572800 314572800, and docker stats shows 300MiB in the LIMIT column.

Swap as a buffer

A swap file does not replace RAM, but it gives the server headroom for peaks: during a short shortage, the kernel moves rarely used pages to disk instead of killing a process right away. This is especially useful on servers with 1–2 GB of RAM, for example during a project build or a system upgrade. How to create a swap file is covered in the swap file section of the initial server setup article.

If, however, si and so in vmstat stay nonzero and swap is full, swap is no longer a buffer: the server is slow because it constantly goes to disk. That is a sign it needs more memory.

When to add memory

Add memory if at least one of these holds:

  • available in free -h regularly drops below 10% of total during peak hours;
  • the OOM killer fires under normal load, not because of a memory leak in a single process;
  • si and so in vmstat stay nonzero and swap is full;
  • some avg300 in /proc/pressure/memory stays above a few percent.

If, on the other hand, one process eats memory time after time and keeps growing, that is a memory leak. Adding memory only postpones the problem: limit the process and look for the cause in the application.

How much memory typical workloads need is calculated in How much RAM does a VPS need. At Frogus Cloud, you can add memory to a running server; you cannot reduce it later, so add memory based on measurements, not in advance.

Troubleshooting

What you see What to do
A service restarts, but the kernel log has no out of memory Memory is not the cause: check journalctl -u <service> just before it stopped
sudo dmesg is empty or does not show old entries The kernel buffer is limited in size and cleared on reboot; search journalctl -k -b -1, the log of the previous boot
journalctl -k -g "Out of memory" does not find Memory cgroup out of memory Write the pattern in lowercase: -g "out of memory"
The database is killed even though the application uses the memory Limit the application with MemoryMax or --memory, and give the database a negative OOMScoreAdjust
A service with MemoryHigh has "frozen" It hit the soft threshold: raise MemoryHigh or set MemoryMax
A container exits with code 137, but OOMKilled=false The process was killed with signal 9 manually or by the stop timeout (docker stop), not because of memory

What's next