Linux Admin Interview — Advanced

Use for: deeper Linux internals — OOM kills, special permission bits, deleted-but-open files, load average, resource limits, name resolution, text-processing tools, safe bash scripting.

Search keywords: OOM killed exit code 137 SUID SGID sticky bit deleted file open inode load average uptime ulimit soft hard limit nsswitch resolv.conf DNS grep sed awk bash set -euo pipefail

Q18. What does it mean when a process is "OOM-killed", and how would you spot it?

When the system runs out of memory, the kernel's OOM killer picks a process and sends it SIGKILL to reclaim memory. You spot it two ways. First, the exit code: a process killed by a signal exits with 128 + signal number, and since SIGKILL is 9, an OOM-killed process shows exit code 137. Second, the kernel log: dmesg or the journal will contain an Out of memory: Killed process line naming the victim.

This is a favorite because exit 137 shows up constantly in container land, and knowing it means "the kernel killed me for using too much memory" saves hours.

Q19. Explain SUID, SGID, and the sticky bit.

These are the special permission bits beyond rwx. SUID on an executable makes it run with the file owner's privileges rather than the caller's, which is how passwd lets an ordinary user update /etc/shadow. SGID does the same with the group, and on a directory it makes new files inherit that directory's group. The sticky bit on a directory (you have seen it on /tmp) means only a file's owner can delete it, even if others can write to the directory. In a long listing they replace the execute bit: an s where x would be for SUID/SGID, a t for the sticky bit.

What they're really testing: SUID is a real privilege-escalation surface, so an answer that mentions the security angle stands out.

Q20. What happens if you delete a file that a running process still has open?

The file's directory entry disappears immediately, but the data is not freed until the last open file descriptor is closed. Linux removes the name, while the inode and its blocks stay alive as long as a process holds the file open.

$ exec 3>/tmp/openfile.log
$ rm /tmp/openfile.log
$ ls -l /proc/$$/fd/3
l-wx------ 1 root root 64 ... /proc/1/fd/3 -> /tmp/openfile.log (deleted)

The (deleted) marker is the whole answer. This is exactly why deleting a huge log file does not free disk space if a process still has it open: you have to restart or signal the process (or truncate the file with > file instead of deleting it). It connects straight to the "disk full but df shows space" troubleshooting question.

Q21. What does the load average actually measure?

The three numbers from uptime are the 1, 5, and 15 minute averages of the number of processes that are either running or waiting to run, plus, on Linux specifically, processes in uninterruptible sleep (the D state, usually blocked on I/O).

$ uptime
 06:42:24 up 1 min,  0 user,  load average: 1.00, 0.40, 0.15

The senior nuance: load is not CPU percentage. On a 4-core box a load of 4.0 means fully busy, while the same 4.0 on a single core means heavily overloaded. And because Linux counts I/O waiters, a high load with idle CPUs points at disk or network blocking, not compute. Comparing the 1-minute against the 15-minute number tells you whether the spike is rising or fading.

Q22. What is ulimit, and what is the difference between a soft and a hard limit?

ulimit controls per-process resource caps, like the maximum number of open file descriptors.

$ ulimit -Sn   # soft limit
1048576
$ ulimit -Hn   # hard limit
1048576

The soft limit is the value actually enforced and can be raised by the user up to the hard limit. The hard limit is the ceiling: any process can lower its own hard limit, but raising it needs privilege (root, or CAP_SYS_RESOURCE). This shows up in the wild as the dreaded "too many open files" error: the fix is to raise the soft limit (often in /etc/security/limits.conf or a systemd unit's LimitNOFILE), and knowing the soft-versus-hard distinction is what lets you raise it correctly.

Q23. How does hostname resolution work on Linux?

When something needs to resolve a name, the order is governed by /etc/nsswitch.conf, whose hosts: line typically says files dns. That means the system checks /etc/hosts first and only then asks DNS, using the servers listed in /etc/resolv.conf. The reason this matters in an interview: a box that resolves a name "wrong" is often a stale /etc/hosts entry winning before DNS, and the person who checks nsswitch.conf and /etc/hosts before blaming DNS looks like someone who has debugged it before.

Q24. What is the difference between grep, sed, and awk?

grep finds lines that match a pattern. sed is a stream editor for transforming text, most often substitution. awk is a small language for column-oriented data.

# grep: show only error responses
$ grep -E " (401|403)$" access.log
192.168.1.11 POST /login 401

# awk: count requests per client IP
$ awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn
2 192.168.1.11
2 192.168.1.10
1 192.168.1.42

# sed: redact the last octet of each IP
$ sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1.XXX/' access.log | head -1
192.168.1.XXX GET /index.html 200

The one-line summary that lands: grep to find, sed to change, awk to work with fields.

Q25. How do you write a bash script that fails safely?

Start it with set -euo pipefail. -e exits on the first command that fails, -u treats an unset variable as an error instead of an empty string, and -o pipefail makes a pipeline fail if any stage fails, not just the last one. Then quote your variables ("$var"), because an unquoted variable that is empty or contains spaces is how scripts delete the wrong files.

The difference between a junior and a senior script is not cleverness, it is that the senior one stops at the first sign of trouble instead of charging ahead with a bad value.