Skip to content

Linux Admin Interview — Scenario & Troubleshooting

Use for: live-incident style questions where they want to hear your diagnostic method, not a one-line fact.

Search keywords: disk full df shows space inodes 100% deleted file open service will not start systemctl status journalctl exit code 203 slow ssh diagnose load average iostat cron job fails environment path cannot write file own immutable read-only filesystem find process listening port ss lsof

These mirror real incidents, not trivia — the best way to prepare is to have actually fixed one like it.

Q26. A server reports "disk full" but df -h shows free space. What is going on?

Two usual suspects, and a good candidate names both.

First, inodes are exhausted even though blocks are free, which happens with millions of tiny files. Check the other dimension:

$ df -i /
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
overlay        30574800 616702 29958098    3% /

If IUse% is at 100, you are out of inodes, not space, and the fix is deleting or consolidating small files.

Second suspect: a deleted file still held open by a process (same mechanism as "what happens when you delete an open file" — see the Advanced topic page), where the space will not return until that process is restarted. You find those with lsof | grep deleted.

Naming both causes, and knowing df -i exists, is the senior answer here.

Q27. A service will not start after a config change. Walk me through debugging it.

Check status first, then the logs, then the config. The status line usually tells you most of it:

$ systemctl status badapp.service
x badapp.service - App with a bad config path
     Active: failed (Result: exit-code) since Wed 2026-06-17 06:44:06 UTC; 10ms ago
    Process: 74 ExecStart=/usr/local/bin/myapp --config /etc/myapp/missing.conf (code=exited, status=203/EXEC)

That status=203/EXEC is a specific, useful clue: systemd could not execute the binary (wrong path, or not executable). Then go to the journal for the why:

$ journalctl -xeu badapp.service
... badapp.service: Main process exited, code=exited, status=203/EXEC
... badapp.service: Failed with result 'exit-code'.

From there: confirm the binary path and its execute bit, validate the new config (many daemons have a -t test mode, like nginx -t), check file permissions on the config, and run journalctl for the actual application error.

The method — status, then journal, then config — is what they want to hear, not a lucky guess.

Q28. You SSH into a box and it is painfully slow. How do you diagnose it?

Narrate a triage order, because that is the whole point of the question.

  1. Start with uptime to read the load average and decide if the box is overloaded at all.
  2. Split the problem: top/htop to see if a process is pinning CPU; free -h to check whether memory is exhausted and the box is swapping.
  3. Because high load with idle CPU points at I/O, check disk with iostat or look for processes in the D state.
  4. Glance at dmesg for hardware errors or OOM kills.
  5. If even logging in was slow, suspect DNS (a reverse-lookup timeout on the SSH server).

The structure matters more than any single tool: is it CPU, memory, I/O, or network, and what is the evidence.

Q29. A cron job runs fine when you run it by hand but fails from cron. Why?

Almost always the environment. Cron runs with a minimal environment and a bare PATH (often just /usr/bin:/bin), not your login shell's rich one, so a command you call by its short name is not found, or a variable you rely on is unset.

The fixes: use absolute paths to every binary, set the variables you need explicitly inside the script, and redirect output (>> /tmp/job.log 2>&1) so you can actually see the error, because cron emails output into a void on most boxes.

This question is really testing whether you understand that "it works in my shell" and "it works in cron" are different environments.

Q30. You own a file but cannot write to it. Give me the possible reasons.

Run through the layers out loud:

  • The file's own permissions may not include write for you.
  • The directory may lack write and execute permission, which you need to modify entries in it.
  • The file may carry the immutable attribute (chattr +i), which blocks writes even for root until you clear it with chattr -i; you check with lsattr.
  • The filesystem may be mounted read-only, often because it flipped to read-only after detecting errors.
  • The disk or inodes are full (see Q26).

A candidate who lists permissions, then the immutable bit, then a read-only mount is showing real field experience, because each of those has bitten people in production.

Q31. How do you find which process is listening on a given port?

ss is the modern tool (netstat is the older one you will still see):

$ ss -ltnp | grep 8080
LISTEN 0  5  0.0.0.0:8080  0.0.0.0:*  users:(("python3",pid=2701,fd=3))

lsof answers the same question from the file-descriptor side:

$ lsof -nP -i :8080
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python3 2701 root    3u  IPv4  13574      0t0  TCP *:8080 (LISTEN)

Both give you the PID, which is what you actually wanted: the thing holding the port so you can decide whether to stop it or reconfigure it. Reaching for ss over netstat is a small signal that you have kept current.