Just because a file system is lazily unmounted (or so claims the mount command and /proc/mounts), doesn’t mean it is no longer in use. Here’s how find all the file handle holders on the now mostly-invisible file system: We recently migrated a heavily-used read-only network filesystem from one NAS to another. On machines with references to the filesystem, attempting to unmount the old filesystem gives the familiar error:

umount: /share: device is busy

To make the transition seamless, we lazily unmounted the old filesystem and mounted the new filesystem in its place. This way, new processes will use the new filesystem and old references will go away as processes die.

However, doing a lazy unmount complicates finding the references to the old filesystem. Turning off the old NAS could cause processes still referencing it to hang unpredictably at some point in the future.

Finding references to lazily unmounted filesystems is complicated because the full path no longer appears in, say, lsof. We noticed the path was modified in a peculiar way: the filesystem prefix was truncated. While we might normally see this:

$ sudo lsof -nn
COMMAND     PID        USER   FD      TYPE             DEVICE      SIZE                 NODE NAME
init          1        root  cwd       DIR              253,0      4096                    2 /
init          1        root  rtd       DIR              253,0      4096                    2 /
[...]
daemonxyz 18473        root  txt       REG               0,22  13820130               266929 /share/bin/daemon_xyz (nas:/share)

We instead saw:

daemonxyz 18473        root  txt       REG               0,22  13820130               266929 bin/daemon_xyz

lsof doesn’t appear to list non-absolute paths otherwise, so we use that to find references. Unfortuntately, lsof’s output is irregular (columns can be blank, and don’t have a fixed number of columns between different runs). Worse still, lsof doesn’t list other filesystem dependencies like memory maps. Files can serve as the backing-store of a memory mapping, which is how Linux implements on-demand paging of program text and shared libraries. So, our script ended up looking at /proc/*/maps, removing known false positives:

!/bin/bash

cat /proc/*/maps 
  | awk '{print $6}'
  | grep -v '^/'         # remove absolute paths
  | grep -v '^$' 
  | grep -v '(deleted)' 
  | grep -v '^.vdso.$' 
  | grep -v '^.heap.$' 
  | grep -v '^.stack.$' 
  | grep -v '^.vsyscall.$' 
  | grep -v '^socket:$'

The maps file presents the memory mappings which belong to each process, indicating the kind of mapping and, if it’s a file, its path. Like lsof, the absolute path is unavailable if the filesystem hosting the file was lazy unmounted. This left us reasonably confident that we’d gotten all old references, though we’d still like a better way.