Introduction
Autofs/automount is a combination of a user space program and some pieces in the kernel that work together to allow filesystems (many kinds of filesystems, but often NFS) to be mounted “just in time” and then be unmounted when they are no longer in use. As soon as a process wants to access an automounted filesystem, the kernel intercepts the access and passes control to a user space program (typically automount, but systemd now supports some automount functionality as well).
The user space program does whatever is necessary to mount the file system (which usually involves some invocation of mount(8)) and then reports success or failure back to the kernel so the kernel can either allow the process to continue to, or signal a failure.
We use automount for a number of things here at Jane Street. Recently, users
started reporting that directories that shouldn’t exist (i.e. some path on an
automounted filesystem for which the automount daemon has no configuration) were
spontaneously appearing and not going away. Most commonly, users were seeing
dubious output from commands like hg root and hg status, both cases in which
Mercurial calls stat(2) on any path that seems like it might be a “.hg”
directory. The problem was that “.hg” directories kept popping up in places
where they didn’t actually exist, causing these stat(2) calls to succeed and
Mercurial to believe it had found a valid repository directory. Because attempts
to access this ghost “.hg” directory obviously fail, Mercurial provides odd
output for hg root and hg status. We were stumped so I dug into automount to
try to find out where things were going wrong.
Debugging
I’m a big fan of dynamic tracing tools such as DTrace, SystemTap and Ktap for troubleshooting problems like this. In this instance I used a SystemTap script (developed as I went and presented at the end of this blog post) coupled with simply browsing the available source code to better understand the automount daemon’s behavior and ultimately present enough information to the right people to get our problem fixed.
For further info about using Dynamic Tracing to better understand your systems, I highly recommend reading Brendan Gregg’s book, DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD.
Maps
The configuration of the automount daemon is based on maps. The automount daemon
takes as input the location where it can find its master map. That map can be in
a file, LDAP or anywhere else but the automount daemon needs a master map (see
auto.master(5)).
Assuming the master map is a file, each line of the master map consists of several fields. There is the mountpoint for the entry, the type (i.e. one of “file”, “program”, “yp” or several others), the format and then any options that are to be applied to this master map entry. If the master map entry specifies a mount point of “/-” then the map that it references is considered “direct”. Otherwise, it is considered “indirect.”
In addition, both direct and indirect mounts can have “offsets” (see
autofs4-mount-control.txt,
auto.master(5) and autofs(5) for further details).
In the end, from the view point of autofs filesystems, it’s important to remember that with indirect mounts the autofs filesystem is mounted on the mountpoint specified in the master map (or in submaps if you have multiple levels of nesting) and then filesystems are automounted as subdirectories of the autofs filesystem. For mounts that are direct, direct with offsets and indirect with offsets the autofs filesystem is mounted at the point where the automounted filesystem will ultimately be mounted. As such, once that filesystem is mounted, the underlying autofs filesystem has been shadowed.
We use both indirect and indirect with offset mounts here at Jane Street.
User to Kernel Communication
Interfaces to the Kernel
All communication from user space to the kernel is done via ioctl(2).
The latest version of this interface, v5, uses a distinct set of ioctls from prior versions. The newer interface has advantages over the older interfaces, so it always makes sense to use the newer interface when you can.
For example, automount(8) from the autofs package will always try to use the
new interface and only if it runs into some kind of a problem will it fall back
to the older interface (e.g. if access to “/dev/autofs” is denied due to
SELinux).
This document has a good description of the problem and motivation for the newer interface. If you really want to understand automount/autofs, you should read that document.
For the rest of this post, I’ll focus specifically on the newer interface, which all new implementations should be using.
The New Interface
Some of the main points of the newer interface are:
-
All ioctls are issued to the “/dev/autofs” device node. Previously the user space daemon needed to open the directory where the autofs filesystem was mounted and use that file descriptor to issue ioctls.
-
There is a command that allows the daemon to request that the kernel open the autofs filesystem at a given path. The return value of this command is the file descriptor that the daemon should use for all subsequent requests pertaining to that filesystem. This allows you to access an autofs mount point even if that mount point is currently shadowed by an automounted mount. This can happen with direct maps and any maps that use offsets if the automount daemon was restarted while automounted filesystems are still in use.
-
There are 14 different types of requests (or commands) that can be sent from the user space daemon down to the kernel:
enum { /* Get various version info */ AUTOFS_DEV_IOCTL_VERSION_CMD = 0x71, AUTOFS_DEV_IOCTL_PROTOVER_CMD, AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD, /* Open mount ioctl fd */ AUTOFS_DEV_IOCTL_OPENMOUNT_CMD, /* Close mount ioctl fd */ AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD, /* Mount/expire status returns */ AUTOFS_DEV_IOCTL_READY_CMD, AUTOFS_DEV_IOCTL_FAIL_CMD, /* Activate/deactivate autofs mount */ AUTOFS_DEV_IOCTL_SETPIPEFD_CMD, AUTOFS_DEV_IOCTL_CATATONIC_CMD, /* Expiry timeout */ AUTOFS_DEV_IOCTL_TIMEOUT_CMD, /* Get mount last requesting uid and gid */ AUTOFS_DEV_IOCTL_REQUESTER_CMD, /* Check for eligible expire candidates */ AUTOFS_DEV_IOCTL_EXPIRE_CMD, /* Request busy status */ AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD, /* Check if path is a mountpoint */ AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD, }; -
The parameter that is passed with each of the possible ioctl requests is always the following structure (filled in and accessed with different member fields depending on the ioctl request):
struct autofs_dev_ioctl { __u32 ver_major; __u32 ver_minor; __u32 size; /* total size of data passed in, including this struct */ __s32 ioctlfd; /* automount command fd */ /* Command parameters */ union { struct args_protover protover; struct args_protosubver protosubver; struct args_openmount openmount; struct args_ready ready; struct args_fail fail; struct args_setpipefd setpipefd; struct args_timeout timeout; struct args_requester requester; struct args_expire expire; struct args_askumount askumount; struct args_ismountpoint ismountpoint; }; char path[0]; };
The size field encodes the total size of the parameter (i.e. the total size of
the current struct autofs_dev_ioctl being passed). The ioctlfd field encodes
what autofs filesystem this ioctl is meant to operate on and the rest are ioctl
specific arguments.
Each of the structs referenced in the union type provide field names (and data
types) that allow access to the particular arguments that were passed in. I
won’t go through each of them, but as an example, here is the definition of
struct args_requester:
struct args_requester {
__u32 uid;
__u32 gid;
};
This allows the kernel to write code like param->requester.uid to access the
uid field in the parameter to the AUTOFS_DEV_IOCTL_REQUESTER_CMD ioctl call.
The Kernel Ioctl Dispatcher
When the autofs4 kernel module is loaded (yes, it’s the autofs4 kernel module
that supports the version 5 protocol), it creates the /dev/autofs device node
and registers the proper components such that ioctls issued from user space to a
file descriptor associated with /dev/autofs will wind up invoking the kernel
function autofs_dev_ioctl.
That function uses a dispatch table to map the ioctl request it receives to a dedicated function in the kernel:
static ioctl_fn lookup_dev_ioctl(unsigned int cmd)
{
static struct {
int cmd;
ioctl_fn fn;
} _ioctls[] = {
{cmd_idx(AUTOFS_DEV_IOCTL_VERSION_CMD), NULL},
{cmd_idx(AUTOFS_DEV_IOCTL_PROTOVER_CMD), autofs_dev_ioctl_protover},
{cmd_idx(AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD), autofs_dev_ioctl_protosubver},
{cmd_idx(AUTOFS_DEV_IOCTL_OPENMOUNT_CMD), autofs_dev_ioctl_openmount},
{cmd_idx(AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD), autofs_dev_ioctl_closemount},
{cmd_idx(AUTOFS_DEV_IOCTL_READY_CMD), autofs_dev_ioctl_ready},
{cmd_idx(AUTOFS_DEV_IOCTL_FAIL_CMD), autofs_dev_ioctl_fail},
{cmd_idx(AUTOFS_DEV_IOCTL_SETPIPEFD_CMD), autofs_dev_ioctl_setpipefd},
{cmd_idx(AUTOFS_DEV_IOCTL_CATATONIC_CMD), autofs_dev_ioctl_catatonic},
{cmd_idx(AUTOFS_DEV_IOCTL_TIMEOUT_CMD), autofs_dev_ioctl_timeout},
{cmd_idx(AUTOFS_DEV_IOCTL_REQUESTER_CMD), autofs_dev_ioctl_requester},
{cmd_idx(AUTOFS_DEV_IOCTL_EXPIRE_CMD), autofs_dev_ioctl_expire},
{cmd_idx(AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD), autofs_dev_ioctl_askumount},
{cmd_idx(AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD), autofs_dev_ioctl_ismountpoint}
};
unsigned int idx = cmd_idx(cmd);
return (idx >= ARRAY_SIZE(_ioctls)) ? NULL : _ioctls[idx].fn;
}
One point worth noting here is that AUTOFS_DEV_IOCTL_VERSION_CMD doesn’t
actually do very much (i.e. its function pointer is NULL).
There is logic in the kernel to validate that the major and minor fields
passed in the autofs_dev_ioctl with the ioctl request are valid and then
update the major and minor fields with the kernel’s values. However, this
validation happens for all ioctls, so a AUTOFS_DEV_IOCTL_VERSION_CMD really is
a nop.
In practice, its invocation looks like:
1396615913363043073: automount(18060)(0xffff8803d19f8aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
1396615913363055840: automount(18060)(0xffff8803d19f8aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
Note: The large numbers at the beginning of the line are nanosecond timestamps. All of the examples showing tracing of various parts of autofs/automount look similar to the above.
Kernel to User Communication
Establishing A Communication Channel
Kernel to user space communication can happen in a few different ways.
First, the kernel can communicate with user space via the return value from
ioctl(2). This is the least interesting communication channel.
Second, the kernel can (and does) modify the data in the autofs_dev_ioctl
passed to it. Once the ioctl call completes, the daemon can access whatever
output parameters should be available in the autofs_dev_ioctl (which depends
on the specific ioctl that was requested).
Third, notification of missing mounts or the need to umount existing mounts
happens via a regular unix pipe(2). The daemon creates a pipe just before it
mounts a given autofs filesystem and passes the write end of that pipe as the
‘fd’ parameter to the mount command:
1396451770814396287: automount(18388)(0xffff8803d5ae0aa0) -> mount: dev: /etc/auto.foo, dir: /foo, type: autofs, data: fd=7,pgrp=18388,minproto=5,maxproto=5,indirect
In addtion, it passes the process group id of its process to the kernel so when the kernel receives VFS requests for the autofs filesystems, it can determine whether it should pass the requests to the automount daemon or should operate in “Oz mode” and let the process “see the man behind the curtain.”
I didn’t make that up, it’s in the kernel sources and it’s really called
oz_mode.
In the case where you’re mounting a new autofs filesystem, giving the kernel the pipe to use for communication is fairly straightforward. But, what about the case where the daemon has just been restarted and the autofs filesystems that you wish to operate on are actually shadowed by other (say NFS) filesystems?
The AUTOFS_DEV_IOCTL_OPENMOUNT_CMD ioctl was created to address this problem.
This ioctl’s parameter includes a path (as part of struct autofs_dev_ioctl
described earlier). The return from this call will have ioctlfd field set to
-1 for the call, since we’re asking for the mount point to be opened.
Once you have an ioctlfd with which to operate, you can create a pipe and issue
a AUTOFS_DEV_IOCTL_SETPIPEFD_CMD ioctl which has the same effect as passing
the fd argument to mount(8). However, there is the additional restriction
that you cannot issue a AUTOFS_DEV_IOCTL_SETPIPEFD_CMD against an autofs
filesystem without first having set that autofs filesystem to be “catatonic”
with AUTOFS_DEV_IOCTL_CATATONIC_CMD.
See autofs4-mount-control.txt for additional details.
Message Format
Messages from the kernel to the user space daemon via the pipe have the following format:
struct autofs_v5_packet {
struct autofs_packet_hdr hdr;
autofs_wqt_t wait_queue_token;
__u32 dev;
__u64 ino;
__u32 uid;
__u32 gid;
__u32 pid;
__u32 tgid;
__u32 len;
char name[NAME_MAX+1];
};
Note: This structure has some interesting history associated with it. The structure is 300 bytes, but due to alignment issues, on x86_64, the compiler pads it to 304.
This isn’t a problem if both the kernel and the user space daemon are both 32 bit or both 64 bit, but becomes a problem when the daemon is 32 bit, but the kernel is 64 bit. In that case, the daemon is expecting a 300 byte packet from the kernel but will receive a 304 byte packet.
You can read more about this problem in this lwn.net article.
Message Types
There are only 4 types of messages that are sent from the kernel to the user space daemon in the v5 protocol. From the automount source:
static int handle_packet(struct autofs_point *ap)
{
union autofs_v5_packet_union pkt;
if (get_pkt(ap, &pkt))
return -1;
debug(ap->logopt, "type = %d", pkt.hdr.type);
switch (pkt.hdr.type) {
case autofs_ptype_missing_indirect:
return handle_packet_missing_indirect(ap, &pkt.v5_packet);
case autofs_ptype_missing_direct:
return handle_packet_missing_direct(ap, &pkt.v5_packet);
case autofs_ptype_expire_indirect:
return handle_packet_expire_indirect(ap, &pkt.v5_packet);
case autofs_ptype_expire_direct:
return handle_packet_expire_direct(ap, &pkt.v5_packet);
}
error(ap->logopt, "unknown packet type %d", pkt.hdr.type);
return -1;
}
When the user space daemon receives one of these messages, it knows what autofs filesystem it is for because the message arrives on the pipe that the daemon explicitly created for this communication. As a result, the packet itself doesn’t need to identify what filesystem it is associated with.
However, one very important piece of information that comes through the pipe is
the wait_queue_token. This is used by the user space daemon in its ioctl back
to the kernel to notify the kernel whether or not it was able to process the
request (via AUTOFS_DEV_IOCTL_READY_CMD or AUTOFS_DEV_IOCTL_FAIL_CMD) and
uniquely identifies the associated request.
Messages of type autofs_ptype_missing_direct and
autofs_ptype_missing_indirect are sent when a process is trying to access a
directory entry and the kernel needs the userspace daemon to resolve it.
Messages of type autofs_ptype_expire_direct and autofs_ptype_expire_indirect
are sent when the kernel has realized that a mount is no longer active and that
it can be umounted, possibly in response to a AUTOFS_DEV_IOCTL_EXPIRE_CMD
command from the daemon. That last case would mean the daemon makes a
synchronous ioctl call into the kernel, the kernel roots around for a while,
figures out something can be unmounted, then sends a message back to the daemon,
which must be handled concurrently.
For example, here is automount reqeuesting an immediate expiration of /a/b/c,
which causes (1) the kernel to call back into the daemon, (2) the daemon to
perform the umount and signal success (or failure) via another ioctl, and
finally (3) the orginal ioctl to complete.
1396454302262508340: automount(18388)(0xffff8803d3cda040) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 307: how: AUTOFS_EXP_IMMEDIATE (/a/b/c)
1396454302262523253: automount(18388)(0xffff8803d3cda040) -> autofs4_notify_daemon: autofs_ptype_expire_direct: pipefd: 48, proto: 5: {.hdr={.proto_version=5, .type=6}, .wait_queue_token=13052, .dev=1048629, .ino=9285341, .uid=0, .gid=0, .pid=486, .tgid=18388, .len=16, .name="ffff8800488dcbc0"}
1396454302262714273: automount(18388)(0xffff8803d6cd0080) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 57, ioctlfd: -1: type: 0, path: /a/b/c
1396454302262728637: automount(18388)(0xffff8803d6cd0080) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 57, ioctlfd: -1: devid: 38, magic: 0x6969
1396454302289469919: umount.nfs( 489)(0xffff8803d64b4aa0) -> umount: name: /a/b/c
1396454302289942712: automount(18388)(0xffff8803d6cd0080) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 57, ioctlfd: -1: type: 0, path: /a/b/c
1396454302289956871: automount(18388)(0xffff8803d6cd0080) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 57, ioctlfd: -1: devid: 1048629, magic: 0x187
1396454302289984247: automount(18388)(0xffff8803d6cd0080) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_READY_CMD: major: 1, minor 0, size: 24, ioctlfd: 307: token: 13052 (/a/b/c)
1396454302290005575: automount(18388)(0xffff8803d6cd0080) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_READY_CMD: major: 1, minor 0, size: 24, ioctlfd: 307: token: 13052 (/a/b/c)
1396454302290072586: automount(18388)(0xffff8803d3cda040) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 307: how: AUTOFS_EXP_IMMEDIATE ()
Note: In the above output the path in parenthesis is not actually a part of
the ioctl. As part of the SystemTap script used to trace this I am taking the
given ioctlfd and looking it up in the open file descriptors for the automount
process and translating it into a path. This made it easier to match up various
calls that automount daemon was making to better understand its behavior.
The User Space Daemon
At startup the user space daemon will do some initial sanity checking. To do this it mounts a few temporary autofs filesystems and issues ioctls to them to ensure things are as it expects them to be (version numbers match up etc.).
The startup steps look like the following:
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
automount(29803)(0xffff8803d6cd1540) -> mount: dev: automount, dir: /tmp/autoQzUuVP, type: autofs, data: fd=5,pgrp=29803,minproto=3,maxproto=5
automount(29803)(0xffff8803d6cd1540) -> autofs4_fill_super: s=0xffff8803d40b8000 data=0xffff88034ce14000 silent=0x0
automount(29803)(0xffff8803d6cd1540) <- autofs4_fill_super: {.magic=?, .pipefd=?, .pipe=?, .oz_pgrp=?, .catatonic=?, .version=?, .sub_version=?, .min_proto=?, .max_proto=?, .exp_timeout=?, .type=?, .reghost_enabled=?, .needs_reghost=?, .sb=?, .wq_mutex={...}, .pipe_mutex={...}, .fs_lock={...}, .queues=?, .lookup_lock={...}, .active_list={...}, .expiring_list={...}}
automount(29803)(0xffff8803d6cd1540) <- mount
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: -1: devid: 23, path: /tmp/autoQzUuVP
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: 5: devid: 23, path: /tmp/autoQzUuVP
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 5
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 2
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoQzUuVP)
automount(29803)(0xffff8803d6cd1540) umount: name: /tmp/autoQzUuVP
mount(29807)(0xffff8803d24d2080) -> mount: dev: /tmp/autoEXHYpt, dir: /tmp/auto4X1sU6, type: none, data:
mount(29807)(0xffff8803d24d2080) <- mount
umount(29808)(0xffff8803d269e040) -> umount: name: /tmp/auto4X1sU6
mount(29810)(0xffff8803d81aeae0) -> mount: dev: /tmp/autoQgrTpK, dir: /tmp/autoWZ7jVn, type: none, data:
mount(29810)(0xffff8803d81aeae0) <- mount
umount(29811)(0xffff8803d56d4080) -> umount: name: /tmp/autoWZ7jVn
Next, the daemon consults its maps, sets up the necessary autofs filesystems and waits on the pipes it has setup for communication from the kernel.
What do the various Ioctls Do?
A lot of this detail is described in autofs4-mount-control.txt, but I’ve gone through each one to show examples of their actual invocation as observed with SystemTap. If something I’ve written here conflicts with the document linked, I’m probably wrong.
AUTOFS_DEV_IOCTL_VERSION_CMD
As mentioned earlier, this ioctl is basically a no-op. It does some validation
on the major and minor version numbers that you pass in the parameter (All
ioctls on /dev/autofs do this same validation).
Specifically it validates that the major number is equal to
AUTOFS_DEV_IOCTL_VERSION_MAJOR and that the minor number is less than or equal
to AUTOFS_DEV_IOCTL_VERSION_MINOR:
#define AUTOFS_DEV_IOCTL_VERSION_MAJOR 1
#define AUTOFS_DEV_IOCTL_VERSION_MINOR 0
These same major and minor numbers are included in all ioctls from user
space to kernel space.
It’s also worth noting that the ioctlfd in this request is simply set to -1.
1396617872001439657: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
1396617872001453752: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
AUTOFS_DEV_IOCTL_PROTOVER_CMD
This ioctl is issued on a per autofs filesystem basis and therefore requires a
valid ioctlfd argument to identify the autofs file system in question.
The parameter to the ioctl contains a struct args_protover which is initially
- On return from the ioctl, the kernel will have filled out this struct with
the
versionfield of thestruct autofs_sb_infoassociated with this autofs filesystem in the kernel.
In practice, this call is only made once on startup for a temporary test autofs filesystem and is never used again after that.
1396617872001615443: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: -1: devid: 21, path: /tmp/autoV2cNiF
1396617872001623807: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: 5: devid: 21, path: /tmp/autoV2cNiF
1396617872001629141: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5
1396617872001633477: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5
(These prior commands are run to open a file descriptor for the mount point after mounting)
1396617872001636885: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0
1396617872001648323: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 5
AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD
This ioctl is issued on a per autofs filesystem basis and therefore requires a
valid ioctlfd argument to identify the autofs file system in question.
The parameter to the ioctl contains a struct args_protosubver which is
initially 0. On return from the ioctl, the kernel will have filled out this
struct with the sub_version field of struct autofs_sb_info associated with
this autofs filesystem in the kernel.
In practice, this call is only made once on startup for a temporary test autofs filesystem and is never used again after that.
1396617872001651494: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0
1396617872001654267: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 2
AUTOFS_DEV_IOCTL_OPENMOUNT_CMD
This ioctl is issued after an autofs filesystem is mounted in order to obtain
the ioctlfd that can then be used for subsequent ioctls referencing that
autofs file system. It’s purpose is to obtain the ioctlfd and therefore you do
not need to have a valid ioctlfd in order to issue this ioctl.
This actually opens a new file descriptor in the process for the directory which is the mount point of the autofs file system.
The parameter to the ioctl contains a struct args_openmount. In addition, the
parameter contains the actual path of the autofs filesystem that you are trying
to open as a string.
In practice this ioctl is issued once for each autofs filesystem that the user space daemon is managing.
1396879456563815509: automount(10681)(0xffff8803d82ce040) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 31, ioctlfd: -1: devid: 24, path: /a/b/c
1396879456563823041: automount(10681)(0xffff8803d82ce040) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 31, ioctlfd: 28: devid: 24, path: /a/b/c
[cperl@tot-qws-u12114d ~]$ sudo lsof -p $(pgrep -f /etc/auto.master) | grep /a/b/c
automount 10681 root 28r DIR 0,24 0 16335017 /a/b/c
As you can see, the ioctlfd field is -1 on entry to the ioctl and is filled
out by the kernel. Furthermore, we can confirm with lsof that the file
descriptor is now open for the directory.
AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD
This ioctl is issued to signal that the user space daemon no longer needs to issue ioctls for a given autofs filesystem.
For this ioctl, there are no additional details passed in a structure. The only
piece of information needed is the ioctlfd that the daemon is requesting to be
closed.
This actually closes the file descriptor in the process.
In practice, this ioctl is used to close down the temporary autofs filesystem that is used during early initialization to check the various version information. Otherwise it only appears to be used when shutting down automount.
1396617872001657498: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoV2cNiF)
1396617872001663579: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 ()
AUTOFS_DEV_IOCTL_READY_CMD
This ioctl is one of two ioctls issued in response to a request from the kernel. This particular ioctl signals that the request processing was successful and the kernel can continue.
The parameter to the ioctl contains a struct args_ready. This contains the
same token that was received from the kernel on the pipe where the request
originated. This is how the kernel can match a response to its initial request
(along with the fact that the ioctlfd means its for a particular autofs
filesystem).
In practice, this ioctl is used all the time. Every time the user space daemon successfully handles automounting a filesystem, it communicates it to the kernel via this ioctl.
1396617916437019856: hg(30925)(0xffff8803d18c4aa0) -> autofs4_notify_daemon: autofs_ptype_missing_indirect: pipefd: 7, proto: 5: {.hdr={.proto_version=5, .type=3}, .wait_queue_token=52225, .dev=21, .ino=11251159, .uid=12114, .gid=32771, .pid=30925, .tgid=30925, .len=6, .name=".hg"}
...
1396617916440211973: automount(30790)(0xffff8803d11db500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_READY_CMD: major: 1, minor 0, size: 24, ioctlfd: 11: token: 52225 (/foo)
1396617916440230486: automount(30790)(0xffff8803d11db500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_READY_CMD: major: 1, minor 0, size: 24, ioctlfd: 11: token: 52225 (/foo)
In the above output I’ve included the request from hg that caused the request
to be sent to the automount daemon (the kernel function which does that
notification is autofs4_notify_daemon). Here you can see the matching of
wait_queue_token in the request and token in the response.
AUTOFS_DEV_IOCTL_FAIL_CMD
This ioctl is the second of the two ioctls issued in response to a request from the kernel. This command is similar to the ready command described above, but indicates failure.
The parameter to the ioctl contains a struct args_fail which contains both the
token that the daemon received from the kernel as well as a status code.
This status code is what the kernel should return to the process that requested the access. Many times this will be ENOENT, but it can also be other things like ENOMEM or ENAMETOOLONG.
Its worth pointing out that the kernel uses negative numbers to communicate
errors and therefore the status returned is -2 or -ENOENT. For instance, when
the automounter calls get_ioctl_ops()->send_fail, the argument that it passes
is always something like -ENOENT, or -ENOMEM or -ENAMETOOLONG.
1396617921296295641: stat(30998)(0xffff8803d191c040) -> autofs4_notify_daemon: autofs_ptype_missing_indirect: pipefd: 7, proto: 5: {.hdr={.proto_version=5, .type=3}, .wait_queue_token=52228, .dev=21, .ino=11251159, .uid=12114, .gid=32771, .pid=30998, .tgid=30998, .len=3, .name=".hg"}
...
1396617921296848457: automount(30790)(0xffff8803d19f8040) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_FAIL_CMD: major: 1, minor 0, size: 24, ioctlfd: 11: token: 52228, status: -2 (/foo)
1396617921296870360: automount(30790)(0xffff8803d19f8040) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_FAIL_CMD: major: 1, minor 0, size: 24, ioctlfd: 11: token: 52228, status: -2 (/foo)
As you can see here, the status code for the failure is -ENOENT.
AUTOFS_DEV_IOCTL_SETPIPEFD_CMD
This ioctl can be used to set the pipe file descriptor that the kernel will use
to send notifications to the user space daemon. The other option would be to
pass the file descriptor during the actual call to mount(8). This ioctl only
makes sense when you already have a valid ioctlfd to issue it against.
If the automount daemon was restarted and some of its autofs filesystems that its supposed to be managing are shadowed by the real mounts (and therefore still mounted), there is no way to establish this pipe file descriptor. This is the purpose of this ioctl.
1396617872058414208: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_SETPIPEFD_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: pipefd: 13 (/home)
1396617872058417490: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_SETPIPEFD_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: pipefd: 13 (/home)
Additional data from autofs4-mount-control.txt:
The call requires an initialized struct autofs_dev_ioctl with the ioctlfd field set to the descriptor obtained from the open call and the arg1 field set to descriptor of the pipe. On success the call also sets the process group id used to identify the controlling process (eg. the owning
automount(8)daemon) to the process group of the caller.
AUTOFS_DEV_IOCTL_CATATONIC_CMD
This ioctl is issued against a specific autofs filesystem and therefore you must
have a valid ioctlfd to use before it can be issued.
This ioctl asks the kernel to mark the struct autofs_sb_info as no longer
being responsive to mount requests. In addition, it closes the kernel’s side of
the pipe.
If an autofs filesystem is marked as cataonic, then there is no longer any
“magic” going on. Requests are not dispatched to the user space daemon and all
processes are allowed to see the raw filesystem (not just the process id that
was given as the pgrp mount option when the filesystem was mounted).
More details from
autofs4-mount-control.txt
state that this command is a prerequisite for AUTOFS_DEV_IOCTL_SETPIPEFD_CMD:
In order to protect mounts against incorrectly setting the pipe descriptor we also require that the autofs mount be catatonic (see next call).
And a few examples of its use:
1396617872001439657: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
1396617872001453752: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
1396617872001538014: automount(30790)(0xffff8803d5ae0aa0) -> mount: dev: automount, dir: /tmp/autoV2cNiF, type: autofs, data: fd=5,pgrp=30790,minproto=3,maxproto=5
1396617872001570079: automount(30790)(0xffff8803d5ae0aa0) -> autofs4_fill_super: s=0xffff8803d8ba5c00 data=0xffff880210282000 silent=0x0
1396617872001582788: automount(30790)(0xffff8803d5ae0aa0) <- autofs4_fill_super: {.magic=?, .pipefd=?, .pipe=?, .oz_pgrp=?, .catatonic=?, .version=?, .sub_version=?, .min_proto=?, .max_proto=?, .exp_timeout=?, .type=?, .reghost_enabled=?, .needs_reghost=?, .sb=?, .wq_mutex={...}, .pipe_mutex={...}, .fs_lock={...}, .queues=?, .lookup_lock={...}, .active_list={...}, .expiring_list={...}}
1396617872001602201: automount(30790)(0xffff8803d5ae0aa0) <- mount
1396617872001615443: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: -1: devid: 21, path: /tmp/autoV2cNiF
1396617872001623807: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: 5: devid: 21, path: /tmp/autoV2cNiF
1396617872001629141: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoV2cNiF)
1396617872001633477: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoV2cNiF)
1396617872001636885: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0 (/tmp/autoV2cNiF)
1396617872001648323: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 5 (/tmp/autoV2cNiF)
1396617872001651494: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0 (/tmp/autoV2cNiF)
1396617872001654267: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 2 (/tmp/autoV2cNiF)
1396617872001657498: automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoV2cNiF)
1396617872001663579: automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 ()
1396617872001668692: automount(30790)(0xffff8803d5ae0aa0) -> umount: name: /tmp/autoV2cNiF
1396617872005398387: mount(30794)(0xffff8803d82cd500) -> mount: dev: /tmp/auto3HiGR8, dir: /tmp/auto8FUzqC, type: none, data:
1396617872005425776: mount(30794)(0xffff8803d82cd500) <- mount
1396617872006666370: umount(30795)(0xffff8803d67cd500) -> umount: name: /tmp/auto8FUzqC
1396617872009924917: mount(30797)(0xffff8803d18c4040) -> mount: dev: /tmp/autoFHRl05, dir: /tmp/autowC47zz, type: none, data:
1396617872009966875: mount(30797)(0xffff8803d18c4040) <- mount
1396617872011350661: umount(30798)(0xffff8803d269e040) -> umount: name: /tmp/autowC47zz
OR
1396617872058378954: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 30, ioctlfd: -1: type: 1, path: /home
1396617872058386150: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 30, ioctlfd: -1: devid: 22, magic: 0x187
1396617872058391034: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 30, ioctlfd: -1: devid: 22, path: /home
1396617872058396234: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 30, ioctlfd: 17: devid: 22, path: /home
1396617872058407576: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 17 (/home)
1396617872058410942: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 17 (/home)
1396617872058414208: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_SETPIPEFD_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: pipefd: 13 (/home)
1396617872058417490: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_SETPIPEFD_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: pipefd: 13 (/home)
1396617872058420869: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_TIMEOUT_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: timeout: 604800 (/home)
1396617872058426251: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_TIMEOUT_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: timeout: 604800 (/home)
1396617872058436785: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 30, ioctlfd: 17: type: 0, path: /home
1396617872058441090: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 30, ioctlfd: 17: devid: 22, magic: 0x0
1396617872058471734: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 36, ioctlfd: -1: type: 0, path: /home/cperl
1396617872058477028: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 36, ioctlfd: -1: devid: 27, magic: 0x6969
1396617872058480975: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_REQUESTER_CMD: major: 1, minor 0, size: 36, ioctlfd: 17: uid: 0, gid: 0 (/home)
1396617872058484928: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_REQUESTER_CMD: major: 1, minor 0, size: 36, ioctlfd: 17: uid: 0, gid: 0 (/home)
AUTOFS_DEV_IOCTL_TIMEOUT_CMD
This ioctl is issued against a specific autofs filesystem and therefore you must
have a valid ioctlfd to use before it can be issued.
The parameter to the ioctl contains a struct args_timeout. This contains the
timeout for the autofs filesystem in seconds.
Internally the kernel converts this to jiffies and stores it in the
exp_timeout field of the associated struct autofs_sb_info.
1396617872058420869: automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_TIMEOUT_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: timeout: 604800 (/home)
1396617872058426251: automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_TIMEOUT_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: timeout: 604800 (/home)
AUTOFS_DEV_IOCTL_REQUESTER_CMD
This ioctl is issued against a specific autofs filesystem and therefore you must
have a valid ioctlfd to use before it can be issued.
The parameter to the ioctl contains a struct args_requester. This struct is
initially all 0 on the call and output parameters are filled in before returning
from the ioctl.
1396530871075227821: automount(29803)(0xffff8803d8aeb540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_REQUESTER_CMD: major: 1, minor 0, size: 46, ioctlfd: 24: uid: 0, gid: 0 (/home/cperl)
1396530871075232024: automount(29803)(0xffff8803d8aeb540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_REQUESTER_CMD: major: 1, minor 0, size: 46, ioctlfd: 24: uid: 12114, gid: 32771 (/home/cperl)
The rationale for this ioctl as quoted from autofs4-mount-control.txt:
In addition, to be able to reconstruct a mount tree that has busy mounts, the uid and gid of the last user that triggered the mount needs to be available because these can be used as macro substitution variables in autofs maps. They are recorded at mount request time and an operation has been added to retrieve them.
AUTOFS_DEV_IOCTL_EXPIRE_CMD
This ioctl is issued against a specific autofs filesystem and therefore you must
have a valid ioctlfd to use before it can be issued.
The parameter to the ioctl contains a struct args_expire which is a bit flag
in which two flags can be set:
/* Mask for expire behaviour */
#define AUTOFS_EXP_IMMEDIATE 1
#define AUTOFS_EXP_LEAVES 2
In practice, I’ve seen this ioctl called with no flags set, or just
AUTOFS_EXP_IMMEDIATE set. Its possible that it will use AUTOFS_EXP_LEAVES
under some circumstances, but none that I’ve hit yet.
1396470011128585542: automount(24018)(0xffff8803d49f8aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: how: AUTOFS_EXP_IMMEDIATE (/a/b/c)
1396470011128619772: automount(24018)(0xffff8803d49f8aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: how: AUTOFS_EXP_IMMEDIATE (/a/b/c)
OR
1396453375060672511: automount(18388)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: how: (/a/b/c)
1396453375060687029: automount(18388)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: how: (/a/b/c)
Further details from autofs4-mount-control.txt:
Issue an expire request to the kernel for an autofs mount. Typically this ioctl is called until no further expire candidates are found. The call requires an initialized struct autofs_dev_ioctl with the ioctlfd field set to the descriptor obtained from the open call. In addition an immediate expire, independent of the mount timeout, can be requested by setting the arg1 field to 1. If no expire candidates can be found the ioctl returns -1 with errno set to EAGAIN. This call causes the kernel module to check the mount corresponding to the given ioctlfd for mounts that can be expired, issues an expire request back to the daemon and waits for completion.
AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD
This ioctl is issued against a specific autofs filesystem and therefore you must
have a valid ioctlfd to use before it can be issued.
The parameter to the ioctl contains a struct args_askumount. That number is
set to 0 by the user space daemon and is either set 1 to indicate the mount
point may be unmounted, or left at zero to indicate that the mount point is
still busy.
1396453065009385966: automount(18388)(0xffff8803d64b4aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: may_umount: 0 (/foo)
1396453065009390765: automount(18388)(0xffff8803d64b4aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: may_umount: 0 (/foo)
OR
1396454296928731407: automount(18388)(0xffff8803d5ae0040) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: may_umount: 0 (/a/b/c)
1396454296928738658: automount(18388)(0xffff8803d5ae0040) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: may_umount: 1 (/a/b/c)
AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD
This ioctl can be issued with or without an ioctlfd. If anioctlfd is passed,
that is used to find the mount point to operate on, otherwise you must set
ioctlfd to -1 and pass a string in the argument.
The parameter to the ioctl contains a struct args_ismountpoint. This struct is
a union that defines different arguments for the way in and for the way out.
On the way in, it contains a string path to inquire about. On return it fills out the devid and the magic number. Using the magic number, the user space daemon can determine what type of filesystem is mounted at that path.
For example the following are defined in include/linux/magic.h:
#define AUTOFS_SUPER_MAGIC 0x0187
#define NFS_SUPER_MAGIC 0x6969
And you can see examples of the returned values from the calls. The user space daemon uses this information to determine what action it might need to take.
Below are three examples of what it might return (autofs, nfs, or nothing). The nothing case would correspond to a directory that had been created beneath an indirect autofs mount point for ghosting:
1396454169249848027: automount(18388)(0xffff8803d2643500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 44, ioctlfd: -1: type: 0, path: /home/cperl
1396454169249853221: automount(18388)(0xffff8803d2643500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 44, ioctlfd: -1: devid: 200, magic: 0x187
1396454169249858444: automount(18388)(0xffff8803d2643500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 44, ioctlfd: -1: type: 0, path: /foo
1396454169249863856: automount(18388)(0xffff8803d2643500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 44, ioctlfd: -1: devid: 38, magic: 0x6969
1396454169257121837: automount(18388)(0xffff8803d11db500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 47, ioctlfd: -1: type: 0, path: /home/dlobraico
1396454169257128955: automount(18388)(0xffff8803d11db500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 47, ioctlfd: -1: devid: 25, magic: 0x0
Relevant quotes from autofs4-mount-control.txt:
Since we’re re-implementing the control interface, a couple of other problems with the existing interface have been addressed. First, when a mount or expire operation completes a status is returned to the kernel by either a “send ready” or a “send fail” operation. The “send fail” operation of the ioctl interface could only ever send ENOENT so the re-implementation allows user space to send an actual status. Another expensive operation in user space, for those using very large maps, is discovering if a mount is present. Usually this involves scanning /proc/mounts and since it needs to be done quite often it can introduce significant overhead when there are many entries in the mount table. An operation to lookup the mount status of a mount point dentry (covered or not) has also been added.
The call requires an initialized struct autofs_dev_ioctl. There are two possible variations. Both use the path field set to the path of the mount point to check and the size field adjusted appropriately. One uses the ioctlfd field to identify a specific mount point to check while the other variation uses the path and optionally arg1 set to an autofs mount type. The call returns 1 if this is a mount point and sets arg1 to the device number of the mount and field arg2 to the relevant super block magic number (described below) or 0 if it isn’t a mountpoint. In both cases the the device number (as returned by new_encode_dev()) is returned in field arg1.
SystemTap Script
Below is the SystemTap script that came out of debugging/tracing automount(8). The script started off very small and targeted to see what was happening inside the kernel, but continually grew over time.
Each time I answered one question about how automount worked I would uncover several more. This script should not be considered a general purpose tool as it is placing specific user space probes at specific lines in the automount source and therefore would certainly need to be adjusted if used in the future (i.e. line numbers will certainly change).
One particular limitation of SystemTap (or perhaps user space spacing probing, I haven’t looked that hard) that I came across while investigating this is that the SystemTap script that places user space probes must be setup and running before the process that those probes are meant trace.
If you start the process first (or leave the process running), you won’t see any of the user space stuff, which was ultimately quite important to understanding automount(8)’s behavior and finding the bug in its handling of negative cache entries.
/*
* !!! DO NOT USE THIS ON A PRODUCTION MACHINE !!!
*
* Run this with:
* stap -vv --vp 10101 \
* -d /usr/sbin/automount \
* -d /usr/lib64/autofs/lookup_file.so \
* -d /usr/lib64/autofs/parse_sun.so \
* -d /usr/lib64/autofs/mount_nfs.so \
* --ldd
* -D MAXSTRINGLEN=1024 \
* -g \
* ./autofs.stp
*/
/* for selective tracing */
global trace;
/* for constant to string translation */
global _autofs_kernel_notify_types[4]
global _autofs_nsswitch_return_status[5]
probe begin {
/** Autofs kernel to user space notify types **/
_autofs_kernel_notify_types[0x3] = "autofs_ptype_missing_indirect";
_autofs_kernel_notify_types[0x4] = "autofs_ptype_expire_indirect";
_autofs_kernel_notify_types[0x5] = "autofs_ptype_missing_direct";
_autofs_kernel_notify_types[0x6] = "autofs_ptype_expire_direct";
/** Autofs nss return codes from include/nsswitch.h **/
_autofs_nsswitch_return_status[0x0] = "NSS_STATUS_SUCCESS";
_autofs_nsswitch_return_status[0x1] = "NSS_STATUS_NOTFOUND";
_autofs_nsswitch_return_status[0x2] = "NSS_STATUS_UNAVAIL";
_autofs_nsswitch_return_status[0x4] = "NSS_STATUS_TRYAGAIN";
_autofs_nsswitch_return_status[0x5] = "NSS_STATUS_MAX";
}
/* Stolen from /usr/share/doc/systemtap-client-1.8/examples/process/pfiles.stp */
function task_file_handle_d_path:string (task:long, fd:long) %{ /* pure */
struct task_struct *p = (struct task_struct *)((long)STAP_ARG_task);
struct files_struct *files;
char *page = NULL;
struct file *filp;
struct dentry *dentry;
struct vfsmount *vfsmnt;
char *path = NULL;
rcu_read_lock();
if ((files = kread(&p->files)) &&
// We need GFP_ATOMIC since we're inside a lock so we
// can't sleep.
(page = (char *)__get_free_page(GFP_ATOMIC)) &&
(filp = fcheck_files(files, STAP_ARG_fd))) {
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,26)
/* git commit 9d1bc601 */
path = d_path(&filp->f_path, page, PAGE_SIZE);
#else
dentry = kread(&filp->f_dentry);
vfsmnt = kread(&filp->f_vfsmnt);
if (dentry && vfsmnt) {
path = d_path(dentry, vfsmnt, page, PAGE_SIZE);
}
#endif
if (path && !IS_ERR(path)) {
snprintf(STAP_RETVALUE, MAXSTRINGLEN, "%s", path);
}
}
CATCH_DEREF_FAULT();
if (page) free_page((unsigned long)page);
rcu_read_unlock();
%}
function log_common:string () {
return sprintf("%d: %10s(%5d)(0x%x)", gettimeofday_ns(), execname(), pid(), task_current());
}
function autofs_dev_ioctl2str:string (cmd:long, pkt:long, outgoing:long) {
major = user_uint32(pkt);
minor = user_uint32(pkt+4);
size = user_uint32(pkt+8);
ioctlfd = user_int32(pkt+12);
task = task_current();
common = sprintf("major: %d, minor %d, size: %d, ioctlfd: %d", major, minor, size, ioctlfd);
cmd = cmd & 255;
if (cmd == 0x71) {
s = "AUTOFS_DEV_IOCTL_VERSION_CMD";
return sprintf("%s: %s", s, common);
}
else if (cmd == 0x72) {
s = "AUTOFS_DEV_IOCTL_PROTOVER_CMD";
v = user_uint32(pkt+16);
return sprintf("%s: %s: %d", s, common, v);
}
else if (cmd == 0x73) {
s = "AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD";
v = user_uint32(pkt+16);
return sprintf("%s: %s: %d", s, common, v);
}
else if (cmd == 0x74) {
s = "AUTOFS_DEV_IOCTL_OPENMOUNT_CMD";
sz = size - 24;
devid = user_uint32(pkt+16);
path = user_string_n(pkt+24, sz);
return sprintf("%s: %s: devid: %d, path: %s", s, common, devid, path);
}
else if (cmd == 0x75) {
s = "AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD";
path = task_file_handle_d_path(task, ioctlfd);
return sprintf("%s: %s (%s)", s, common, path);
}
else if (cmd == 0x76) {
s = "AUTOFS_DEV_IOCTL_READY_CMD";
token = user_uint32(pkt+16);
path = task_file_handle_d_path(task, ioctlfd);
return sprintf("%s: %s: token: %d (%s)", s, common, token, path);
}
else if (cmd == 0x77) {
s = "AUTOFS_DEV_IOCTL_FAIL_CMD";
token = user_uint32(pkt+16);
status = user_int32(pkt+20);
path = task_file_handle_d_path(task, ioctlfd);
return sprintf("%s: %s: token: %d, status: %d (%s)", s, common, token, status, path);
}
else if (cmd == 0x78) {
s = "AUTOFS_DEV_IOCTL_SETPIPEFD_CMD";
pipefd = user_int32(pkt+16);
path = task_file_handle_d_path(task, ioctlfd);
return sprintf("%s: %s: pipefd: %d (%s)", s, common, pipefd, path);
}
else if (cmd == 0x79) {
s = "AUTOFS_DEV_IOCTL_CATATONIC_CMD";
path = task_file_handle_d_path(task, ioctlfd);
return sprintf("%s: %s (%s)", s, common, path);
}
else if (cmd == 0x7A) {
s = "AUTOFS_DEV_IOCTL_TIMEOUT_CMD";
timeout = user_uint64(pkt+16);
path = task_file_handle_d_path(task, ioctlfd);
return sprintf("%s: %s: timeout: %d (%s)", s, common, timeout, path);
}
else if (cmd == 0x7B) {
s = "AUTOFS_DEV_IOCTL_REQUESTER_CMD";
uid = user_uint32(pkt+16);
gid = user_uint32(pkt+20);
path = task_file_handle_d_path(task, ioctlfd);
return sprintf("%s: %s: uid: %d, gid: %d (%s)", s, common, uid, gid, path);
}
else if (cmd == 0x7C) {
s = "AUTOFS_DEV_IOCTL_EXPIRE_CMD";
how = user_uint32(pkt+16);
path = task_file_handle_d_path(task, ioctlfd);
h = "";
if (how & 1) {
h = "AUTOFS_EXP_IMMEDIATE";
}
if (how & 2) {
t = "AUTOFS_EXP_LEAVES";
h = h == "" ? t : sprintf("%s|%s", h, t);
}
return sprintf("%s: %s: how: %s (%s)", s, common, h, path);
}
else if (cmd == 0x7D) {
s = "AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD";
may_umount = user_uint32(pkt+16);
path = task_file_handle_d_path(task, ioctlfd);
return sprintf("%s: %s: may_umount: %d (%s)", s, common, may_umount, path);
}
/* ISMOUNTPIONT is interpretted different on the way in than out */
else if (cmd == 0x7E) {
s = "AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD";
if (!outgoing) {
sz = size - 24;
type = user_uint32(pkt+16);
path = user_string_n(pkt+24, sz);
return sprintf("%s: %s: type: %d, path: %s", s, common, type, path);
}
else {
devid = user_uint32(pkt+16);
magic = user_uint32(pkt+20);
return sprintf("%s: %s: devid: %d, magic: 0x%X", s, common, devid, magic);
}
}
else {
return "UNKNOWN"
}
}
function autofs_kernel_notify2str:string (typ:long) {
return typ in _autofs_kernel_notify_types ? _autofs_kernel_notify_types[typ] : "UNKNOWN"
}
function nsswitch_status2str:string (typ:long) {
return typ in _autofs_nsswitch_return_status ? _autofs_nsswitch_return_status[typ] : "NSS_STATUS_UNKNOWN"
}
probe module("autofs4").function("autofs_dev_ioctl").call
{
if (execname() == "automount") {
printf("%s -> %s: %s\n",
log_common(),
"autofs_dev_ioctl",
autofs_dev_ioctl2str($command, $u, 0));
}
}
probe module("autofs4").function("autofs_dev_ioctl").return
{
if (execname() == "automount") {
printf("%s %s: dev: %s, dir: %s, type: %s, data: %s\n",
log_common(), "mount", dev, dir, typ, dat);
}
probe kernel.function("sys_mount").return
{
printf("%s %s: name: %s\n", log_common(), "umount", name);
}
probe module("autofs4").function("autofs4_fill_super").call
{
if (execname() == "automount") {
printf("%s -> %s: %s\n", log_common(), probefunc(), $$parms);
}
}
probe module("autofs4").function("autofs4_fill_super").return
{
if (execname() == "automount") {
printf("%s %s: %s: pipefd: %d, proto: %d: %s\n",
log_common(),
probefunc(),
autofs_kernel_notify2str($pkt->v5_pkt->hdr->type),
$sbi->pipefd,
$pkt->v5_pkt->hdr->proto_version,
$pkt->v5_pkt->v5_packet$$);
}
/* user space probes, requires the debuginfo package for the version of autofs
* be installed, and for the backtraces requires debuginfo for glibc */
probe process("/usr/sbin/automount").function("handle_packet_missing_indirect").call {
printf("%s -> %s: AP: %s: PACKET: %s\n", log_common(), probefunc(), $ap$$, $pkt$$);
}
probe process("/usr/sbin/automount").function("handle_packet_missing_indirect").return {
printf("%s %s: %s\n", log_common(), probefunc(), path);
print_ubacktrace();
}
probe process("/usr/sbin/automount").function("mkdir_path").return {
printf("%s %s: %s\n", log_common(), "st_add_task", "ST_READMAP");
print_ubacktrace();
}
}
probe process("/usr/sbin/automount").function("st_add_task").return {
t = task_current();
if (trace[t]) {
delete trace[t];
printf("%s %s: %s\n", log_common(), probefunc(), $$parms$$);
}
probe process("/usr/lib64/autofs/lookup_file.so").function("lookup_mount").return {
printf("%s %s: %s\n", log_common(), "lookup_ghost", $me$$);
}
probe process("/usr/sbin/automount").function("cache_update").call,
process("/usr/sbin/automount").function("cache_add").call {
key = user_string($key);
t = task_current();
trace[t] = 1;
printf("%s -> %s: (KEY: %s): %s\n", log_common(), probefunc(), key, $$parms$$);
print_ubacktrace();
}
probe process("/usr/sbin/automount").function("cache_update").return,
process("/usr/sbin/automount").function("cache_add").return {
t = task_current();
if (trace[t]) {
delete trace[t];
printf("%s %s: %s\n", log_common(), "lookup_prune_cache", $$parms$$);
}
probe process("/usr/sbin/automount").function("lookup_prune_cache").return {
printf("%s %s: %s\n", log_common(), "lookup_prune_one_cache", $me$$);
}
probe process("/usr/sbin/automount").statement("st_readmap@daemon/state.c:587") {
printf("%s -> %s: now: %d\n", log_common(), "st_readmap (starting do_readmap thread)", $ra->now);
}
