The following is part of a series of posts about 2026 summer intern projects—for more, see “What the interns have wrought, special jumbo 2026 edition”

We centrally store kernel logs from all Linux hosts at JS, and these end up being invaluable for debugging or wider analysis. However, sometimes the Linux subsystems these logs rely on fail, like a disk failure or a kernel panic taking down everything from storage to networking. And often these are the most interesting logs to collect!

In fairness, on enterprise-grade servers, we typically have a lights-out management controller of some sort, so it’s still possible (if inconvenient) to log onto its graphical console to see what’s up. But this isn’t an option on workstations (or servers with famously flaky lights-out management). And it anyway isn’t a great way to grab the structured logs that we normally use.

So in the summer of 2025, Jacob Root spent his first project as a Linux Engineering intern ensuring that we can gather logs off a Linux server even when we don’t have access to the disk or to the network.

Log shipping without touching the disk

It turns out that there’s loads of subtle ways a simple log-shipping program can end up depending on the disk. For instance, you might have to page in executable pages from disk or read a config file under /etc.

Jacob squashed a bunch of these dependencies carefully. To start with, you have to consider the log-shipment binary itself—the executable lives on disk, and Linux loads instruction pages on demand rather than pulling them all into memory at once. So when the binary starts up, it immediately mlockall’s itself into memory.

Then you have to be careful reading in the kernel logs. dmesg lives on disk, so ideally we wouldn’t use it. Instead, our shipper reads from /dev/kmsg directly, which is a special character device exposed by the kernel to give userspace programs access to the kernel-log ring buffer. read()s on this device return entire log lines in a structured format, so we can just easily parse the lines we get back.

Now that we have the logs, we need to send them somewhere. Obviously we want to send them somewhere remote—we don’t trust the local persistent storage! So Jacob built a simple server to accept and persist these logs. However, the catch is service discovery, i.e., we need to figure out the host and port we’re sending to without taking a disk dependency. We commonly use DNS for service discovery at JS, but DNS relies on /etc/resolv.conf and /etc/hosts… To resolve this problem, the log-shipper does DNS resolution in a background loop and caches the discovered IPs, carefully ensuring that the main shipment loop is not blocking on the resolution process. That way, it continues running with old server IPs even if the disk is failing.

Finally, we have to choose a protocol for sending the logs over to the server. Again, you gotta be careful: if you use something like HTTPS, you risk depending on TLS certificates stored on disk. Our log-shipper ended up just using plain UDP and sending one copy of each log line to every collection server. This imposes no backpressure on log sending and allows the shipper to not bother with keeping many TCP connections alive.

What if there’s no network either?

We configure Linux such that when the kernel panics, it kexecs into a crash kernel, which saves the crashdump. However, that relies on a) the disk working, b) there being sufficient memory reserved for the crashkernel, and c) the crashkernel working properly. Jacob looked into ensuring that we save some logs from a kernel panic even when we don’t manage to save a crashdump. The catch is that once the kernel panics, we only have access to a very pared-down context, with no storage or network stack.

It turns out that Linux offers the pstore subsystem, which provides a very bare-bones persistent storage mechanism, often backed by the non-volatile memory that also stores system firmware. The kernel can write to these with simple direct firmware calls, so it doesn’t need to invoke the disk or network stack. There is usually very little storage space available, so the kernel typically compresses the log and then writes it out in reverse order until it runs out of room, so that the most interesting newest logs are most likely to survive.

So the first step here was simply adding the kernel arguments efi_pstore.pstore_disable=0 crash_kexec_post_notifiers=Y. The first tells the kernel that it’s allowed to use the EFI backend for pstore, which effectively widens the reach of this logging to more machines. The second tells the kernel to only kexec into the crashkernel after capturing logs into pstore.

The next step was ensuring that these logs make it off-box when the host next boots up. Again, happily, Linux offers a partial solution—the systemd-pstore service runs early in the boot process, drains the pstore and writes the logs out to disk. So we gave our log-shipper the ability to read these files and ship them off.

We also actually wrote our own parser for pstore because systemd-pstore stopped being able to read the EFI backend (our kernel picked up a change renaming the efi pstore backend to efi_pstore, but our systemd is old enough that it doesn’t recognise the new name). Parsing pstore involved walking it manually to grab the raw fragments, reassembling them into the original logs and then shipping them back off.

Finally, we also implemented some monitoring to confirm that pstore does end up empty, by adding a check in the log-shipper that raises an alert if pstore is not empty. This helps ensure that we don’t accidentally fill up the firmware store and cause the hardware to misbehave.

What do we do with these logs?

We’re a relatively small firm with a variety of specialised hardware, so we care much more about these individual crashes than you might expect; each of these can be disruptive to the business, so we try to debug and root-cause as many as we can.

And so we exposed these logs via a new CLI command, as well as incorporating them into an existing why-reboot tool, such that our sysadmins can now easily figure out why a box crashed and inspect the logs from around that time.

They have proven useful on desktop workstations, which don’t have lights-out management and so historically have been difficult to debug. For example, we had a week where desktops were just freezing randomly, and our pstore logs came in pretty handy. Here’s an example trace (showing just the interesting bits):

PSTR<01><2d8h55m12.005821s>: BUG: kernel NULL pointer dereference, address: 0000000000000030
PSTR<04><2d8h55m12.005826s>: Oops: 0000 [#1] PREEMPT SMP NOPTI
PSTR<04><2d8h55m12.005828s>: CPU: 5 PID: 2169648 Comm: chrome_crashpad
PSTR<04>< 2d8h55m12.00583s>: RIP: 0010:nvidia_vma_access+0x38/0x480 [nvidia]
PSTR<04><2d8h55m12.005934s>: Call Trace:
PSTR<04><2d8h55m12.005938s>:  __access_remote_vm+0x233/0x330
PSTR<04><2d8h55m12.005942s>:  mem_rw+0x148/0x2c0
PSTR<04><2d8h55m12.005944s>:  vfs_read+0xad/0x2f0
PSTR<04><2d8h55m12.005947s>:  ksys_pread64+0x65/0xa0

We’re seeing that Chrome crashed, which causes its crash reporter process (chrome_crashpad) to build a crash report. It walks the memory of the crashed Chrome process, but reaches a region mapped by an Nvidia GPU driver. That triggers the nvidia_vma_access call into the driver to access that memory, but the driver then attempts to reference a null pointer and triggers the oops. Note that the crashkernel doesn’t work on all of our machines because of some restrictions with our system configuration, and so we would have had no logs for this case before Jacob’s pstore project. Jacob’s pstore project also helps us capture kernel panics that coincide with disk or filesystem issues.

Having these logs meant we could be confident that a recent driver update was behind these crashes. We could then reach out to our contacts at Nvidia with the details (while rolling back the driver update). We expect more cases where such logs, which wouldn’t exist without Jacob’s intern project, will come in handy.