/tmp/sortXXXXXX files

A backup script that runs on several Linux systems recently produced the following error:

ls: cannot access '/tmp/sortrq9hq8': No such file or directory

It happens every time the backup runs. Other than the above message, there seems to be no ill effect on the backup, which completes successfully. The ls command did not find a file that it expected to be there.

This article explains how the missing file was created by the Linux sort command as a temporary storage area, how error messages about these files are likely to crop up from time to time, how to reproduce the error, and some background about the behaviour of the sort command on Linux and Solaris.

Reproducing the Error

The error originated from a series of pipelines, as the script was building list of files prior to backup. They can be summarized by the following command, which also serves to reproduce the error at the command line:

# find / -xdev | sort | grep /tmp/sort | xargs ls
ls: cannot access '/tmp/sortht3LhW': No such file or directory

Cause of Error

The Linux sort command uses temporary files to do its job, as needed, placing them under /tmp. Here are some examples:

/tmp/sortt6GsQk
/tmp/sortA67Qe9
/tmp/sortdyHOJN

When the sort is finished, the files are removed.

With our pipeline above, some of the temporary files created by sort are caught and listed by the find command as it traverses /tmp. When the pipeline finishes, those temporary files no longer exist, but the final ls command nonetheless tries to list them, triggering the error.

Note: sort does not always create temporary files. It won’t create them, for example, if the input files are quite small, or if the system has an abundance of free memory. My system has only half a gigabyte of memory, so sort nearly always creates temp files. The issue is more easily reproduced on a weak system than on a powerful one.

Other Linux Distributions

The server above runs Debian 9.4. I have reproduced the error on other Debian 9.4 systems (with up to 1 GB of memory) and on Solaris 11 (2 GB of memory), but not so far on any Red Hat 5 or 6 system, or on Linux Mint 18, which is based on Debian 9.

Incidentally, on those Debian systems, /usr/bin/sort is part of the “coreutils” package, version 8.26-3.

# dpkg -S /usr/bin/sort
coreutils: /usr/bin/sort
# dpkg -l coreutils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                  Version         Architecture    Description
+++-=====================-===============-===============-================================================
ii  coreutils             8.26-3          armel           GNU core utilities

The sort.c source code can be browsed here. It isn’t exactly obvious from the code when temporary files will or will not be created.

Solaris

Regarding Solaris, sort creates temporary files under /var/tmp by default, and the names are prefixed with “stm”. Our command needs some slight adjustment, but the error is still reproduced:

solaris11:~# find / /var/tmp -xdev | sort | grep /var/tmp | xargs ls
/var/tmp/stmAAAziaa4c.00000001: No such file or directory
/var/tmp/stmAAAziaa4c.00000002: No such file or directory
/var/tmp/stmAAAziaa4c.00000003: No such file or directory
/var/tmp/stmAAAziaa4c.00000004: No such file or directory
/var/tmp/stmAAAziaa4c.00000005: No such file or directory
/var/tmp/stmAAAziaa4c.00000006: No such file or directory

Note: /var/tmp was explicitly added to the find list because / and /var/tmp are on different file systems on this server. / was retained because without it, the input list contains too few elements to trigger the error (ie. the whole sort is done in memory, without temporary files).

Creating More Temporary Files

It is possible to create the temporary files for examination using a nonsensical sleep. Here, the files will stay around for 60 seconds:

# find / -xdev | sort | sleep 60 &
[1] 29999
root@sheevaplug:~# ls -l /tmp/sort*
-rw------- 1 root root 2182126 Jun  6 12:14 /tmp/sortDXCnVw
-rw------- 1 root root       0 Jun  6 12:14 /tmp/sortQmaEuz

Or to create more temporary files by increasing the size of the sort list. In the following example, a repeated ls shows more temporary files being created steadily, as the sort progresses.

(NB: the following “find” is an outrageous command that will put a very heavy load on the root file system. I am running it on a small test virtual machine on my PC, backed by SSD. Please don’t run it on anything more important, and certainly not on any live system. If you need to kill it, use kill %1).

# find / / / / -xdev | sort | sleep 60 &
[1] 30042
root@sheevaplug:~# ls -l /tmp/sort*
-rw------- 1 root root 2182126 Jun  6 12:25 /tmp/sortnYfsoE
root@sheevaplug:~# ls -l /tmp/sort*
-rw------- 1 root root 2211569 Jun  6 12:25 /tmp/sortavarmO
-rw------- 1 root root 2182126 Jun  6 12:25 /tmp/sortnYfsoE
root@sheevaplug:~# ls -l /tmp/sort*
-rw------- 1 root root       0 Jun  6 12:25 /tmp/sort0pLFYo
-rw------- 1 root root 2211569 Jun  6 12:25 /tmp/sortavarmO
-rw------- 1 root root 2182126 Jun  6 12:25 /tmp/sortnYfsoE
root@sheevaplug:~# ls -l /tmp/sort*
-rw------- 1 root root 2207716 Jun  6 12:25 /tmp/sort0pLFYo
-rw------- 1 root root 2211569 Jun  6 12:25 /tmp/sortavarmO
-rw------- 1 root root       0 Jun  6 12:25 /tmp/sortlhaq3p
-rw------- 1 root root 2182126 Jun  6 12:25 /tmp/sortnYfsoE
root@sheevaplug:~# ls -l /tmp/sort*
-rw------- 1 root root 2207716 Jun  6 12:25 /tmp/sort0pLFYo
-rw------- 1 root root 2211569 Jun  6 12:25 /tmp/sortavarmO
-rw------- 1 root root       0 Jun  6 12:25 /tmp/sortlhaq3p
-rw------- 1 root root 2182126 Jun  6 12:25 /tmp/sortnYfsoE

Sort is here creating temporary files steadily, as it needs them. Going back to the error at the start of this article, only the first one or two temporary files would trigger it. That is to say, only files created before the original find traverses the /tmp directory. Later temp files would not make it to the final ls (at the end of the pipeline), and ls would not complain about them.

END.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.