Linux disk partition names such as /dev/sda1, /dev/sda2 are not as fixed as they once were. From time to time they can change. Perhaps due to a hardware change or kernel upgrade, or sometimes for no apparent reason. If and when this happens on your system, things can break. In this case, an encrypted swap partition had been configured as “/dev/sda6”, and failed to activate following a change in the /dev/sdXX partition names. This article describes the symptoms and a fix.

The article also discusses a bug affecting Ubuntu 14.04 based distributions, which can make it more difficult to recover an encrypted swap configuration broken by a device name change.

Swap not There

The system of interest was running Linux Mint 17.3, which is based on Ubuntu 14.04. Today I noticed that it had no configured swap space.

# swapon -s
Filename				Type		Size	Used	Priority

There should be an 8 GB swap partition there. Trying to activate it gave:

# swapon -a
swapon: /dev/mapper/cryptswap1: stat failed: No such file or directory

The swapon command is reading /etc/fstab to get the location of the swap device, described by this entry:

# grep swap /etc/fstab
/dev/mapper/cryptswap1 none swap sw 0 0

cryptswap1 is an encrypted block device representing a physical disk partition. But as the error message says, the expected device file simply isn’t there. Checking with the ls command:

# ls /dev/mapper/cryptswap1
ls: cannot access /dev/mapper/cryptswap1: No such file or directory
# ls /dev/mapper
control

Encrypted Block Devices

On this system, swap is an encrypted disk partition, originally created with the ecryptfs-setup-swap command. The missing mapper device cryptswap1 should have been be created when the system booted. Something has gone wrong. The encryption subsystem reads the file /etc/crypttab to see what encrypted devices need to be mounted. Checking that file:

# cat /etc/crypttab
#                  
cryptswap1 /dev/sda6  /dev/urandom swap,cipher=aes-cbc-essiv:sha256

And there is the problem. /dev/sda6 no longer points to a valid swap area, because all of the /dev/sdaX devices have changed.

Attempting to mount the encrypted volume now confirms the error:

# /etc/init.d/cryptdisks start
 * Starting remaining crypto disks...
 * cryptswap1 (starting)..
 * cryptswap1: the precheck for '/dev/sda6' failed:  - The device /dev/sda6 contains a filesystem type ext4.
 * cryptswap1 (failed)...

The cryptdisks command realizes that /dev/sda6 is not a swap partition and does not commence swapping. Just as well, because /dev/sda6, although it used to refer to the swap partition, now refers to my root partition (/).

# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda6             10190136   7035288   2614176  73% /

Swap Configuration

I checked around the system and ascertained that the 8GB swap partition is now called /dev/sda8, not /dev/sda6. This problem could be fixed easily, by just replacing “/dev/sda6” in the /etc/crypttab file with “/dev/sda8”. However, that might lead to the same problem happening again, if and when the kernel decides to rename all the /dev/sdaX devices again. Better to use a more persistent name.

File System UUID

Block devices are often referred to by their UUIDs. You will often see this kind of thing on a modern Linux system:

# grep home /etc/fstab
UUID=0c986925-1948-4507-8432-6bff7e31b305 /home	          ext4    errors=remount-ro 0       1

The file system UUID of /home is there on the left. It has been used instead of a device name /dev/sdXX. The UUID is persistent. It will not change even if the underlying /dev/sdXX device name changes. It is therefore a good idea to use the UUID in configuration files. See Archwiki for a good summary of persistent device names.

Swap Area UUID

Swap devices, although they are not “file systems”, can also have file system UUIDs. Editing /etc/crypttab and inserting a UUID for the swap area, instead of /dev/sda6, should fix the current problem. All we need is a valid UUID. The blkid command will show all file system UUIDs on the system. In this case, the swap partition did not have a UUID. No problem, just create one. While we are at it, let’s create a friendly label too:

# mkswap -L swap /dev/sda8
Setting up swapspace version 1, size = 8388604 KiB
LABEL=swap, UUID=6c7b8efe-87bc-4f7a-806f-a1d3eac91169

Now the label “swap” can be used in in /etc/crypttab. It should survive any more device renaming by the kernel, and it is quite readable.

Reconfiguring Encrypted Swap

I edited /etc/crypttab and changed the contents to:

#               
cryptswap1 LABEL=swap  /dev/urandom swap,offset=8,cipher=aes-cbc-essiv:sha256

And restarted the encryption device and activated the swap area:

# /etc/init.d/cryptdisks start
 * Starting remaining crypto disks...
 * cryptswap1 (starting)..
 * cryptswap1 (started)...                                                                                                     [ OK ] 
# swapon -a
# swapon -s
Filename				Type		Size	Used	Priority
/dev/mapper/cryptswap1                  partition	8388600	0	-1

The system swap area is now activated properly. I also did a reboot, just to confirm that it would also work properly on startup.

NOTE: Instead of “LABEL=swap”, we could have used the UUID directly, like this:

#                    
cryptswap1 UUID=6c7b8efe-87bc-4f7a-806f-a1d3eac91169  /dev/urandom swap,offset=8,cipher=aes-cbc-essiv:sha256

…which also works. However I think using the LABEL offers a nice balance between persistence and readability. If you have more than one swap area, you could label them swap1, swap2 etc.

Encrypted Swap Bug

Look at the changes made to /etc/crypttab again. Notice the “offset=8” bit ? It is very important, due to a known bug with Ubuntu 14.04 based systems (such as this Linux Mint 17.3 system). Without the offset, the UUID and LABEL of the swap partition are erased by the action of mapping and unmapping the encrypted device. In tests I did, without the “offset=8”, the UUID was lost after two map/unmap cycles. You don’t even have to activate the swap area. In other words, “offset=8” is required or the swap will fail to activate after one or two reboots.

The “offset” tells the kernel to create the swap area not at the start of the physical disk partition, but 8 blocks (of 512 bytes) beyond it. This presumably skips over the area in which the UUID information is stored.

Using Partition UUIDs (GPT Disks Only)

Besides referring to a disk partition by device name (/dev/sdXX), or the UUID or LABEL of its contents, there is another way: partition UUID. This applies only if the underlying disk has the modern GUID Partition Table (GPT) layout. The partition UUID, or GUID, is stored in the disk partition table (Wikipedia), rather than within physical partition. It is therefore not subject to being overwritten by the encryption subsystem, and obviates the bug just described.

To use the partition UUID, first identify it for the partition of interest. In this case, /dev/sda8:

# ls -l /dev/disk/by-partuuid | grep sda8
lrwxrwxrwx 1 root root 10 Mar 19 19:12 4ee83fc9-c5ee-42d1-ba28-9fb78008acde -> ../../sda8

Then just enter the full path of that file into /etc/crypttab, replacing the “LABEL=swap” bit.

# cat /etc/crypttab
#                 
cryptswap1 /dev/disk/by-partuuid/4ee83fc9-c5ee-42d1-ba28-9fb78008acde   /dev/urandom swap,cipher=aes-cbc-essiv:sha256

This time there is no need for the “offset” entry.

And now activate the swap area:

# /etc/init.d/cryptdisks start
 * Starting remaining crypto disks...
 * cryptswap1 (running)...
   ...done.
# swapon -a ; swapon -s
Filename				Type		Size	Used	Priority
/dev/mapper/cryptswap1                  partition	8388604	0	-1

Notice that, in the absence of the “offset” argument, the swap size has increased by 4 KB, from 8388600 to 8388604. This corresponds to the offset which was specified, ie 8 x 512 byte blocks.

Conclusion

I hope the above article has served to illuminate a tricky subject, and was not too long and rambling. Persistent identifiers should always be used in configuration files, to avoid severe system errors when the block device names (/dev/sdXX) change. Using the human readable part of the UUID (the “label”), rather than the long hexadecimal string, will aid readability. A label scheme should be chosen so that conflicts (where two or more items have the same label string) are avoided.

There is another slight advantage to using the label part of the UUID. If the UUID itself changes, or has to be recreated, any file referencing the label (as opposed to the hexadecimal string) will not need to be updated, so long as the same label is used by the new or replacement UUID. Tests carried out while writing of this article required the swap partition label to be recreated many times. Using the same label each time saved much editing of /etc/crypttab.

In the special case of encrypted swap on systems descendent from Ubuntu 14.04 (Such as Mint 17 LTS and others), an offset of at least 8 blocks should always be used. Alternatively, for GPT disks, the partition UUID (GUID) can be safely used as shown above. I didn’t try it, but for a GPT disk, a partition label could presumably be defined (for example with gdisk) and then used in /etc/crypttab or other configuration files, providing a handle which is persistent, human readable and not susceptible to the encrypted swap bug.

Ubuntu Encrypted Swap Bug

It is unfortunate that the bug fix at Ubuntu 15.04 has not been back ported to 14.04. Many distributions are derived from 14.04, and since it is LTS (long term support), will continue to have the problem baked-in until 2019. This includes Linux Mint, the most popular Linux distro according to Distrowatch.

A fix could be as simple as modifying line 145 the short ecrypt-setup-swap script:

	echo "cryptswap$i UUID=$uuid /dev/urandom swap,cipher=aes-cbc-essiv:sha256" >> /etc/crypttab

This is all a bit crass for a feature which is supposed to be providing “enterprise strength” encryption.

---

Appendix – Testing the Encrypted Swap Bug

For reference, here are some tests I did on the encrypted swap bug.

First, with an offset specified, and starting with the encrypted device being unmapped (shown by ls /dev/mapper), and the label being in place:

# cat /etc/crypttab
cryptswap1 LABEL=swap  /dev/urandom swap,offset=8,cipher=aes-cbc-essiv:sha256

# ls /dev/mapper
control

# blkid | grep sda8
/dev/sda8: LABEL="swap" UUID="6c7b8efe-87bc-4f7a-806f-a1d3eac91169" TYPE="swap"

The encrypted device will be mapped and unmapped 5 times. In order to avoid messing up the output with the many control characters generated by cryptdisks, I will just grep for errors:

# ( /etc/init.d/cryptdisks start ; /etc/init.d/cryptdisks stop ) | grep fail
# ( /etc/init.d/cryptdisks start ; /etc/init.d/cryptdisks stop ) | grep fail
# ( /etc/init.d/cryptdisks start ; /etc/init.d/cryptdisks stop ) | grep fail
# ( /etc/init.d/cryptdisks start ; /etc/init.d/cryptdisks stop ) | grep fail
# ( /etc/init.d/cryptdisks start ; /etc/init.d/cryptdisks stop ) | grep fail
# 
# ls /dev/mapper
control  efs2.file

No errors. Map the device and activate swap, just to check it is all working:

# /etc/init.d/cryptdisks start > /dev/null
# ls /dev/mapper
control  cryptswap1
# swapon -a ; swapon -s
Filename				Type		Size	Used	Priority
/dev/mapper/cryptswap1                  partition	8388600	0	-1

Looks good. Now deactivate and change the configuration to remove the “offset” argument:

# swapoff -a
# /etc/init.d/cryptdisks stop > /dev/null
# 
# echo "cryptswap1 LABEL=swap  /dev/urandom swap,cipher=aes-cbc-essiv:sha256" > /etc/crypttab
# cat /etc/crypttab
cryptswap1 LABEL=swap  /dev/urandom swap,cipher=aes-cbc-essiv:sha256

And run the test again:

# ( /etc/init.d/cryptdisks start ; /etc/init.d/cryptdisks stop ) | grep fail
# ( /etc/init.d/cryptdisks start ; /etc/init.d/cryptdisks stop ) | grep fail
# ( /etc/init.d/cryptdisks start ; /etc/init.d/cryptdisks stop ) | grep fail
   ...fail!

Oh dear. The encrypted swap area failed to map after two successful map/unmap cycles. Sure enough, the device isn’t there:

# ls /dev/mapper
control

…because the UUID has disappeared:

# blkid | grep sda8
#

Some further testing with blkid -c /dev/null showed that the UUID and label disappear after one activation/deactivation cycle, but remap successfully the second time, presumably due to the UUID/device association being cached somewhere.

Testing the Encrypted Swap Bug Using the GUID

When the same test was run using the partition UUID (GUID) in place of the LABEL in /etc/cryptsetup, no failures were encountered, whether or not the “offset” flag was present.