Here’s a quick script to show duplicate files on Linux. It should cope with arbitrary spaces in file names, and to save time and CPU resources, it will checksum only files of the same size.
Usage: Save the script to dups.sh or whatever, then run it with no arguments. A list of duplicated files is output.
Continue readingSD cards are used in digital cameras, phones and other devices, where their speed and large capacity makes them useful for storing pictures, video and other voluminous multi-media items. It is quite common these days for a mobile device to contain a 16 Gb or 32 Gb SD card.
With the devices being so mobile, backups are easily overlooked. And it is quite easy for an SD card to become corrupted, for example if the card is removed while the device is on, or the battery is taken out while a video is being shot.
I was given a corrupted 16 Gb card and asked to recover the files, if possible. The rest of this post explains how the data was safely restored using simple Linux tools. Continue reading
The venerable fat16 and fat32 file systems are still in widespread use today. Devices such as digital cameras, satellite navigation systems, memory sticks and mp3 players all make use of FAT
Mount a FAT file system in Linux, and it will appear as something like this:
[root@pluto ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdb 7716112 7274796 441316 95% /media/0EB5-6037 /dev/sdc1 15549952 14225152 1324800 92% /media/1FC3-3137
Those hex numbers on the left are the default volume labels, and pretty unfriendly they are. The two entries above actually correspond to an MP3 player (Sansa Clip+) and the player’s expansion SD card. I used mlable to allocate more meaningful volume names. Continue reading
Flash drives (as in memory sticks) are a popular choice for backups. A 64 Gb drive can be bought for just over £20 at the time of writing. Once your data is backed up to the drive, it is easy to store in a safe place or transport off site for added protection. Actually that applies to any portable hard drive, as does the following procedure.
Most flash drives come formatted as FAT32. That is fine from day-to-day but there is a strict 4 GB limit applying to the size of any file. Backup software is likely to produce large archive files well over 4 GB, so won’t work well with FAT32. What’s needed therefore, is a better file system. Continue reading
Deleting any file under Unix/Linux is usually a simple matter of using the “rm” command. Some files are more stubborn. If the file name contains special characters, or begins with a dash (“-“), it can be hard to get rid of:
bash-4.2$ ls -l total 0 -rw-rw-r--. 1 james james 0 Aug 25 14:46 -a -rw-rw-r--. 1 james james 0 Aug 25 14:46 logfile -rw-rw-r--. 1 james james 0 Aug 25 14:45 some'file bash-4.2$ rm some'file > bash: unexpected EOF while looking for matching `'' bash: syntax error: unexpected end of file
One obvious solution is to use a GUI. Highlight the awkward file in any file manager, hit the delete key, and it’s gone. But GUIs aren’t available everywhere. You may have only shell access to a server, for example. Continue reading
So you have a directory with millions of files, and ls just hangs ?
Use ls -1 -f to show the files immediately. To delete the files, if you want to remove ALL files in the current directory, use something like
ls -1 -f | xargs rm
After cleaning up very many unwanted files, you are likely to be left with a huge and sparse directory object. Three million files in one directory, for example, apart from taking up space in themselves, will likely push the directory object to occupy over 100 Mb of space. Continue reading
Backups on a LAMP server were much smaller than expected, and the cause was excessively large log files. This post explains the apparent paradox and the steps taken to trace and fix the problem.
The server in question runs the usual set of grandfather-father-son backups. Before each backup, the software prints the expected size of the backup into a log file. On other servers the estimate is usually bang-on. On this server, the backup turned out to be about 12 times smaller than the estimate. Not a problem in itself, but puzzling and indicative of something untoward on the system. Continue reading
This article explains how zfs errors can sometimes persist even after the files containing the offending blocks have been deleted, and presents an easy way of removing corruption.
Most solaris sysadmins will be familiar with the following situation. A zpool gives errors about failed checksums and enters a degraded state: Continue reading
This post describes a problem with receiving Linux kernel updates after installing Fedora 16 in a dual boot environment. The Fedora installation process allows the user to rewrite Grub in the master boot record (MBR) or to leave the MBR untouched. I chose to leave it untouched, leading to a later kernel update problem which was fairly easily solved with some Grub 2 commands. Continue reading
HP-UX file systems can be extended with a combination of the lvextend and fsadm commands. It’s well known procedure that has been in widespread use for several years, allowing online file system extensions, ie. the file system can be extended while it is mounted and busy. The file system type must be of type “vxfs” and enclosed in an LVM logical volume.
Occasional difficulties can occur if the file system is 100% full. I recently encountered the following vx_nospace error when trying to extend a file system from 9 Gb to 59 Gb under HP-UX 11.31. Continue reading