Category Archives: Unix

Recovering Data from a Corrupted SD Card

Posted on by
5

SD cards are used in digital cameras, phones and other devices, where their speed and large capacity makes them useful for storing pictures, video and other voluminous multi-media items. It is quite common these days for a mobile device to contain a 16 Gb or 32 Gb SD card.

With the devices being so mobile, backups are easily overlooked. And it is quite easy for an SD card to become corrupted, for example if the card is removed while the device is on, or the battery is taken out while a video is being shot.

I was given a corrupted 16 Gb card and asked to recover the files, if possible. The rest of this post explains how the data was safely restored using simple Linux tools. Continue reading →

vi: Terminal too wide

Posted on by
7

Workaround for vi “Terminal too wide” problem

Use the Unix command line and sooner or later, you will be editing text files. One of the best ways of doing that is with the vi editor. It is available as standard on almost every unix/linux system. While other editors are available (ed, emacs, vim, etc), vi is quick and convenient. It offers a good balance between usability and ubiquity.

This article offers a workaround for the annoying “Terminal too wide” problem encountered by vi users on Solaris.

Vi was originally written for screens (terminals) which were 80 characters wide. In a modern windowing environment, the terminal has been replaced by virtual terminal apps – xterm, lterm, Terminal and many others. The width of a virtual terminal depends on how much big you make the window. On a large screen it could easily be 200 characters or more. Continue reading →

Sorting with "-k" on Unix and Linux

Posted on by
9

The “sort” command on Solaris has a “-k” switch for sorting by a particular field. For example, “sort -k 2” will sort by the second field on each line of input. Parts of fields can be further specified with “-k n.m“, says the man page.

For example, “sort -k 2.3” should sort by the second field, starting with the third character in that field. But the man page isn’t the clearest, and getting the “-k x.y” notation to work is tricky. Tricky until you realize it never works you also supply the “-b” argument. Same on Linux. Continue reading →

Using Tcpdump to See Background DNS Requests

Posted on by
Reply

This post explains how to use tcpdump on Linux to detect and investigate DNS requests. One of our Red Hat client systems was making requests to an old DNS server, even though it had been adjusted, through a change to/etc/resolv.conf, to point to a new one.

Requests to the old server were identified as follows.

[root@pluto root]# tcpdump -i eth0 -l -vvv dst host 192.168.1.103 and dst port 53
(...waited 15 mins or so...)
tcpdump: listening on eth0
16:38:18.019703 pluto.mycompany.com.41783 > olddnsbox.mycompany.com: [bad udp cksum 48f5!]  21331+ A? somebox.mycompany.com. (42) (DF) (ttl 64, id 48623, len 70)
16:38:18.033461 pluto.mycompany.com.41783 > olddnsbox.mycompany.com: [bad udp cksum 5919!]  12099+ A? somebox.mycompany.com. (42) (DF) (ttl 64, id 48625, len 70)

192.168.1.103 is the ip address of the old DNS server. Tcpdump shows network packets sent to the standard DNS port (53) at that IP address. Requests were few so I had to wait 15 or 20 minutes to capture the above.

The client was last rebooted a year ago, many months before /etc/resolv.conf was last edited. Tcpdump shows that some application is still querying the old server. The fix was to reboot the client, restarting the erroneous application and stopping the outdated requests.

Profiling and Tracing Processes in Linux

Posted on by
1

This article shows how a Linux process can be traced and profiled. Using the “last” command as an example, profiling is used to explain why a process was very slow, and why another, very similar, process (dump-utmp) was much faster. “Tracing” here means seeing what a process is doing at any moment. “Profiling” means showing (afterwards) how long it spent doing different things.

“Last” is a Linux command that reads and summarizes the utmp file, where login records are stored. I had a “last” command taking hours to complete because the utmp file had grown large (1.9 Gb). Used strace to see what it was doing. Continue reading →

Deleting Awkward Files

Posted on by
Reply

Deleting any file under Unix/Linux is usually a simple matter of using the “rm” command. Some files are more stubborn. If the file name contains special characters, or begins with a dash (“-“), it can be hard to get rid of:

bash-4.2$ ls -l
total 0
-rw-rw-r--. 1 james james 0 Aug 25 14:46 -a
-rw-rw-r--. 1 james james 0 Aug 25 14:46 logfile
-rw-rw-r--. 1 james james 0 Aug 25 14:45 some'file
bash-4.2$ rm some'file
> bash: unexpected EOF while looking for matching `''
bash: syntax error: unexpected end of file

A Few Alternatives

One obvious solution is to use a GUI. Highlight the awkward file in any file manager, hit the delete key, and it’s gone. But GUIs aren’t available everywhere. You may have only shell access to a server, for example. Continue reading →

Sendmail Authentication for Local Mail Delivery

Posted on by
Reply

These days, service providers like British Telecom have tightened up their requirements on email delivery. Most ISPs now accept email only from servers that can authenticate through SMTP. Sendmail has authentication built in, allowing administrators to use an “authinfo” file, for example. The server then authenticates every time it sends mail out to the ISP.

That’s great for reducing spam. But it can have a side-effect of killing local mail on the server. Here is an recent example from Debian Squeeze. The authentication features that allowed this server to successfully send mail through the ISP were preventing local mail from working: Continue reading →

Large Directory Causes ls to "Hang"

Posted on by
14

So you have a directory with millions of files, and ls just hangs ?

Use ls -1 -f to show the files immediately. To delete the files, if you want to remove ALL files in the current directory, use something like

ls -1 -f | xargs rm

After cleaning up very many unwanted files, you are likely to be left with a huge and sparse directory object. Three million files in one directory, for example, apart from taking up space in themselves, will likely push the directory object to occupy over 100 Mb of space. Continue reading →

Perl – OR Pattern Match Slow, Use Two Patterns Instead

Posted on by
1

A handy feature of regular expressions is their ability to “or” match. Searching for two strings in a file is easy with a construct like “egrep ‘root|uucp’ /etc/passwd”. The vertical bar (“|”) acts as an “or” operator. Perl supports the vertical bar too, and the same match could be achieved in Perl thus:

bash-4.2$ perl -n -e 'print $_ if /root|uucp/' < /etc/passwd root:x:0:0:root:/root:/bin/bash uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin

Weirdly though, this construction is up to 10 times slower than 2 separate matches performing the same search, as can be shown with a quick demonstration. Continue reading →

Process Substitution and Pipes

Posted on by
Reply

Command substitution is a widely used feature of the Bash and Korn shells, allowing the output of one command to be captured and used in another. Like this:

$ echo "Backup started at $(date)"
Backup started at Fri Mar 16 15:35:14 GMT 2012

Command substitution is not to be confused with that less well known (and, to be honest, less useful) shell feature, process substitution. Despite being rarely used, process substitution is worth knowing about, if only because it illuminates other fundamental unix features – the shell, sub processes, named and unnamed pipes.

This post discusses process substitution, command substitution and the vertical bar (|). Three very different shell features, but all making use of unnamed pipes, and so not as different as they first appear. The examples are from Linux but also work on Solaris 10 and, due to the ubiquity of pipes, are likely to work on other unixes too. Continue reading →