An Example of Parallel Processing

This post shows how to use parallel processing to get a CPU intensive job done faster in Unix/Linux. By splitting a large task into several parts, it is quite easy to give each part to a separate CPU, and complete the task many times faster than it would on a single processor.

These days, even small PCs and other devices often come equipped with several CPU cores. But some tasks will use only one core, sometimes using 100% of it, while other cores stand by idle. Sometimes this is a waste of resources.

Look at those CPUs

I am writing this on a Linux laptop containing 8 CPU cores. Actually it is a quad core Haswell system with 2 hardware threads per core. But Linux thinks it has 8 entirely separate processors, look:

bash-4.2$ grep proc /proc/cpuinfo
processor       : 0
processor       : 1
processor       : 2
processor       : 3
processor       : 4
processor       : 5
processor       : 6
processor       : 7

You would get similar output on any multi-processor system. For example an x86 Xeon system, a Sun Sparc “T” box or a virtual machine. However the “CPUs” are provisioned, the OS just sees them as regular CPUs.

Give the System Some Work to do

To get those processors working, give them some hard work to do. Compressing data is CPU intensive.

For example, compress (gzip) a large text file called big.txt (3.2 Gb).

bash-4.2$ ls -lh big.txt
-rw-rw-r--. 1 james james 3.2G Feb  5 21:46 big.txt
bash-4.2$ time gzip big.txt

real    0m49.897s
user    0m48.857s
sys     0m1.016s

It took about 50 seconds to compress big.txt. Repeating the test gave similar results: 50.7 s, 50.6 s and 51.2 s.

Gzip Uses Only One CPU

Running ‘top‘ in another window during the above test, it can be seen that CPU usage is about 12.5%. Top shows CPU stats across all processors, so 12.5% is one eighth of the 8 CPUs in the system, or one whole CPU. The gzip task completely occupied a single core, taking 100% of its resources. Meanwhile the other 7 processors did pretty much nothing – shown by the 87.4% idle time.

top - 21:53:09 up 37 min,  2 users,  load average: 0.24, 0.20, 0.15
Tasks: 183 total,   2 running, 181 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.3 us,  0.2 sy,  0.0 ni, 87.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8096132 total,  5764756 used,  2331376 free,    45284 buffers
KiB Swap:  8388604 total,        0 used,  8388604 free,  4942988 cached

Also, running ps -eLF during the gzip shows it running on a single processor (processor number 1), as you would expect. And there is only 1 thread.

bash-4.2$ ps -eLF  | egrep 'UID|zip' | grep -v grep
james     2376  1750  2376 90    1  1143   504   1 22:05 pts/1    00:00:04 gzip big.txt

Example 1 – Run Small Jobs in Parallel

If you have many files to compress, and many CPUs, it is quicker to zip them in parallel than one at a time. Putting jobs into the background with an ampersand (&) will make them run at the same time. Doing a “wait” command will make the shell wait until the last one completes. I had a directory containing hundreds of files, each 3 Gb in size, that needed to be compressed. It was easier to write a script than do it by hand:


ls *.dbf |
while read file
   jobcount=$(( $jobcount+1 ))
   print "$jobcount gzip $file &"
   gzip $file &
   if [[ $jobcount -eq $PARALLEL ]] then
      print "waiting for $jobcount jobs to complete"
      print "$jobcount jobs completed"

print "waiting for remaining jobs to complete"

The above piece of script will steadily compress all *.dbf files in the current directory, using up to 4 cpus at once, running up to 4 “gzip” commands at the same time.

The PARALLEL variable (4) is the maximum number of jobs that will run at once. As the system contained 10 CPUs, and was not busy, it was reasonable to commandeer 4 of them. Note the wait command in the inner block will be executed when there are 4 or more files yet to be compressed. The outer loop wait will execute when less than 4 remain. I actually have the above code in a cron job, housekeeping that folder.

Example 2 – Split a Big Task into Smaller Tasks

Try the same test again, but this time split up the big file in to 8 separate parts.

First, take a check sum of the file with good old cksum

bash-4.2$ cksum big.txt
1754962233 3345539400 big.txt

Now use the split command with “-n” to chop the file up:

-rw-rw-r--. 1 james james 3.2G Feb  5 21:46 big.txt
bash-4.2$ split -n 8 big.txt
bash-4.2$ ls -lh
total 6.3G
-rw-rw-r--. 1 james james 3.2G Feb  5 21:46 big.txt
-rw-rw-r--. 1 james james 399M Feb  5 22:22 xaa
-rw-rw-r--. 1 james james 399M Feb  5 22:22 xab
-rw-rw-r--. 1 james james 399M Feb  5 22:22 xac
-rw-rw-r--. 1 james james 399M Feb  5 22:22 xad
-rw-rw-r--. 1 james james 399M Feb  5 22:22 xae
-rw-rw-r--. 1 james james 399M Feb  5 22:22 xaf
-rw-rw-r--. 1 james james 399M Feb  5 22:22 xag
-rw-rw-r--. 1 james james 399M Feb  5 22:22 xah

…giving 8 file pieces of 399 Mb each, called xaa, xab, …, xah.

(split on some older versions of Solaris and Linux doesn’t have the -n switch, but you can use -b instead to get pieces of a given size)

Run the Compress Again, Parallelized

This time, 8 gzip tasks will be run in parallel. This takes a bit of thinking about. In order to run the jobs in parallel (ie. at the same time), they need to be launched in the background of the shell. But the time command only works on jobs running in the foreground.

One solution is to use the wait command, which simply waits for all background jobs to finish. Then the commands needed would be:

gzip xaa &
gzip xab &
gzip xac &
gzip xad &
gzip xae &
gzip xaf &
gzip xag &
gzip xah &

I could put that in a script and invoke it with time, but it is quicker to just use awk.

bash-4.2$ time ls x* | awk '{print "gzip "$1" &"} END {print "wait"}' | sh -x
+ gzip xaa
+ gzip xab
+ gzip xac
+ gzip xag
+ wait
+ gzip xah
+ gzip xae
+ gzip xaf
+ gzip xad

real    0m10.048s
user    1m18.151s
sys     0m1.564s
bash-4.2$ ls
big.txt  xaa.gz  xab.gz  xac.gz  xad.gz  xae.gz  xaf.gz  xag.gz  xah.gz

All of the data has been gzipped in about 10 seconds, one fifth of the time it took before. Notice that the “user” time was recorded as 1 minute 18 seconds, even though only 10 seconds of real time actually passed. This is just a quirk of the time command.

Looking at top during the parralized gzip operation, it is pretty obvious that all 8 CPUs were being hammered by 8 gzip commands:

top - 23:03:44 up  1:48,  2 users,  load average: 0.61, 0.29, 0.23
Tasks: 193 total,   8 running, 185 sleeping,   0 stopped,   0 zombie
%Cpu(s): 85.1 us,  1.5 sy,  0.0 ni, 11.6 id,  1.5 wa,  0.2 hi,  0.1 si,  0.0 st
KiB Mem:   8096132 total,  7938740 used,   157392 free,     3020 buffers
KiB Swap:  8388604 total,     6512 used,  8382092 free,  7096640 cached

 2733 james     20   0    4572    504    352 R  99.8  0.0   0:05.40 gzip
 2735 james     20   0    4572    504    352 R  99.8  0.0   0:05.39 gzip
 2736 james     20   0    4572    504    352 R  99.4  0.0   0:05.39 gzip
 2738 james     20   0    4572    500    352 R  99.1  0.0   0:05.39 gzip
 2734 james     20   0    4572    504    348 R  98.8  0.0   0:05.37 gzip
 2737 james     20   0    4572    504    352 R  97.1  0.0   0:05.33 gzip
 2731 james     20   0    4572    500    352 R  49.2  0.0   0:02.71 gzip
 2732 james     20   0    4572    504    348 D  46.2  0.0   0:02.60 gzip

and ps agrees, showing one gzip job in each processor:

bash-4.2$ ps -eLF  | egrep 'UID|zip' | grep -v grep
james     2764  2763  2764 99    1  1143   504   5 23:05 pts/1    00:00:09 gzip xaa
james     2765  2763  2765 99    1  1143   504   7 23:05 pts/1    00:00:09 gzip xab
james     2766  2763  2766 99    1  1143   504   2 23:05 pts/1    00:00:09 gzip xac
james     2767  2763  2767 99    1  1143   500   3 23:05 pts/1    00:00:09 gzip xad
james     2768  2763  2768 99    1  1143   500   1 23:05 pts/1    00:00:09 gzip xae
james     2769  2763  2769 99    1  1143   500   4 23:05 pts/1    00:00:09 gzip xaf
james     2770  2763  2770 99    1  1143   504   6 23:05 pts/1    00:00:09 gzip xag
james     2771  2763  2771 99    1  1143   504   0 23:05 pts/1    00:00:09 gzip xah

Glue it Back Together

Okay, our compressed data is still in 8 parts. Putting it back together is easy:

bash-4.2$ ls
big.txt  xaa.gz  xab.gz  xac.gz  xad.gz  xae.gz  xaf.gz  xag.gz  xah.gz
bash-4.2$ cat x* > all.gz

Now there is a file all.gz. Because of the way gzip works, this file won’t be identical to the original bigfile.txt.gz, but the decompressed data will be identical. To prove that , just gunzip it and do another checksum:

bash-4.2$ gunzip all.gz
bash-4.2$ cksum all big.txt
1754962233 3345539400 all
1754962233 3345539400 big.txt

Uncompressed, the all file gives the same checksum as big.txt, also matching the original checksum taken at the outset of the test (1754962233)


This is just a rough example in the shell to show the effect of multi-processing. Real applications do it differently: buy dividing processes into many threads and running threads across different CPUs. For example, Firefox will run perhaps 40 threads and spread them over all available CPUs.

Note that the gzip test is definitely CPU bound. Repeating the test gives a similar time. Disk/cache effects can be discounted because CPU is the bottleneck in this case. Even if the while file were cached in memory, it would not improve execution speed.

Obviously, this technique applies only where the main operation is “associative”. That is, where the processed data is identical to the concatenation of the processed pieces of data. This is part of the design of several compress tools and is part of the gzip RFC.

Leave a Reply

Your email address will not be published. Required fields are marked *