|
|
GNU Parallel: Speed Up Processing With Multi-cores, Clusters
By Taylor Gillespie
Expert Author
Article Date: 2010-10-19
Looking at the wall that is Moore's Law, computer processor manufacturers placed more than one processor core, each one essentially an independent operating unit, on each chip, but software must be rewritten to take advantage of being able to each core as an individual processor instead of the entire chip as one brute processor. Most software has not been rewritten to target each individual processor core.
GNU Parallel allows a system administrator to fully utilize the multiple cores that exist in computers today, as well as distribute the processing load to other networked machines. Exhibited as xargs-like program, GNU Parallel can be easily refactored into any existing setup.
The xargs command should be familiar to any Unix system administrator. It allows you to build the parameters to be passed into a program, though many administrators learned of it through the "Argument list too long" error when using find . GNU Parallel's command line syntax is very close to that of xargs, so it should be an easy modification. Using GNU Parallel, a typical optimization to current shell scripts is unrolling any command loops to be processed in parallel. One beautiful thing about GNU Parallel is that behaves more similar to a standard Unix utility. The standard output and standard error are managed so as to be output in the correct input order and not overlapping as the tasks run.
Parallel computing is here. Whether you are processing a batch of files, converting video, rendering some Blender, audio processing, mapping and reducing, GNU Parallel will use multiple cores or even multiple machines to distribute the processing load of any existing Unix command without intense modification. Traditional multiple processing tasks will take less time to finish. GNU Parallel, a program that addresses a fundamental need, should find its way into any Unix installation, but be aware to watch and adjust the niceness to appropriately modify the priority if the machine is running any other jobs.
About the Author:
Taylor is a Staff Writer for WebProNews
|
|
