Code snippet: Combining samples files

For the past few months, I have to go through a heap of data files containing measurement data from routers (packet throughput, packet loss etc). Creating the graphs in gnuplot does not require combining these files, but importing all the different data files into another program (LaTeX/Word/Excel/OO Writer/whatever) is practically a torture.

If, for whatever reasons, you need to combine a lot of files of space-separated values into one big file, I'm using this script which I call cf2.

(2 stands for version. cf1 was kinda crap :D )

#!/bin/bash

# cf2: Combine Files :)

if [ $# -eq 1 ]; then

exec 7<$1

let eof=0

while [ $eof -eq 0 ]

do

if read line1; then

if read line2 <&7; then

echo -e -n "$line2\t$line1\n"

else

let eof=1

fi

else

let eof=1

fi

done

exec 7>&-

elif [ $# -eq 2 ]; then

exec 7<$1

exec 8<$2

let eof=0

while [ $eof -eq 0 ]

do

if read line1 <&7; then

if read line2 <&8; then

echo -e -n "$line1\t$line2\n"

else

let eof=1

fi

else

let eof=1

fi

done

exec 7>&-

exec 8>&-

else

SELF=$0

FIRST=$1

shift

$SELF $@ | $SELF $FIRST

fi

How to use it:

# cf2 datafile1.dat datafile2.dat [...] datafileX.dat > newCombinedFile.dat

So if your data files are something like this:

# file1.dat

1 2 3 4

1 2 3 4

1 2 3 4

# file2. dat

5 6 7 8

5 6 7 8

5 6 7 8

Running cf2 file1.dat file2.dat will give you:

1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

I hope somebody will find this helpful. It has certainly sped up data manipulation for me.