Skip to content

Bash Script to read in a big text file and do something with it

Published by NiteMayr on July 9, 2010

At the office I often end up producing little scripts to do this and that and today I had to deal with a large file that was causing a custom app to bork. In short it needed to be read to the app in chunks; usually I have to do more prosaic stuff, but this is a neat little foundational app to get things done.

I needed this today to solve a very basic problem, maybe you can use it too

#!/bin/bash

declare -i RESET=0
declare -i TOTAL=0
declare -i LINES=$(cat $1 | wc -l)
echo -n > somefile

while read LINE ; do
	RESET=$RESET+1	

	if [ $RESET -lt 1000 ]; then
		echo $LINE >> somefile
	else
	   RESET=0
	   somecommand -file somefile  | mail -s "Output from somecommand" 2someguy@somewhere.koo
	   echo -n > somefile
	   TOTAL=$TOTAL+1000;
	fi

	declare -i REMAINS=$LINES-$TOTAL
	if [ $REMAINS -lt 0 ]; then
	   tail -$REMAINS $1 > somefile
	   somecommand -file somefile  | mail -s "Final Run of somecomand" someguy@somewhere.kooo
	fi

done < $1
Published inLondonWork