There have been many times in my IT career I have been required to solve very specific problems. My first step to solving these problems almost always begins with the same question: “Has it been done before?” Most of the time, if the problem is specific enough the answer is a resounding, “No.”
The other day I came across the following conundrum. I was asked to whois 5000 hosts in a reasonable amount of time (minutes) given a server with only one processor core and 2GB of RAM.
I ended up creating a BASH script to do the following:
1. Break down the huge text file into smaller text files each containing only a few IPs.
2. Spin up a separate process for each of these text files to do the whois lookups.
#Super fast whois lookup – Jamin B
echo Enter the file you want to read in:
echo Spawn into how many threads?
totalLines=`<$file wc -l` #determine the total number of lines
echo Each file will contain $splitSize IPs.
rm -rf tmpdir
rm -rf whois
cat $file | while read line
(( currentLine = currentLine +1))
if [ $(($currentLine%$splitSize)) == 0 ]; then
(( currentFile = currentFile + 1))
echo Creating file $currentFile in tmpdir
cat /proc/meminfo | grep MemFree
echo $line >> tmpdir/$currentFile
while [ $i -lt $splitNumber ]; do
(( i = i + 1 ))
echo Spawning process for file number $i
cat tmpdir/$i | while read line
whois $line > whois/$line.txt &
The result: 5000 whois reports generated in under a minute!