5,000 whois lookups in under a minute

There have been many times in my IT career I have been required to solve very specific problems. My first step to solving these problems almost always begins with the same question: “Has it been done before?” Most of the time, if the problem is specific enough the answer is a resounding, “No.”

The other day I came across the following conundrum. I was asked to whois 5000 hosts in a reasonable amount of time (minutes) given a server with only one processor core and 2GB of RAM.

I ended up creating a BASH script to do the following:

1. Break down the huge text file into smaller text files each containing only a few IPs.

2. Spin up a separate process for each of these text files to do the whois lookups.

[code language=”bash”]
#!/bin/bash
#Super fast whois lookup – Jamin B
echo Enter the file you want to read in:
read file
echo Spawn into how many threads?
read splitNumber
totalLines=`<$file wc -l` #determine the total number of lines
currentLine=0
currentFile=0
splitSize=$(($totalLines/$splitNumber))
echo Each file will contain $splitSize IPs.
rm -rf tmpdir
mkdir tmpdir
rm -rf whois
mkdir whois
cat $file | while read line
do
(( currentLine = currentLine +1))
if [ $(($currentLine%$splitSize)) == 0 ]; then
(( currentFile = currentFile + 1))
clear
echo Creating file $currentFile in tmpdir
cat /proc/meminfo | grep MemFree
fi
echo $line >> tmpdir/$currentFile
done
i=0
while [ $i -lt $splitNumber ]; do
(( i = i + 1 ))
clear
echo Spawning process for file number $i
cat tmpdir/$i | while read line
do
whois $line > whois/$line.txt &
done
done
[/code]

The result: 5000 whois reports generated in under a minute!

Screen Shot 2014-01-10 at 1.27.54 PM

-Jamin Becker

Leave a Reply

Your email address will not be published. Required fields are marked *