Data Analysis

Data analytics is the hottest thing around.  Knowing how to structure and manipulate data, and then find answers is the sexiest job out on the market today! [caption id="" align="alignnone" width="1024"] Cisco Live[/caption] Before getting into cool things like apache spark, hadoop, or using EMR, let's just start out with a basic example:  Word count on a book. Project Guttenburg has a ton of books out there.  I'm going to choose one and then just do some basic text manipulation.  Not everything needs big data and there's a lot you can do just from your own laptop. I'm going to borrow from a great post here and do some manipulation on some text I found.  I'm not going to use any hadoop.  Just plain python. I'll use the same script, but I stripped out the punctuation with some help from Stack Overflow:

!/usr/bin/env python

import sys 

# input comes from STDIN (standard input)
for line in sys.stdin:
    # remove leading and trailing whitespace
    line = line.strip()
    # split the line into words
    words = line.split()
    # increase counters
    for word in words:
        w = ''.join(e for e in word if e.isalnum())
        # write the results to STDOUT (standard output);
        # what we output here will be the input for the
        # Reduce step, i.e. the input for reducer.py
        #   
        # tab-delimited; the trivial word count is 1
        print '%s\t%s' % (w, 1)

  Here's the pass:

cat ../data/bom.txt | ./mapper.py | tee -a ../out/bom-out.txt

We're simply printing all the text and then piping to our mapper program and then storing the results in the bom-out.txt file. So now we have bom-out.txt which is just a word with a count next to it: foo 1 Now we need to sort it.

cat bom-out.txt | sort -k1,1 | tee bom-sorted-out.txt

So now we have bom-sorted-out.txt.  So next up, we do the word count.

cat ../out/bom-sorted-out.txt | ./reduce.py | tee ../out/bom-reduced-out.txt

This gives us the output, but now let's see which word is used the most.  This is another sort.

cat ../out/bom-reduced-out.txt | sort -k2n

This gives some of the usual suspects:

...
their	2800
it	3061
he	3145
I	3306
unto	3641
in	3674
they	4446
And	4565
to	6445
that	6842
and	11765
of	11787
the	19120

We could probably do better if we were to make it so case didn't matter.  We could then also do it in a one pass script.  Let's try it.

Version 2

In the mapper script we change the last line to be:

print '%s\t%s' % (w.lower(), 1)

This way it spits everything out in lowercase.  Now to run the script in one line, we do the following:

cat ../data/bom.txt | ./mapper.py | sort -k1,1  | ./reduce.py | sort -k2n | tee ../out/results2.txt

Now our output looks a little different:

...
their	2807
it	3075
he	3173
i	3306
unto	3641
in	3693
they	4485
to	6450
that	6864
of	11814
and	16331
the	19230

What we've shown here is the beginning of what things like Hadoop do for us.  We have unstructured data and we apply two operations:  Map:  This is where we do the count of each word.  Reduce:  This is where we count how many times each word was done.  In this case our data set wasn't too huge and could be done on our laptop. Here's another book:

...
a	8177
his	8473
i	8854
for	8970
unto	8997
shall	9838
he	10420
in	12667
that	12913
to	13562
of	34618
and	51696
the	63924

There are still a few problems but this seems to work well.  The next step is to use a natural language processing kit and find similar phrases.  We then could dump this into HDFS and process all kinds of books.  Lots of interesting places to go from here! Last note: I did upload the data and scripts to github.