Project 5: What's in a Name? Shakespeare Meets Hadoop

Due: Thursday, December 9, 11:59 PM

Quick Links

Project Overview

In this project, you will walk through a Hadoop tutorial and apply two examples learned there (WordCount and LineIndexer) to a publicly available data set of Shakespeare's works. You will submit the results of running these programs to a Google Spreadsheet Form.

General Requirements

Here are your goals for this project:

• Install Hadoop- You can download Hadoop (version 0.20.2) from here. Dr. Bill Pugh has some overview slides from a previous iteration of this project available here, which include installation instructions for Hadoop and the Hadoop Eclipse Plugin. (Note that we will not be using the HDFS features of the Hadoop plugin, as you will be running the examples in stand-alone mode, but you may find that the plugin does make it easier to work with Hadoop projects.)
• Import the Skeleton Code- In addition to the standard steps, you will need to modify the Java Build Path entries to add the jars from your own local Hadoop installation. Adding all jars from the Hadoop root and lib directories should remove all of the project's conflicts. Notice that we have given you a working, updated WordCount example and a template file for an updated LineIndexer (to be used in working through the tutorial). Also notice that the Shakespeare data you'll need is included as a folder.
• Work through Module 4 of the Yahoo tutorial- Available here. Two caveats for the tutorial: It is outdated (based on a previous version of Hadoop) and it assumes a "pseudo-distributed" local HDFS setup. You will be using the much simpler stand-alone setup. In working through the tutorial, you may find the updated Apache Hadoop tutorial here to be a helpful supplement, but note that its examples are still using the deprecated *.mapred.* functions. In our skeleton code, we explain many of the changes made to update the WordCount example. A detailed explanation of differences is also available here. While working through the tutorial, you should run the WordCount tool with and without the Combiner on the Shakespeare data. You will need to pass two arguments to the main method: an input directory, which should be the shakespeare folder we've provided and an output directory to contain results.
• Write an Updated LineIndexer class- The end of the tutorial provided a solution to the LineIndexer problem posed earlier. Like the rest of the tutorial, this example is out-dated. Using resources we've provided and the Hadoop API, update the LineIndexer to use non-deprecated functions (i.e. those in *.hadoop.mapreduce.* instead of *.hadoop.mapred.*). Run your solution on the Shakespeare input with and without the combiner.
• Report your results- Report your results to a Google Form available here. You may make duplicate submissions. Note that the results we are asking for are printed to the console during each run.

What to Turn In

Submit the results of running your code as printed to the console to the Grading Form. Projects will be graded based on the values submitted here. Note that you will need the input and output number of records to reduce for combined and non-combined versions of WordCount and LineIndexer.

Also, you should submit your code to the submit server so that we can check that you implemented the updated LineIndexer. Feel free to submit the entire codebase or simply the LineIndexer file. We will look for the LineIndexer.java file and perform a few lexical checks.

Web Accessibility