Project 5: What's in a Name? Shakespeare Meets Hadoop

Due: Wednesday, December 11, 11:59 PM

Quick Links

Project Overview

In this project, you will walk through a Hadoop tutorial and apply two examples learned there (WordCount and LineIndexer) to a publicly available data set of Shakespeare's works. You will submit the results of running these programs to a Google Spreadsheet Form.

General Requirements

Here are your goals for this project:

• Install Hadoop- You can download Hadoop (version 0.20.2) from here. Unpack it wherever you like, but remember where you do it.
• Install the Hadoop Eclipse plug-in (optional) - You can download the Eclipse plug-in for Hadoop from here. This seems to present no problems for the most recent version of Eclipse, but YMMV. It's not critical that you have this plug-in. In particular, we will not be using the HDFS features of the Hadoop plugin, as you will be running the examples in stand-alone mode, but you may find that the plugin does make it easier to work with Hadoop projects.
• Import the Skeleton Code- You can download this code from here. To import it, go to the Eclipse File menu and import the .zip file (or else unpack it, and import the top-level directory). once you do, you will see that some of the files don't build. You need to add the proper Hadoop libraries to the build path. To do this, go to your project's "Build Path" -> "Configure Build Path" and select the "Libraries" tab. Click "Add External JARs", go to the hadoop-0.20.2 folder you downloaded in step 1, and select and add all the jars in the root directory (should be 5 of them named "hadoop-0.20.2-....jar"). Then, click "Add External JARs" again, go to the lib folder, and add all the .jars from there as well. You should not need to add anything from the subdirectories. After this step, all of the project errors should go away. Notice that we have given you a working, updated WordCount example and a template file for an updated LineIndexer (to be used in working through the tutorial). Also notice that the Shakespeare data you'll need is included as a folder.
• Work through Module 4 of the Yahoo tutorial- Available here. Two caveats for the tutorial: It is outdated (based on a previous version of Hadoop) and it assumes a "pseudo-distributed" local HDFS setup. You will be using the much simpler stand-alone setup. In our skeleton code, we explain many of the changes made to update the WordCount example. While working through the tutorial, you should run the WordCount tool with and without the Combiner on the Shakespeare data (you will need to figure out how to do this!). You will run the examples as standalone applications. You need to pass two arguments to the main method (when you do "run as ..." you will want to edit the run configuration to specify command line arguments in Eclipse): an input directory, which should be shakespeare (it will find the directory in the skeleton code) which is the folder we've provided, and some output directory to contain results (it will be created at the same level as the shakespeare directory). After you have run WordCount for the first time, you can refresh the project and see this new directory. (Note you will need to delete the directory if you want to use it again when rerunning the code.) Inside it will be the results of running WordCount.
• Write an Updated LineIndexer class- At the point labeled Checkpoint in the tutorial you are asked to implement (as a class LineIndexer) a reverse index, mapping words to the files they appear in. The end of the tutorial provided a solution to the LineIndexer problem posed earlier. Like the rest of the tutorial, this example is out of date. Using resources we've provided and the Hadoop API, update the LineIndexer to use non-deprecated functions (i.e. those in *.hadoop.mapreduce.* instead of *.hadoop.mapred.*). Run your solution on the Shakespeare input. (Note that the example solution does not avoid duplicates in the output index; try to make your solution avoid duplicates, and sort the results, since it will assist in providing one of the answers.)
• Report your results- Report your results to a Google Form available here. You may make duplicate submissions. Note that the results we are asking for are printed to the console during each run.

Hints

The Context object that is passed into the map and reduce methods is an implementation of the MapContext and ReduceContext interfaces, respectively. These interfaces both extend the TaskInputOutput interfaces. The LineIndexer sample solution does not use Context objects, but your solution will. We have included code that extracts the filename in the skeleton provided to you (implementing the hint provided in the middle of the tutorial). Your job is to write the rest of the code using this hint.

What to Turn In

Submit the results of running your code as printed to the console to the Grading Form. Projects will be graded based on the values submitted here.

Also, you should submit your code to the submit server so that we can check that you implemented the updated LineIndexer. Feel free to submit the entire codebase or simply the LineIndexer file. We will look for the LineIndexer.java file and perform a few lexical checks.

Web Accessibility