MapReduce Function

MapReduce on Amazon AWS Follow Amazon Web Services Setup Guidelines ([login to view URL]~song/adc/Assignments/[login to view URL]) to create a free account and setup EC2 and S3. We will use data from the Google Books n-gram viewer corpus. N-grams are fixed size tuples of items. In this case the items are words extracted from the Google Books corpus. The n specifies the number of elements in the tuple, so a 5-gram contains five words. This data set is freely available on Amazon S3 in a Hadoop friendly file format and is licensed under a Creative Commons Attribution 3.0 Unported License. The original dataset is available from [login to view URL] You can use a subset of the original Google dataset for this assignment, for example a subset of the 4- gram English dataset. The data is in a simple txt file, and each row of the dataset is formatted like: ngram TAB year TAB match_count TAB page_count TAB volume_count NEWLINE (note that the file is TAB delimited). For example, 2 sample lines in the dataset could be: analysis is often described 1991 10 1 1 analysis is often described 1992 30 2 1 where ‘analysis is often described’ is a 4-gram and line tells us that it occurred 10 times in the year 1991 in 1 book in the Google books sample, 30 times in the year 1992 and so on. Refer to the setup guidelines to see how to set the data as input to your MapReduce job. It a l s o provides a screenshot to configure the EMR cluster, which demonstrates how to access input data from some given bucket. Q 1. (50 points) Plot the frequency distribution for the occurrence counts of the k-grams (e.g., k=4), i.e., a plot where the x-axis is the occurrence count (say n), and y-axis is the number of 4-grams which occur n times. Occurrence count is the just the total number of times a particular k-gram has occurred over all the years in the sample. Hint: It will be easiest if you write a MapReduce job to pull out just the occurrence information from the dataset; download it to your local machine and then compute and plot the distribution again locally on your machine. Q 2. (40 points) Write a MapReduce job to output all 2-grams using the same dataset. Store the output (i.e. the 2-gram dataset) in a bucket in S3. Q 3. (10 points) Run the same code for Q 1 on the dataset generated above, and similarly plot the frequency distribution for the occurrence counts of the 2-grams this time. i.e. a plot where the x-axis is the occurrence count (say n), and y-axis is the number of 2- grams which occur n times. Homework Deliverables: For Q 1: Submit the mapper and reducer files in addition to the plot. For Q 2: Submit the mapper and reducer files for computing the 2-grams from the dataset. For Q 3: Submit your plot.