Create a Web Application Platform for Automated Parallel Computing with Word Count Application

Assignment Questions (Submission Required, 50 points)
In this assignment, you will create a Web application platform that allows user to perform automated parallel computing that mimics the Hadoop process. A sample user interface is given in Figure 1 The Web application consists of the following components.
 GUI component and the connection to the other components. [10 points]
 NameNode: It splits the uploaded Data File into N subsets and provides a subset to each TaskTracker. [10 points]
 TaskTracker: It consists of Map function and a Reduce function.
 Map function: It converts data from subset of Data File into Key-value pairs, based on the functionality requirement. [10 points]
 Reduce function: It reduces the key-value pairs into one or smaller number of key-value pairs, based on the functionality requirement. [10 points]
 Combiner: It synchronizes and combines the results from all the Reduce functions and put the final results. [10 points]

In this assignment, you will implement the Word Count application. It counts the number of occurrences each word in the uploaded Data File. The components and the computation process is illustrated in Figure 2
Option 1: You implement the Web application as a service-oriented application. You implement Map, Reduce, and Combiner as Web services (either RESTful services or WSDL services). You make parallel calls to the services for each Map and Reduce, and you configure or assume that a new instance of service will be created for each call, and thus the tasks are done in parallel. The sample GUI given Figure 1 is for Option 1.

Option 2: You implement the Web application without calling Web services. You implement Map, Reduce, and Combiner as local components. You must use multithreading techniques (read Text Chapter 2) to run the Map and Reduce functions in parallel. In this option, a large Data file, with at least 5000 words must be used. The execution times for single thread execution and for multithread execution must be reported. A sample GUI for Option 2 is given Figure 3.

Option 3: This option is very different from options 1 and 2. You will use VIPLE to simulate the Hadoop process, instead of using a web application. Figure 4 and Figure 5 give a sample application, where string’s characters are counted. It is not a realistic application that improves the performance of computing. The main purpose of this example is to show how you can automatically start N threads to perform parallel computing, where N can be entered from the keyboard.

The key idea of generating N threads based on the input N is to use event-driven programming. The Custom Activity NameNode will generate an event output (circular output port in Figure 5), instead of a data output (triangular port in Figure 5). Once the event output port is used, it will trigger an event through the Custom Event activity in the Main Diagram every time it is executed. As can be seen in Figure 5, there is no computation inside the activity. Its purpose is to trigger a new thread.

In the Main diagram, the NameNode is placed in a loop, which will be called N times. Each time will trigger a new thread through the Custom Event activity. The main computations are performed in these parallel threads. The loop will be executed sequentially. Since we do not have much computation in the activity, it will be iterated and the parallel threads are generated without delay.

You tasks in the option are as follows. You can make use of the given code as your starting point.
 Write a service or a code activity to load the text in a text file into variable InputData. Use it to replace the Data activity that provided the Input data. [10 points]
 Write a service or a code activity to counts number of appearances for each word in the InputData. [10 points]
 In the main diagram, you must automatically create the threads based on the input N, as shown in Figures 4 and 5. [10 points]
 Write a code activity to save the result (the number of words) into a text file called FileForOutput.txt. [10 points]
 Make sure that your program can execute correctly even if the data length is not a multiple of N. In the given example program in Figure 4, this check is not performed to keep the program small. [10 points]