Project Descriptions

Project 1: Getting started.

String operations and files. Write a program that reads paragraphs from several text files (those that end with .txt extension), counts the number of characters in each paragraph, and writes the counts to a .counts file, one for each text file. The program takes a folder name as a command-line argument.

Project 2: Communication and sockets.

Write two programs. The first program sends a series of strings via sockets to the second program. The second program performs strlen() on each string and sends the results back. The first program writes these results into a file.

Project 3: Multi-threaded programming.

Develop a multi-threaded program where each thread does the following operation. It reads paragraphs from a given text file, counts the number of characters in each paragraph, and writes the counts to a .counts file. Each thread has its own input text and output .counts file so there are no conflicts.

Project 4: Locking.

Develop a multi-threaded program. First the program reads paragraphs from a given set of text files and puts this information in a queue. Then each thread does the following operation. It picks an entry from the queue, marks it as active, counts the number of characters in the paragraph, and writes the counts to a .counts file. There is one .counts file per text file so multiple threads will want to access it. The queue is also shared; multiple threads may want to modify it at the same time.

Project 5: Putting it all together.

This project implements the fundamental code needed for cloud computing. It may be customized to any distributed computation task. Several nodes are part of this cloud. To simplify implementation, the cloud has only one registration node; however, it has an arbitrary number of compute nodes that perform actual computations, and job management (distribution and results collection) nodes. The registration node serves to coordinate the initial handshake between compute nodes and job management nodes. The job management nodes send out jobs to the compute nodes; they also collect results and assemble them according to job constraints.
Again, to simplify implementation, the registration node and all job management nodes will reside on one physical machine. The compute nodes may reside on any number of machine(s).

Each node type behaves as follows:

1. Registration node. This node is launched explicitly via a command. After it is launched, it opens a new socket port for listening, writes this port and IP address on the screen for other nodes to use when registering. It also initializes a job queue of pending jobs. The queue contains records (status: active/pending, input record position, input record ID, output record position, output record ID).
   The registration node then waits in a loop for new compute nodes to register, listening on its port. When a compute node connects, the registration node creates a new job management node (implemented as a thread) to service that client, opens a new port for that client, and sends the port number to the client via its socket. And then goes back in its waiting loop.

2. Compute node. This node is also launched explicitly via a command. This can be done on any machine. The registration node's listening port and IP address are supplied at the command line.
   The compute node receives a new port number from registration node, closes the old port used for registration, opens the new port to connect to its dedicated job node asking for a job, gets the job, and performs the operation. Sends results back to job via the new port number. The node then determines if it has time for another job. If yes, it gets another job by asking for it. Otherwise it sends a "no longer available" message and dies.

3. Job management node. The node is notified by a compute node that it wants work or it is no longer available. If the compute node wants work, the job node examines the job queue, picks the first pending job, marks it as active (this needs to be locked so that only one thread can modify it), and sends it to the client, via its new dedicated port. If there are no PENDING/ACTIVE jobs at all, the node dies. Otherwise it waits for a pending job or until all jobs are marked as DONE.
   If results are returned from the client, the job is marked as DONE in the queue. The job node locks the output resource so that it can modify it. It waits to obtain this lock if someone else has it. Then writes the output in its correct location, releases the resource. If client has indicated that it needs more work, the job node re-examines the queue (repeats the entire process).
4. In our implementation the task is to take a folder of text files, and perform an arbitrary operation on each paragraph of each file. We'll start with the "strlen" operation. The outputs must be stored in files, preserving the order of the paragraphs.

Project 6: Making it your own.

Here, the above project is customized to perform any arbitrary distributed computation/task. One example is searching in parallel in response to a query. Another is to fetch parts of a web page and put the results together. Yet another may involve reading from a hardware port, bus, or device.