Benchmarks

• Write is a simple program that generates data, and writes it out to the disks it is striped over in the Jovian-2 file-system.

• Read is the dual of Write, in that it reads from a striped Jovian-2 file.

For each of these benchmarks, one of three operation modes can be used. First is Local mode, where all data that a given client thread needs is located on the disks local to where the thread is running. One-to-many mode is basically an extreme of client-server processing where there is a single client thread making requests for data, and all other nodes have only a server thread, as shown in Figure 1(a). The last mode is Many-to-many or peer-to-peer processing where all client threads are colocated with an active server thread, as shown in Figure 1(b).

For these experiments, we should have 300MB of data being read or written by every node to insure we are avoiding disk cache effects. Due to the setup of our alpha farm, too little disk space is available to make this possible, so we are using only 70MB of data. Note that these tests are scaled to insure every client reader or writer will be processing 70MB. To maximize bandwidth we also want to insure that all the disks are busy as much as possible. In the benchmarks, the size of an individual request becomes stripe_size x number_of_disks_used. In between Write and Read we flush the disk cache by scanning a dummy 300MB file.

Table 3 summarizes the results of running these benchmarks on various sized configurations ranging from 1 node to 6 nodes. For the One-to-many mode, as the number of server nodes involved increases, the I/O rate increases nicely until we saturate the interconnect. Read underperforms compared to Write because there is an optimization in Jovian-2, similar to write behind, in that as soon as a block of data to write arrives at the server, an acknowledgement is immediately returned, even though it has not yet hit the disk. Read on the other hand, cannot use a similar technique, and must wait until the data is actually read from the local disk. The Many-to-many mode results show how the contention for the shared interconnect and outstanding requests from all the clients at each of the servers, causes a reduction in the overall I/O rate as the number of nodes increases. If this were not the case, we would see linear speedup with the number of peers.

 
Table 3: Benchmark I/O rates, in MB/sec . uses Ethernet interconnect. uses Atm without MPI point-to-point optimizations. cx-sy means x clients and y server only nodes. pp-x means x nodes total and each is a client and a server.