This section describes the lessons learned from the Peer-to-Peer style Parallel I/O work described herein.
Multiple processors lessen impact
Peer-to-Peer can not provide the complete aggregate bandwidth of the disks due to arbitration of shared resources. The message passing and other library code can degrade the client application. On single processor systems, the scheduling can be a major problem, in that Jovian-2 does not have any future knowledge of I/O requests. Without this knowledge, the scheduler can cause context switching between the client and server threads at inopportune times. This is worsened by potential message backups behind the first. Multiple processors allow the application code to not be switched out, and the server thread can be alive when a request arrives instead of waiting to be swapped back in.
Fast communication is essential
A very fast interconnect, such as ATM or a high speed switch is absolutely necessary for attaining good performance. In a peer-to-peer configuration, when all nodes are reading/writing to each other simultaneously, the communication between nodes must be fast enough and the call startup latency must be low enough to make reading the disk over the interconnect feasible.