A well known problem is the hierarchy that exists in memory systems. Memory is used here to mean the entire range from registers in a CPU, to on-chip instruction and data caches, to main memory, to disks and tapes. The classic trade-offs being price and performance. If price were not an issue, we could use a large number of registers, or use all main memory, etc. If speed did not matter, we could store everything on tape because of its cost-effectiveness. Since neither of these two scenarios are practical, we must find ways to make the speed problem less severe to impact running times less. For the purposes of this paper, the disk to memory boundary will only be considered.
Large data sets, as used for global land cover analysis, etc., are placing greater demands on the I/O subsystem. One approach is to increase the raw performance of the hardware. This works well, but the theoretical boundary from a mechanical device is being approached, while the need for more I/O bandwidth continues. More creative software based solutions are needed.
One possible solution is to use Disk Striping to overcome the limited bandwidth of the disks. This way, data can be spread out over multiple disks, in a round-robin fashion, and if large enough disk requests are read, the multiple disks can all act asynchronously in parallel to satisfy the request. The net result could be as good as the aggregate bandwidth of the multiple disks. This too can be approached in a hardware only, software only or hybrid fashion. The software only solution will be considered here.