The Cray X1
John Kleint
UMD CMSC 714
[Arrow keys or mouse to lower right corner to navigate; press t for printable version]
Cray X1
- Combines vector and MPP designs
- Collection of SMP nodes linked by custom interconnect
- Distributed shared memory, but not cache coherent
- Each node contains 4 "Multi-Streaming Processosrs" (MSPs).
- Each MSP contains 4 vector processing "cores" (SSPs)
What's all this about vectors?
With conventional scalar processing, the processor fetches and executes several instructions for every element in the vector. It must increment the loop counter, test the loop bound, jump back to the start of the loop, and perform the next operation.
In vector processing, operands are operated on in batches. On the Cray X1, a single vector load instruction will load 64 elements into a 64-element-long vector and a single vector add instruction will add 64 pairs of elements, one pair per cycle. This reduces overhead and, more importantly, allows the functional units to be deeply pipelined for maximum cycle speed and maximum throughput.
So why doesn't everybody use vector processing? The answer is memory bandwidth: if the processor is to sustain one three-operand operation per clock (c[i] = a[i] + b[i]), the memory must be able to transfer three operands (24 bytes) per clock. Nowadays processors are an order of magnitude faster than memory, so to keep a vector processor fed, you'd need an order of magnitude more memory banks operating in tandem. More memory banks + more wires to connect them = more expensive. Another reason is that your code has to be expressible in terms of vector operations to benefit from vector processing, and most "desktop" software isn't.
There is one exception: "multimedia" applications. Modern PC architectures have adopted short vector processing, operating on two or four operands at a time, to tackle repetetive operations in video, graphics, and audio. This takes the form of Intel's SSE (Streaming SIMD Extensions) and Apple's Altivec. They rely on data reuse and fast caches to keep the processor fed with data.
Cray X1 Node
- 4 MSP processors, 12.8 GFLOPS each (18 on X1E)
- 8-32 GB 400 MHz RDRAM, 26 GB/s/MSP + 100 GB/s for remote accesses
X1 Physical
- Each MSP is an 8-chip MCM (2 MSPs on one MCM in X1E)
- Spray Evaporative Cooling!
- 65 kW of power per 128-proc X1E cabinet
Single Processor Performance (DGEMM)
Single MSP performance is 2.5x better than nearest competitor
X1 Node Summary
- 4 Multi-Streaming Processors per node share 32 GB RAM, interconnect, I/O
- Each MSP contains 4 Single-Streaming Processors that share 2MB L2 cache
- Each SSP contains 2 800 MHz vector units, 1 400 MHz scalar unit
Aggregate Memory Bandwidth
Pleasantly parallel, ~20 GB/s per node out of 26 GB/s peak
MPI Swap Bandwidth
Pairs of processes exchanging data; 4 MSPs share ~25 GB/s network BW
NAS Parallel Benchmarks (MG)
MG solves dense linear system using multigrid method
ORNL Spectral Shallow Water Model
Single SMP node results, component of atmospheric modeling code
LANL Parallel Ocean Program
X1 keeps up with Earth Simulator