6. Practice Problems and Solutions

Problem 1

1. Distinguish between system-level benchmarks and component-level benchmarks.

2. Name any three subsystems of a computer organization that can be tested with component-level benchmarks.

3. What level of benchmarks would you use to measure the performance of multi-layered cache in the memory hierarchy of your computer?

4. What type of benchmarks would you use to evaluate a massively parallel system built with multiple-context processors and a 3-D torus architecture?

5. Describe two drawbacks of application benchmarks.

Problem 1 Solutions:

1. System-level benchmarks measure the overall performance of a machine, whereas component-level benchmarks measure the performance of a subsystem.

2. • CPU
   • Memory
   • I/O Devices

3. Component-level benchmarks

4. Application benchmarks

5. • Large and difficult to execute
   • Subject to the developer's interpretation of real usage.

Problem 2

PR233 Cyrix 6x86MX processor	52.3
Pentium II processor 233 MHz	51.8

1. What conclusion can you draw about the relative performance of these two processors?

2. Name four factors that must be kept constant during this benchmarking process.

1. Winstone 97 benchmark is a system-level application benchmark that measures a PC's overall performance when running Windows-based 32-bit business applications. Thus a PC with Cyrix 6x86MX processor will run those applications faster than a PC with Pentium II 233 MHz processor.

2. • Version of the compiler
   • Version of the OS
   • Size and speed of cache
   • Amount of Main Memory

Problem 3

1. Synthetic benchmarks predict performance of real programs.

2. Factors influencing the usefulness of a benchmark remain valid indefinitely.

3. Improving the performance of an old benchmark by optimizing the compiler is one way of updating the benchmark.

4. Application benchmarks reflect the typical nature of the workload better than synthetic benchmarks.

5. Synthetic benchmarks are shorter and easier to run than application benchmarks, and thus they are better performance metrics.

1. false - Synthetic benchmarks simulate real programs, and do not reflect the actual behavior of real programs.

2. false - Improvements in technology lead to the necessity of revising old and obsolete benchmarks.

3. fasle - Compiler optimization can change what a benchmark measures producing inaccurate benchmark results.

4. true - Application benchmarks are real programs.

5. false - They simulate real programs.

Problem 4

1. Which benchmark results are more reliable? Why?

2. Suggest a reason why SYSmark for Windows 3.1* can predict the performance of system A consistently, but cannot give the true performance of system B.

3. If the benchmark performance score of system B is 40.35 for 32-bit benchmark, and 32.8 for 16-bit benchmark respectively, and the performance differences of system A and system B are 1.91 and 1.68 respectively, compute the benchmark performance scores of system A for both benchmarks.

1. SYSmark/NT* benchmark results are more reliable because it is more up-to-date than SYSmark for Windows 3.1*, and thus takes into account of sophisticated capabilities of the two systems that SYSmark for Windows 3.1* might have overlooked.

2. System A may be a 16-bit architecture machine, and thus a 16-bit benchmark, SYSmark for Windows 3.1*, can represent the workload for system A more consistently than it does for system B. System B may be a 32-bit architecture machine, and thus a 16-bit benchmark cannot test the performance effectively.

3. SYSmark/NT* performance score of A = 40.35 - 1.91 = 38.44

   SYSmark Windows 3.1* performance score of A = 32.8 - 1.68 = 31.12

Problem 5

• throughput
• speed
• cracking
• compiler efficiency
• simulations of real applications

Supply the missing words in the following sentences from the above list:

1. System-level benchmarks can be used to predict the _________________________ of a PC running real applications with Cyrix 6x86MX processor.

2. The most important characteristic of a reliable benchmark is its ability to resist _________________________.

3. _________________________ and _________________________ are two performance metrics that a typical application benchmark measures.

4. Improving the _________________________ of a system while other variables are kept constant during benchmarking will not yield the accurate result.

5. Application benchmarks are more reliable than synthetic benchmarks because they are not _________________________.

1. speed

2. cracking

3. Throughput, speed

4. compiler efficiency

5. simulations of real applications