The discrete event simulator that we implemented has various kinds of events emulating the actual execution of a parallel job. The events are - raw computation, message sends and receives for barrier arrivals and releases, lock requests, grants and releases and memory requests and replies, context switch in and out. Each event at its termination schedules more events that is expected to follow.
The simulator is designed as a collection of event queues, one for each process that is present in the system. There is also a future event schedule, which places events destined to happen in the future, into the process event queues. All the queues are driven by a timer, which de-queues the most recent event from the queue, takes appropriate actions for the event (i.e. schedules some other events) and updates the current time.
All messages are assumed to be atomic, i.e. they cannot be split across a context switch. For ease of implementation, we handle the errant cases by terminating the message at the beginning of context switch.
The scheduling module has been made separate from the design of the rest of the simulator so that we can easily replace one scheduling mechanism with another. We implemented the co-scheduling technique where context switches across all processes occur at the same time.
We also implemented priority based implicit scheduling, where the context switch
decisions are made independently by each processor. The scheduling
in each processor is priority based. We used the dynamic priority assignment
scheme as used for 4.3 BSD Unix [4]. In this, the system clock
ticks 100 times a second, and the priorities of the processes are
updated on every 
tick. The CPU usage of a process is increased
by one each time the process is found to be using the CPU. The
CPU usage is decayed by half the current value once each second
when it did not use the CPU. Here, processes are switched out
when its time slice expires or it is still busy waiting with the
CPU with its spin-wait phase over.
To contrast the utility of the priority based scheduler over a simple round-robin scheduler, we also implemented a round-robin local scheduling policy, where a new process for a given processor is picked from the ready to run queue in a round robin fashion. The decision to switch out a process is made as described for a priority based local scheduler.