To do:

- try running code on emulab; do it without throttling at first, and
  then try it with throttling.  Probably want to watch fd's to make
  sure that when sharing a link that things are working.
- separate out loop event handling stuff from particular handlers we are
  interested in
  - without creating events
  - by using RVR-style events

Key points:

1) In principle, my scheduler needs to handle timeout events and I/O events,
   and I particularly want to make sure that I get to the timeout events
   quickly.  However, in practice I won't have any of these events.  So:

   I will need to be able to handle overload conditions; that is, trying to
   write more data than is possible.  I also need to handle special-purpose
   (i.e. non-I/O) events, for example due to reconfigurations

5) One big change to make is to just have the underlying link used for
   packet counting, and not for blocking.  That is, excising the
   "simulation" code for bandwidth throttling should significantly simplify
   things.

2) I'd like the event system to not be tied to the component system.  This
   should keep both pieces simpler.  I think the component system in the
   current arrangement is pretty separate, but the loop is not.

   One way to do this is to generate an input event whenever a packet is
   received, and an output event whenever I want to send one.  This allows
   the loop to not have to invoke a component inport, but instead just
   generate an event that will be handled by that component inport when
   processed.  For output packets, we'll basically always be queuing the
   packet, rather than forwarding them optimistically.

   I could try both optimistic and queued versions, and see how the
   performance differs; for the opt. recieve, I just need the generic
   handler function to be called immediately, rather than dispatching the
   event.

   I would similarly generate events for closing a socket, or getting an
   error.  This would separate the error handling code from the normal
   code.  It might lose context though ...

   If I do this, I think I definitely want to use some version of rvr's
   event implementation: each event will have a next pointer, so that it can
   be both queued on a channel, and be on the free list of events for a
   given event type.  I need to figure out the difference between event_data
   and event_data_src.

3) We want to apply back-pressure so that if I have queued events on an
   output fd, I don't bother to read from its input fd.  On the other hand,
   the input_check functions will not read when there is data if there are
   events outstanding.  This will result in blocks all the way back to the
   sender, which we don't want; we prefer to drop packets in the middle of
   the network.  Want a different policy based on queue length, for
   example.  To do this, we'd have to change the channel abstraction to have
   a "pending" flag, that indicates the events conceptually waiting on that
   channel.

4) One good idea is to associate the last partial packet written with the
   activity, so that when the mask check comes through, we just write that
   packet and complete.  In unblocking the channel (as a result), the events
   on that channel become eligible and are subsequently processed.

   This illustrates the benefit of having a single global event queue, but
   noting the number of events we have per channel.  This way, we can queue
   the packets in the order they come in (sharing between different
   streams), but still drain for a particular channel if the other is
   blocked.

---------------------------------------------------------------------------

Notes on the mmsched event system:

1) Each packet that is sent generates an event that is added to the global
   event queue.  In turn the event points to a dst and a release channel;
   the former is the "destination" of the event (e.g. a file descriptor of
   an outbound socket), and the latter is the source.  Processing the event
   occurs by invoking the handler associated with the channel, with the
   event as an argument.

2) There is a notion of an "activity" that is responsible for processing
   events.  As far as I can tell, this is basically an encapsulation of a
   file descriptor and a count of events waiting to be sent on it: It
   includes two functions for setting and checking the select mask, and a
   count of the outstanding events.  There are special purpose functions for
   creating tcp_input, tcp_output, and tcp_server activities.  This is a
   nice arrangement because a) activities are independent of whether we're
   using TCP or UDP, and b) they should be able to deal with simplex and
   duplex (the former having two activities sharing the same fd).  My
   current system would require one handler deal with both input and output
   activity on a fd.

   Particular activities have an associated event channel input, which is
   written to by the event dispatcher.

3) Another key idea is that when I try to send an event, it increments a
   count on the "release" channel.  This effectively applies back-pressure
   in that channels without outstanding events are preferred over ones that
   have them.  This way, if I have a lot of data queued on one output port,
   I won't read any data from its corresponding input port, reading from
   somewhere else instead.  I don't do this now.

4) It appears that the tcp would-block stuff isn't going to work.  In
   particular, when an tcp input handler is called, the return code is not
   checked if the full packet isn't written.  It appears that it should
   queue the rest of the unwritten packet to the handler env.  When the
   select_mask comes out as true, then, it will write that packet and
   complete.  Future packets will be on the ev_channel, and will then be
   eligible for sending.