.. highlight:: python
   :linenothreshold: 50


#######################################
Conventions for distributed programs
#######################################


This section describes some conventions regarding 
addressing in distributed services,
directory structure of sesf modules,
and shell commands and options.

--------------------------------------------------
Location ids for distributed services
--------------------------------------------------

A distributed service spans multiple locations.
In real-life, locations are identified in various ways,
eg, MAC addresses, IP addresses and TCP/UDP port numbers, and URLS.
For notational simplicity, our programs use *integers* to identify locations,
specifically, ids 0, 1, ..., N-1,
where N is the number of possible locations.

A distributed service is implemented by a distributed system,
which consists of multiple systems, one at each location of the service.
We refer to the system at a location as a **node** of the distributed system.
Each node has at least the following two attributes:
the *number of nodes* in the distributed system (ie, N),
and the *id of its location* in the service (ie, an integer in 0..N-1).
Here is an example of a distributed system *AB* of N nodes,
each executing a program *ab*,
that implement a distributed service *X* spanning N locations.

.. .. _conventions-AB:
.. image:: conventions-AB.svg
   :width: 670 px
   :align: center
   :alt: Distributed service X implemented by distributed system AB

The distributed service *X* implemented by the distributed system *AB*
may itself be used by another distributed system *CD* to implement another
distributed service *Y*,
with each node in *CD* executing a program *cd*.

.. .. _conventions-ABCD:
.. image:: conventions-ABCD.svg
   :width: 670 px
   :align: center
   :alt: Distributed service Y implemented by distributed system CD
         using distributed service X implemented by distributed system AB

In the example above,
services *X* and *Y* have the same number of locations,
and the *Y*\ -id at a location is the same as the *X*\ -id at that location.
But this need not be the case, as illustrated below.

.. .. _conventions-ABCD-mixed-ids:
.. image:: conventions-ABCD-mixed-ids.svg
   :width: 700 px
   :align: center
   :alt: Distributed service Y implemented by distributed system CD
         using distributed service X implemented by distributed system AB

--------------------------------------------------
Directory structure of modules
--------------------------------------------------

The root directory for sesf modules contains the following:

- ``util``: directory with module files ``service.py`` and ``pyro.py``,
  containing utility functions.
- ``start_nodes.py``: utility script for starting the nodes of 
  a distributed system.
- "service directories", one for each service in sesf
  (eg, ``rwlock``, ``msgtransfer``, ``distlock2``).

The **service directory** for a distributed service generally contains
the following:

- ``service.ppy``: pseudo-Python module file with a class ``Service``,
  which is the abstract service program.

- ``servicetester.ppy``: pseudo-Python module file with a class
  ``ServiceTester``, which is the abstract servicetester program.

- ``service_parts.py``: module file with a class ``ServiceParts``,
  which defines (in Python) the various parts of the 
  pseudo-Python class ``Service`` and ``ServiceTester``.
  (Output conditions in the abstract service and servicetester 
  are transformed into randomized code.)

- ``service.py``: module file with a class ``Service``,
  which is the concrete version of the abstract service program.
  It accepts incoming RPC calls (from users of the service) iff 
  they are valid and returns arbitrary valid responses.

- ``servicetester.py``: module file with a class ``ServiceTester``,
  which is the concrete version of the abstract servicetester program.
  It issues arbitrary valid RPC calls to an implementation and
  checks the validity of the implementation's responses.

- ``user_node.py``: module file with a class ``UserNode``,
  a generic user node of the service. This can drive any implementation
  or service node.

- ``test_node.py``: module file with a class ``TestNode``, an rpc wrapper
  for any implementation node. Routes incoming RPC calls
  (from servicetester) to its implementation node.

- ``service_node.py``: module file with a class ``ServiceNode``, 
  an rpc router that routes incoming calls from a user node
  to RPC calls to a service program instance.

- one or more "implementation directories".

An **implementation directory** has modules defining an implementation,
ie, a distributed system that implements the service.
It generally contains the following:

- ``node.py``: module file with a class ``Node``, an instance of which
  is a node of the distributed system that implements the service.
  **This is the only module needed to run the implementation without
  testing** (see commands below).

- ``checker.py``: module file with a class ``Checker``, which checks
  global assertions specific to this implementation, ie, the distributed
  system of nodes. Used by the servicetester when testing this
  implementation.

.. note::

  For a single-location service (ie, N=1), the service directory is as
  above except that "node" is replaced by "imp" in module and class names.
  For example, module file ``node.py`` with class ``Node`` becomes
  module file ``imp.py`` with class ``Imp``. Similarly, ``test_node.py``
  becomes ``test_imp.py``, and it would contain class ``TestImp``
  instead of ``TestNode``.


--------------------------------------------------
Commands for running distributed systems
--------------------------------------------------

The command examples below are for distributed systems and services 
spanning three locations: 0, 1, 2.
The commands are executed in the sesf root directory.
Assume the following directories:

- Service directory ``msgtransfer`` containing an implementation directory
  ``imp_0``, whose nodes use TCP (and no sesf service).
- Service directory ``xyz`` containing an implementation directory
  ``imp_x``, whose implementation uses the ``msgtransfer`` service.

A **Pyro4 nameserver** process has to be running in order to 
execute commands involving the service program or 
the servicetester program.
Run ``python -m Pyro4.naming &`` to start a Pyro4 nameserver.



Running implementation msgtransfer.imp_0
---------------------------------------------------

.. image:: msgtransfer-imp_0-3.svg
   :width: 610 px
   :align: center

Run the following command to create a process that 
starts an instance of ``UserNode`` of module ``msgtransfer.user_node``,
which then starts an instance of ``Node`` of module 
``msgtransfer.imp_0.node`` 
with arguments ``3`` (number of nodes) and ``0`` (node id).

- ``python msgtransfer/user_node.py 3 0 --use msgtransfer.imp_0.node 3 0``

To complete the distributed system that implements 
service ``msgtransfer`` for three locations,
the other two peer processes with ``imp_0`` nodes have to be started.
Run the following additional commands:

- ``python msgtransfer/user_node.py 3 1 --use msgtransfer.imp_0.node 3 1``
- ``python msgtransfer/user_node.py 3 2 --use msgtransfer.imp_0.node 3 2``

The commands can be executed in any order.
Instead of running the three commands above (in different shells),
you can run the following one commmand
(useful when the number of nodes is large):

- ``python start_nodes.py 3 msgtransfer/user_node.py --use msgtransfer.imp_0.node``

If either ``user_node`` or ``imp_0.node`` has any additional arguments, 
they can be entered after its *N* and *j* arguments,
eg, 

- ``python msgtransfer/user_node.py 3 1 arg1 --use msgtransfer.imp_0.node 3 1 arg2``
- ``python start_nodes.py 3 msgtransfer/user_node.py arg1 --use msgtransfer.imp_0.node arg2``


Of course, you can substitute ``user_node.py`` by any node program
that makes use of ``msgtransfer.imp_0.imp.Imp``.


Running implementation xyz.imp_x using msgtransfer.imp_0
--------------------------------------------------------------------

.. image:: xyz-msgtransfer-imp_0-3.svg
   :width: 610 px
   :align: center


Because implementation ``imp_x`` of service ``xyz`` makes use of
the msgtransfer service, we can run the following 
(assuming the `xyz` nodes do not have any additional arguments):

- ``python xyz/user_node.py 3 1 --use xyz.imp_x.node 3 1 --use msgtransfer.imp_0.node 3 1`` 

This starts a process that instantiates `xyz.user_node.UserNode`,
which instantiates `xyz.imp_x.Node`,
which instantiates `msgtransfer.imp_0.Node`.
Starting the corresponding peer processes will then start the distributed
system shown above.
Or run the following:

- ``python start_nodes.py 3 xyz/user_node.py --use xyz.imp_x.node --use msgtransfer.imp_0.node``



Running msgtransfer user_nodes over msgtransfer service
--------------------------------------------------------------------

.. image:: msgtransfer-user+service-3.svg
   :width: 630 px
   :align: center

Start a Pyro4 nameserver if one is not already running
(``python -m Pyro4.naming &``).

Start the msgtransfer service model for 3 locations:

- ``python msgtransfer/service.py 3``

Run the following commands to have the msgtransfer users 
use the msgtransfer service 
(instead of a msgtransfer implementation):

- ``python msgtransfer/user_node.py 3 0 --use msgtransfer.service_node 3 0``
- ``python msgtransfer/user_node.py 3 1 --use msgtransfer.service_node 3 1``
- ``python msgtransfer/user_node.py 3 2 --use msgtransfer.service_node 3 2``

Or instead of the above three commmands, run

- ``python start_nodes.py 3 msgtransfer/user_node.py --use msgtransfer.service_node``


Of course, the msgtransfer user application can be replaced by any 
application that uses the msgtransfer service, for example,
xyz user_nodes over xyz imp_x nodes.

Running msgtransfer servicetester over implementation msgtransfer.imp_0
-------------------------------------------------------------------------

.. image:: msgtransfer-servicetester-imp_0-3.svg
   :width: 630 px
   :align: center

Start a Pyro4 nameserver if one is not already running
(``python -m Pyro4.naming &``).

Start the imp_0 nodes with test_node wrappers:

- ``python msgtransfer/test_node.py 3 0 --use msgtransfer.imp_0.node 3 0`` 
- ``python msgtransfer/test_node.py 3 1 --use msgtransfer.imp_0.node 3 1`` 
- ``python msgtransfer/test_node.py 3 2 --use msgtransfer.imp_0.node 3 2`` 

Or run the following command instead of the above three commands:

- ``python start_nodes.py 3 msgtransfer/test_node.py --use msgtransfer.imp_0.node``

Start the service tester:

- ``python msgtransfer.servicetester.py 3``

Or start the service tester with imp_0 checker (if available):

- ``python msgtransfer.servicetester.py 3 --checker msgtransfer.imp_0.checker``

|

Running xyz servicetester on implementation xyz.imp_x using msgtransfer service
--------------------------------------------------------------------------------

Start a Pyro4 nameserver if one is not already running
(``python -m Pyro4.naming &``).

Start msgtransfer service:

- ``python msgtransfer/service.py 3``  

Start 3 processes, each running xyz test_node over xyz.imp_x node
over msgtransfer service_node:

- ``python start_nodes.py 3 xyz/test_node.py --use xyz.imp_x.node --use msgtransfer.service_node``

Start the xyz servicetester:

- ``python xyz/servicetester.py 3``
- Or ``python xyz/servicetester.py 3 --checker xyz.imp_x.checker``