2. Conventions for distributed programs

This section describes some conventions regarding addressing in distributed services, directory structure of sesf modules, and shell commands and options.

Location ids for distributed services

A distributed service spans multiple locations. In real-life, locations are identified in various ways, eg, MAC addresses, IP addresses and TCP/UDP port numbers, and URLS. For notational simplicity, our programs use integers to identify locations, specifically, ids 0, 1, …, N-1, where N is the number of possible locations.

A distributed service is implemented by a distributed system, which consists of multiple systems, one at each location of the service. We refer to the system at a location as a node of the distributed system. Each node has at least the following two attributes: the number of nodes in the distributed system (ie, N), and the id of its location in the service (ie, an integer in 0..N-1). Here is an example of a distributed system AB of N nodes, each executing a program ab, that implement a distributed service X spanning N locations.

The distributed service X implemented by the distributed system AB may itself be used by another distributed system CD to implement another distributed service Y, with each node in CD executing a program cd.

In the example above, services X and Y have the same number of locations, and the Y-id at a location is the same as the X-id at that location. But this need not be the case, as illustrated below.

Directory structure of modules

The root directory for sesf modules contains the following:

• util: directory with module files service.py and pyro.py, containing utility functions.
• start_nodes.py: utility script for starting the nodes of a distributed system.
• “service directories”, one for each service in sesf (eg, rwlock, msgtransfer, distlock2).

The service directory for a distributed service generally contains the following:

• service.ppy: pseudo-Python module file with a class Service, which is the abstract service program.
• servicetester.ppy: pseudo-Python module file with a class ServiceTester, which is the abstract servicetester program.
• service_parts.py: module file with a class ServiceParts, which defines (in Python) the various parts of the pseudo-Python class Service and ServiceTester. (Output conditions in the abstract service and servicetester are transformed into randomized code.)
• service.py: module file with a class Service, which is the concrete version of the abstract service program. It accepts incoming RPC calls (from users of the service) iff they are valid and returns arbitrary valid responses.
• servicetester.py: module file with a class ServiceTester, which is the concrete version of the abstract servicetester program. It issues arbitrary valid RPC calls to an implementation and checks the validity of the implementation’s responses.
• user_node.py: module file with a class UserNode, a generic user node of the service. This can drive any implementation or service node.
• test_node.py: module file with a class TestNode, an rpc wrapper for any implementation node. Routes incoming RPC calls (from servicetester) to its implementation node.
• service_node.py: module file with a class ServiceNode, an rpc router that routes incoming calls from a user node to RPC calls to a service program instance.
• one or more “implementation directories”.

An implementation directory has modules defining an implementation, ie, a distributed system that implements the service. It generally contains the following:

• node.py: module file with a class Node, an instance of which is a node of the distributed system that implements the service. This is the only module needed to run the implementation without testing (see commands below).
• checker.py: module file with a class Checker, which checks global assertions specific to this implementation, ie, the distributed system of nodes. Used by the servicetester when testing this implementation.

Note

For a single-location service (ie, N=1), the service directory is as above except that “node” is replaced by “imp” in module and class names. For example, module file node.py with class Node becomes module file imp.py with class Imp. Similarly, test_node.py becomes test_imp.py, and it would contain class TestImp instead of TestNode.

Commands for running distributed systems

The command examples below are for distributed systems and services spanning three locations: 0, 1, 2. The commands are executed in the sesf root directory. Assume the following directories:

• Service directory msgtransfer containing an implementation directory imp_0, whose nodes use TCP (and no sesf service).
• Service directory xyz containing an implementation directory imp_x, whose implementation uses the msgtransfer service.

A Pyro4 nameserver process has to be running in order to execute commands involving the service program or the servicetester program. Run python -m Pyro4.naming & to start a Pyro4 nameserver.

Running implementation msgtransfer.imp_0

Run the following command to create a process that starts an instance of UserNode of module msgtransfer.user_node, which then starts an instance of Node of module msgtransfer.imp_0.node with arguments 3 (number of nodes) and 0 (node id).

• python msgtransfer/user_node.py 3 0 --use msgtransfer.imp_0.node 3 0

To complete the distributed system that implements service msgtransfer for three locations, the other two peer processes with imp_0 nodes have to be started. Run the following additional commands:

• python msgtransfer/user_node.py 3 1 --use msgtransfer.imp_0.node 3 1
• python msgtransfer/user_node.py 3 2 --use msgtransfer.imp_0.node 3 2

The commands can be executed in any order. Instead of running the three commands above (in different shells), you can run the following one commmand (useful when the number of nodes is large):

• python start_nodes.py 3 msgtransfer/user_node.py --use msgtransfer.imp_0.node

If either user_node or imp_0.node has any additional arguments, they can be entered after its N and j arguments, eg,

• python msgtransfer/user_node.py 3 1 arg1 --use msgtransfer.imp_0.node 3 1 arg2
• python start_nodes.py 3 msgtransfer/user_node.py arg1 --use msgtransfer.imp_0.node arg2

Of course, you can substitute user_node.py by any node program that makes use of msgtransfer.imp_0.imp.Imp.

Running implementation xyz.imp_x using msgtransfer.imp_0

Because implementation imp_x of service xyz makes use of the msgtransfer service, we can run the following (assuming the xyz nodes do not have any additional arguments):

• python xyz/user_node.py 3 1 --use xyz.imp_x.node 3 1 --use msgtransfer.imp_0.node 3 1

This starts a process that instantiates xyz.user_node.UserNode, which instantiates xyz.imp_x.Node, which instantiates msgtransfer.imp_0.Node. Starting the corresponding peer processes will then start the distributed system shown above. Or run the following:

• python start_nodes.py 3 xyz/user_node.py --use xyz.imp_x.node --use msgtransfer.imp_0.node

Running msgtransfer user_nodes over msgtransfer service

Start a Pyro4 nameserver if one is not already running (python -m Pyro4.naming &).

Start the msgtransfer service model for 3 locations:

• python msgtransfer/service.py 3

Run the following commands to have the msgtransfer users use the msgtransfer service (instead of a msgtransfer implementation):

• python msgtransfer/user_node.py 3 0 --use msgtransfer.service_node 3 0
• python msgtransfer/user_node.py 3 1 --use msgtransfer.service_node 3 1
• python msgtransfer/user_node.py 3 2 --use msgtransfer.service_node 3 2

Or instead of the above three commmands, run

• python start_nodes.py 3 msgtransfer/user_node.py --use msgtransfer.service_node

Of course, the msgtransfer user application can be replaced by any application that uses the msgtransfer service, for example, xyz user_nodes over xyz imp_x nodes.

Running msgtransfer servicetester over implementation msgtransfer.imp_0

Start a Pyro4 nameserver if one is not already running (python -m Pyro4.naming &).

Start the imp_0 nodes with test_node wrappers:

• python msgtransfer/test_node.py 3 0 --use msgtransfer.imp_0.node 3 0
• python msgtransfer/test_node.py 3 1 --use msgtransfer.imp_0.node 3 1
• python msgtransfer/test_node.py 3 2 --use msgtransfer.imp_0.node 3 2

Or run the following command instead of the above three commands:

• python start_nodes.py 3 msgtransfer/test_node.py --use msgtransfer.imp_0.node

Start the service tester:

• python msgtransfer.servicetester.py 3

Or start the service tester with imp_0 checker (if available):

• python msgtransfer.servicetester.py 3 --checker msgtransfer.imp_0.checker

Running xyz servicetester on implementation xyz.imp_x using msgtransfer service

Start a Pyro4 nameserver if one is not already running (python -m Pyro4.naming &).

Start msgtransfer service:

• python msgtransfer/service.py 3

Start 3 processes, each running xyz test_node over xyz.imp_x node over msgtransfer service_node:

• python start_nodes.py 3 xyz/test_node.py --use xyz.imp_x.node --use msgtransfer.service_node

Start the xyz servicetester:

• python xyz/servicetester.py 3
• Or python xyz/servicetester.py 3 --checker xyz.imp_x.checker