PYRO - Implementation

<previous | contents | next> | Pyro Manual

## 11. Pyro's Implementation

The purpose of this chapter is to provide some insight in the Pyro's implementation. However please keep in mind that it is quite tedious for me to update this chapter with each Pyro release, so I want to stress the fact that the source code is probably the best place to look if you want to find out specific details about Pyro's implementation.

Nevertheless this chapter is a nice starting point to explore the different parts that make up Pyro.

- The Pyro Principle - The basic techniques with which Pyro works
- `bin/*` - Command-line tools
- `__init__.py` - Pyro package initialization
- `configuration.py` - Configuration code
- `constants.py` - Global constants used troughout Pyro
- `core.py` - Core code
- `naming.py` - Name Server
- `protocol.py` - Protocol adapters
- `util.py` - Utility code
- `errors.py` - Exception definitions
- `Pyro.EventService` - Event Server package
- `Pyro.ext` - Extensions package

- Possible issues in current implementation - Please help

### The Pyro Principle

How does Pyro work??? Pyro uses two main principles to do what it does: method call interception (actually, attribute access interception) and marshalling.

Your code doesn't call a Pyro object directly. It calls a method on a proxy that acts just like the Pyro object it represents. The proxy intercepts the method call by using a special implementation of `__getattr__`, that passes the call to `__invokePYRO__`. In turn, that method forwards the call to the protocol adapter, that creates a message containing all the details of the call (using `pickle` to marshall all Python objects). The message is passed via the network to the Pyro daemon on the other side.

The daemon receives the message and unmarshalls the request. It now knows the objectID of the required Pyro objects, the desired member function to call, and its arguments. It looks up the Pyro Object in its table and calls the `Pyro_dyncall` method. This method is in `ObjBase` which is the base class of all Pyro objects. This method uses the appropriate `apply` call on itself to call the actual method. The result of the method is handed back to the Pyro daemon, that marshalls it, and passes it back over the network to the calling protocol adapter. The adapter on the client side unmarshalls the result data, hands it back to the proxy that called it, and then the `__invokePYRO__` method of the proxy returns the result data as if it was obtained using a regular method call on a local object!

This was Pyro in a nutshell. There are many more things to consider, such as transporting modules (mobile objects or even mobile agents) and oneway calls, but you have to read the source anyway to find all the gory details.

### The command-line tools (bin/*)

The command-line tools are all in fact very small Python scripts. They contain a few Python statements that start the actual tools, which are part of the Pyro package:

Tool | Implemented in | Notes
--- | --- | ---
es | Pyro.EventService.Server | start Event Server
nsc | Pyro.nsc | control Name Server
xnsc | Pyro.xnsc | Tkinter-GUI version of nsc
wxnsc | Pyro.wxnsc | WxPython/WxWindows version of nsc, with nice tree view of namespace
pyroc | Pyro.pyroc | Creates static proxies (.py source files) for pyro objects - hardly needed anymore
genguid | Pyro.util | create a GUID, entrypoint is `genguid_scripthelper()`
ns / rns | Pyro.naming | start Name Server, entrypoint is `main()`
nsd / esd | Pyro.ext.daemonizer | Unix initd daemon scripts for name server and event server
nssvc / essvc | Pyro.ext.NS_NtService and ES_NTService | Windows NT Service control scripts for NS and ES

### Pyro package initializer (Pyro/__init__.py)

Pyro is a Python package and as such is contained in the 'Pyro' directory. This directory contains the different module files that make up the Pyro system. One of them is the initialization file, `__init__.py`. It loads the configuration settings for Pyro (`Pyro.config.`* items). This way, whenever a module from the Pyro package is imported, it can use the config items right away.

Within the core package is also another package; `Pyro.EventService`. The event service is implemented here: the Event Server itself and the base classes for event publishers and listeners.

### Configuration (Pyro/configuration.py)

This module contains the logic that deals with Pyro's configuration. There is a `Config` class and a `ConfigReader` class. The `Config` class is the container for all configuration items. The instance is available as `Pyro.config` (created in the package initializer, see above). The configuration items are all attributes of this object. `Config` uses the `ConfigReader` to read Pyro's configuration file. It deals with defaults and environment settings too.

### Constants (Pyro/constants.py)

This module contains all globally used constants. By placing them in a distinct module that doesn't import any other modules (apart from util), we can also prevent import cycles between other Pyro modules. Things that are defined here are for example the Pyro version string, and the names for the Name Server and Event Server.

### Core library (Pyro/core.py)

This module contains all core logic of Pyro that is not tied to the network protocol that is used. This means that you will not find any TCP/IP socket specific code in the core module; the `protocol` module contains this. What the core module does contain is all stuff that realize the core functions of Pyro:

`ObjBase`
  Server-side object implementation base class or master class with the actual object as delegate. It supplies the methods for remote method invocation (`Pyro_dyncall`) and remote attribute access (`remote_getattr` etc.)
`PyroURI`
  Pyro Universal Resource Identifier. This class represents a Pyro URI (which consists of four parts: a protocol identifier, an IP address, a portnumber, and an object ID). `PyroURI` can be converted to and from a - human readable - string.
`DynamicProxy` and`DynamicProxyWithAttrs`
  These two classes are the dynamic Pyro proxies. They can be used by clients to invoke objects for which they have no precompiled proxy. The "WithAttrs" proxy is needed if you want to do direct attribute access on remote Pyro objects. The proxies have a special `__invokePYRO__` method that intercepts method calls to pass them via the protocol adapter to the remote Pyro object. Special care is taken to make proxy objects suitable for pickling so they can be transported trough the Pyro protocol across the network.
`getProxyForURI` and `getAttrProxyForURI`
  Helper functions to create proxies directly from a given Pyro URI (object).
`Daemon`
  The server-side Pyro daemon. Accepts and dispatches incoming Pyro method calls. It is derived from the low level `TCPServer` and contains the server end of a protocol adapter. The daemon passes incoming method calls (via its `handleRequest` method) to the protocol adapter that sorts out what to do exactly. The `connect` and `disconnect` methods are used to keep track of server side Pyro objects. The Daemon's `handleError` is used in case of a Pyro error, it processes the error and cleans things up.
Pyro client and Pyro server initialization code
  The `initClient` and `initServer` functions currently only write some friendly strings to the log. But the server init code first checks the availability of the `PYRO_STORAGE` directory.

### Name Server (Pyro/naming.py)

This module contains all logic that deals with object naming and locations. Pyro's Name Server is implemented here as well as various other things.

`NameServerLocator`
  Used to locate the Name Server by using various lookup methods (broadcast, direct lookup). Usually you call the `getNS` method and you get a Pyro Proxy for the Name Server back.
`NameServerProxy`
  A static proxy for the Name Server that takes care of the naming rules (global names, default groups). You get this back from the `NameServerLocator.getNS`.
`NameServer` and `PersistentNameServer`
  The actual pyro object that is the Name Server. It is created by some utility code that also connects it to a daemon. This code is called by the `ns` script that starts the Name Server.
Various classes for hierarchical naming support, they implement a tree structure.
  We have a (name,value) tuple `NameValue` and a `NamedTree` recursive data structure.
`BroadcastServer`
  The helper server that listens to UDP broadcast events from the locator, and replies with the URI of the Name Server object. This server's socket is joined with the regular Pyro Daemon that listens for incoming Pyro calls for the NS object.

### Protocol Adapters (Pyro/protocol.py)

The idea behind this module is to make it fairly straightforward to switch the underlying protocol that Pyro uses. Currently only one implementation is available: the native PYRO protocol that sits on top of TCP/IP. This module contains all TCP/IP socket related code.

### Utilities (Pyro/util.py)

This module defines various utility functions and classes. The most important ones are:

`LoggerBase`
  This is the abstract base class of the various logging classes, see the next two items. Don't use this directly.
`SystemLogger`
  An object of this class is directly available as `Pyro.util.Log` and this is Pyro's system logger that writes to Pyro's system logfile.
`UserLogger`
  An object of this class can be constructed to create a user logger that writes to the user logfile.
`ArgParser`
  A simple command line argument parser like the `getopt` function in ANSI C. Pyro's command line tools use this. It can parse command line options according to an option specification string. The `getOpt` member is used to read the results.
`getGUID`
  Generates a new GUID. This is essentially a big number that is unique in time and in space. Unique in time means that every moment you generate a new GUID it is different from the ones you generated before. Pyro does this by using a time stamp as part of the GUID. Unique in space means that when you generate a GUID on a different machine, it will be different from the GUID you generated elsewhere (even when you generate them exactly the same moment). Pyro does this by using the network address of your computer as part of the GUID, and the Python process id. The latter is necessary because multiple Python processes may be running on the same IP address. The 128-bit GUID is returned as a hexlified string of 32 characters.
`genguid_scripthelper`
  This helper function is used by the `genguid` command line utility, which prints a new GUID.
`getPyroTraceback`
  Gets the remote traceback from the exception that is caught on the client, augments this with the local traceback, and returns a list of traceback steps. Use this to find out which part of the remote code caused the exception.

### Exception definitions (Pyro/errors.py)

This module defines `Pyro.PyroError` and its derived exceptions. `PyroError` is the Pyro exception type that is used for problems within Pyro. User code should not use it! Also, this module defines the `PyroExceptionCapsule` class, which is used to represent any Python exception that has to be transported across the network, and raised on the other side (by invoking `raiseEx` on this object).

### Event Server package (Pyro/EventService)

This package contains the Event Server modules:

`Pyro/EventService/Clients.py`
  Here are the `Subscriber` and `Publisher` base classes defined that you can use to easily build publisher or listener clients for the ES.
`Pyro/EventService/Server.py`
  Contains the actual `EventService` Pyro object that is started by some utility code. This module is called by the `es` script that you can use to start the Event Server.

### Extensions package (Pyro/ext)

This package contains optional modules that provide certain extensions to the core Pyro library. It contains:

`Pyro/ext/remote.py`
  The easy-remoting module that is shown in the "quickstart" example.
`Pyro/ext/remote_nons.py`
  The easy-remoting module that doesn't use the Name Server, it is shown in the "quickstart-noNS" example.
`Pyro/ext/daemonizer.py`
  Utility classes to "daemonize" a Pyro server (Unix only). Used by the nsd and esd initd daemon scripts for instance.
`Pyro/ext/BasicNTService.py`
  Utility classes for creating Windows NT Services (Windows NT only).
`Pyro/ext/NS_NtService.py`
  The Name Server as Windows NT Service.
`Pyro/ext/ES_NtService.py`
  The Event Server as Windows NT Service.
`Pyro/ext/ServiceTest.py`
  Test sets for the Windows NT Services.

## Possible issues in current implementation

There are some issues and possible problems with the current code. Please discuss about these issues on the Mailing List.

- `PyroURI` has `__hash__` method, so it can be used as key in a dict. However, PyroURIs are not immutable! So if it is used in a dict, and the URI changes, you're boned, because it ends up in the wrong hash bucket!
- `DynamicProxy` must have a `__coerce__` method because Python 2.0.x appears to call that when comparing proxy objects. Python 2.1+ doesn't call it. The `__coerce__` method just returns (self,other) and this appears to work.
- `DaemonServant`: self.daemon object is cyclic reference to Daemon object, if weakrefs are not implemented in your Python version.
- Any protocol error that occurs will close the connection. Is this wat we want? Connections could be self-repairing?
- Pyroc cannot deal with packages.
- Problem in M2Crypto (currently a workaround has been implemented): select doesn't work on M2Crypto's SSL sockets if there are still bytes pending.
- Problem in mobile code: when .pyc and .pyo files are not present, the real source file (.py) is transmitted. This file won't compile if it contains windows-style (CR LF) line endings. (on any OS). Luckily, usually the .pyc or .pyo file is present.
- TCPServer and Daemon depend very much on each other. TCPServer even calls methods that are only known in the Daemon. This is ugly.

More can be found in the TODO document that comes with the Pyro distribution.

<previous | contents | next> | Pyro Manual