SPREAD: A Reliable Multicast and Group Communication Toolkit ----------------------------------------------------------- /===========================================================================\ | The Spread Group Communication Toolkit. | | Copyright (c) 1993-2004 Spread Concepts LLC | | All rights reserved. | | | | The Spread package is licensed under the Spread Open-Source License. | | You may only use this software in compliance with the License. | | A copy of the license can be found at http://www.spread.org/license | | | | This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF | | ANY KIND, either express or implied. | | | | Spread is developed at Spread Concepts LLC and the Center for Networking | | and Distributed Systems, The Johns Hopkins University. | | | | Creators: | | Yair Amir yairamir@cs.jhu.edu | | Michal Miskin-Amir michal@spreadconcepts.com | | Jonathan Stanton jstanton@gwu.edu | | | | Major Contributors: | | Cristina Nita-Rotaru crisn@cs.purdue.edu - GC security. | | Theo Schlossnagle jesus@omniti.com - Perl, skiplists, autoconf. | | Dan Schoenblum dansch@cnds.jhu.edu - Java interface. | | John Schultz jschultz@d-fusion.net - contribution to process | | group membership. | | | | Contributors: | | Ryan Caudy wyvern@cnds.jhu.edu - Group membership | | Ben Laurie ben@algroup.co.uk - FreeBSD port and warning fixes | | Daniel Rall dlr@finemaltcoding.com - Java & networking fixes, | | configuration improvements | | Marc Zyngier - Windows fixes | | | | Special thanks to the following for providing ideas and/or code: | | Ken Birman, Danny Dolev, Mike Goodrich, Ben Laurie, | | David Shaw, Robbert VanRenesse. | | | | Partial funding provided by the Defense Advanced Research Projects Agency | | (DARPA) and The National Security Agency (NSA) since 2000. The Spread | | toolkit is not necessarily endorsed by DARPA or the NSA. | | | | WWW : http://www.spread.org and http://www.cnds.jhu.edu | | Contact: spread@spread.org | | | | Version 3.17.3 built 15/October/2004 | \===========================================================================/ October 15, 2004 Ver 3.17.3 --------------------------- 1) Fix memory leak in Skiplist. Reported by Taj Khattra, patch by Theo Schlossnagle. 2) spuser,spflooder, and spmonitor fixed to print correct name in help. Patch by Daniel Rall. 3) Fix incorrect alarm printing where WARNING messages generated by older Alarm() interface were not printed. 4) Modify monitor so it will run correctly on Windows. This was done by modifying monitor.c so it can also build as a threaded program and on windows will use multiple threads to send/recv updates and get user imput. (A POSIX thread option is also added). 5) Fix a bug in the Windows build using VC++ of thread-safe libtspread.lib. Bug and fix by Jacob Green. 6) Fix bug in libspread where if the groups array or message body passed to SP_recv* was too small, the mess_type field returned would be truncated and the sender field was not returned. They are both now returned correctly. Bug report and partial fix provided by John Schultz. 7) Fix bug where SP_Join and SP_Leave do not report an error if a group name is too long (instead they truncated it) Reported with fix by David Parker. ** Warning, this could break buggy applications who use long groups and assume the name is truncated. 8) Cleanup compile warnings where E_queue() used with no-parameter functions (not all uses fixed) and fix incorrect use of signed int with strlen(). 9) Fix few cases in flooder.c and user.c that did not use the defined MAX_MESSLEN constant. Tested to verify that increasing MAX_SCATTER_ELEMENTS in scatter.h and the MAX_MESSLEN defines in user.c and flooder.c is sufficient to support arbitrarily large message sizes with Spread. This is NOT recommended, but several people do it anyway :-) 10) Add new option to spflooder. The -n option allows a fixed "number of members" to be set. This then activates a multi-sender flow control algorithm to allow flooding tests with several senders. All of the processes need to join the group (i.e. they cannot be -wo (write-only)) but not all processes have to send. This allows easy testing with differing numbers of senders (just change how many spflooders start with -ro and how many do not). 11) Make E_delay() work on Win32. 12) Added check that a segment using localhost (i.e. 127.0.0.255) is NOT in a spread.conf with any other segments using real IP addresses. This configuration will not work correctly and is usually because the example localhost configuration provided with Spread is added to. This check will cause the daemon to exit immediately after parsing the config file. 13) Add new Alarm priority flag to print a line with no datestamp (for multi-line output). 14) Fix bug where if more then 22 daemons start at the same time, some will crash or the membership will not complete correctly. This bug was reported by several people including Jesse Noller. 15) Fix Java spread connection problem where several threaded connections from same process cause unexpected connection failures. Reported by Brian Moseley, idea of fix by Ryan Caudy, patch by Jonathan Stanton. 16) Fix EVS bug where AGREED messages may be delivered before a transitional signal on some daemons and after it on others. Bug found and patch created by Ryan Caudy. 17) Make SP_connect_timeout() calls non-blocking for the actual 'connect()' call. This should fix the issue reported by Shlomi Yaakobovich where a hung daemon causes new connections to also hang in connect. Also includes slight cleanup of connect code path. 18) Add new Windows VC++.Net project files to daemon/win32_msvc_net. The current project files in daemon/win32 remain as they can be used in VC++ (version 6). SOURCE INSTALL: --------------- The source install uses autoconf and make to support many Unix and unix-like platforms as well as Windows 95+. Windows is supported by a set of Visual Studio desktop and project files in the win32 subdirectory. These files build the daemon, libraries, and user programs. Any not-specifically supported platform that has reasonably close to normal sockets networking should also be easily portable. See the file PORTING for hints on porting. From the directory where you unpacked the Spread source distribution do the following: 1) Run "./configure" 2) Run "make" 3) Run "make install" as a user with rights to write to the selected installation location. (/usr/local/{bin,man,sbin,include,lib} by default) 4) Create a spread.conf file appropriate to your local configuration. For more info on how to do this look at the sample file "sample.spread.conf" or below in the binary install instructions. BINARY INSTALL: -------- We recommend that if you are experimenting with spread you create a special 'spread' directory (for example /usr/local/spread or /opt/spread) and keep all the files together there so it is easy to find stuff. That also makes it easier to run multiple architectures as the binaries for each are in their own subdirectory. This is not necessary though. You can create that directory anywhere (e.g. your own directory). If you are installing spread for active use it is probably easier to just install the correct version of the binaries, headers, man pages, and libraries into your standard locations. The directions below assume you are doing this. 1) Unpack the spread.tar.gz file into a temporary directory 2) Look at the Readme for any updates 3) Select the appropriate architecture: arch-bsdi arch-sgi arch-sunos arch-sunsol arch-pcsol arch-linux 4) Type 'make arch-????' with your architecture as the option to make 5) Now you need to copy the files, I will assume you use /usr/local/{bin,include,lib,man} Replace "ARCH" with the directory for your architecure. cp -p include/* /usr/local/include/ cp -p ARCH/libspread.a /usr/local/lib/ cp -p ARCH/libtspread.a /usr/local/lib/ cp -p ARCH/spread /usr/local/bin/ cp -p ARCH/spmonitor /usr/local/bin/ cp -p ARCH/spuser /usr/local/bin/ cp -p ARCH/sptuser /usr/local/bin/ cp -p ARCH/spflooder /usr/local/bin/ cp -p docs/*.3 /usr/local/man/man3/ cp -p docs/*.1 /usr/local/man/man1/ 6) To run spread you need to modify the 'spread/sample.spread.conf' file for your network configuration. Comments in the sample file explain this. Then cp -p sample.spread.conf /usr/local/bin/spread.conf (Notice the name change. The config file must be named that to be found. Alternatively you can run spread -c ) You can run "spread usage" to see the spread running options. To use the Java classes and examples you need to have a copy of the main 'spread' daemon running. Then the sp.class file gives you the equivelent of the libsp.a as a java class. The user.java, user.html, and user.class files give you a demonstration applet and source code. The tree.html AllNames.html and packages.html give some documentation for the java interface. For Windows (95/NT) systems use the spread.exe daemon and the libsp.lib to link with your programs. OVERVIEW: --------- Spread consists of two parts, an executable daemon which is executed on each machine which is part of the heavyweight 'machine' group, and a library which provides a programming interface for writing group multicast applications and which is linked into the application. The daemon, called "spread", should be run as a non-priveledged user (we created a 'spread' user) and only needs permissions to create a socket in /tmp and read its config file which should be in the same directory as the executable. By default the daemon binds to and runs off the non-priveledged port 4803 unless the config file indicates otherwise. Each daemon can be started independently and if it does not find any other members of its defined configuration currently active it runs as a machine group of 1 machine. When other daemons are started they will join this machine group. The machines which are running a daemon with a common config file form a 'machine group' (in contrast to a 'process group'). The daemons handle multicasting mesages between each other,providing ordering and delivery guarantees, detecting and handling failures of nodes or links, and managing all applications which are connected to each daemon. Each application utilizing the spread system needs to link with the 'libspread.a', 'libtspread.a', 'libtspread.lib' or 'libspread.lib' library and needs to use the calls defined in the 'sp.h' and 'sp_func.h' header files. Documentation for the interface is currenly incomplete, but a basic application must do at least the following: 1) Before using any other spread calls you need to 'connect' to a daemon by calling the SP_connect(...) call. This will set a mbox which is an integer representing your connection to the daemon. You use this when making other spread calls. You CAN connect to daemons multiple times from the same application (and we know of times when this is very useful). 2) If you want to receive messages from a group you need to call SP_join(). Groups are named with standard alphanumeric strings. 3) To send to a group you do NOT need to join it, just call SP_multicast() with an appropriate group name. You will not receive the messages back if you are not a member of the group. 4) To be nice to everyone else you should call SP_disconnect() when your program is finished using the spread system, if you do not your program's disconnection will eventually be detected. Some important observations when writing spread programs: 1) All the messages for a particular connection are received during an SP_receive() call. This includes messages from different groups and membership messages(like group leave and join), so you must demultiplex them yourself. This is a feature. You can opt to avoid getting membership messages all together by indicating that to SP_connect(). 2) Spread handles endian differences correctly for metadata and our headers but does NOT convert your data for you. It DOES tell you in the SP_receive call whether or not the data originated at machine with a different endianness then yours so you can easily use this to convert it yourself if necessary. 3) Each connection to a daemon has a unique 'private group' name which can be used to send a message to a particular, and only to a particular, application connection. 4) Spread supports the following ordering/delivery guarantees for messages: Unreliable (least) Reliable (will get there, no ordering) Fifo (reliable and ordered fifo by source) Causal (reliable and all mesg from any source of this level are causally ordered) Agreed (reliable and all mesg from any source of this level are totally ordered) Safe (Agreed ordering and mesg will not be delivered to application until the mesg has reached ALL receipients' daemons) For more detail, much of which IS important, see papers on Extended Virtual Sychrony, Transis and Totem. UTILITY PROGRAMS: ----------------- Spread comes with several demonstration programs and useful tools. 1) spmonitor: This program has a special interface to spread which allows it to control a machine group. It can terminate all the spread daemons in the group, change flow control parameters, and monitor the stats for one or all of the machines. It needs to know the spread.conf file used for a particular set of spread daemons. 2) spuser: This program is provided with source and uses all the functions of the spread interface. It also acts as a simple client which is useful for testing. Reading the source code to this program should show you how to use all the features of spread, and answer many questions about syntax details. 3) simple_user: This is just about the simplest spread program possible and is provided with source. It sends and receives text strings. 4) spflooder: This is used as one type of performace test. It 'floods' a spread group with data messages. It is provided with source. CONTACT: -------- If you have any questions please feel free to contact: commedia@cnds.jhu.edu yairamir@cs.jhu.edu jonathan@cs.jhu.edu