% -*- latex -*-
%
% Copyright (c) 2001-2004 The Trustees of Indiana University.  
%                         All rights reserved.
% Copyright (c) 1998-2001 University of Notre Dame. 
%                         All rights reserved.
% Copyright (c) 1994-1998 The Ohio State University.  
%                         All rights reserved.
% 
% This file is part of the LAM/MPI software package.  For license
% information, see the LICENSE file in the top level directory of the
% LAM/MPI source distribution.
%
% $Id: requirements.tex,v 1.16 2003/11/26 00:25:29 pcharapa Exp $
%

\chapter{Requirements}
\label{sec:requirements}

LAM/MPI requires a minimal set of features common on most Unix-like
operating systems in order to configure, compile, install, and
operate.  However, optional plug-in modules may require additional
libraries and operating system features.  The requirements for modules
included with the LAM/MPI distribution are documented in
Section~\ref{sec:requirements:ssi}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{General LAM Requirements}

In order to configure and build LAM, the following are required:

\begin{itemize}
\item A POSIX-like operating system (see
  Section~\ref{sec:releasenotes:platform} for notes on commonly used
  operating systems)
  
\item A modern compiler suite:
  \begin{itemize}
  \item An ANSI C compiler
  \item A C++ compiler with STL and namespace support\footnote{If
      using the GNU compiler suite, it is recommended that version
      2.95 or later be used.}
  \end{itemize}
  
\item Common Unix shell utilities such as \cmd{sed}, \cmd{awk}, and
  \cmd{grep}.
  
\item A ``modern'' \cmd{make}.  Unless noted in the Release Notes
  (Chapter~\ref{sec:releasenotes}), the \cmd{make} shipped with any
  modern version of the supported operating systems should work.
\end{itemize}

Once built, LAM does not require a compiler on all nodes used to run
parallel applications.  However, the wrapper compilers provided with
LAM/MPI will only function properly if they are able to invoke the
compiler originally used to build LAM.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{SSI Module Requirements}
\label{sec:requirements:ssi}

LAM/MPI is built around a core component architecture known as the
System Services Interface (SSI).  One of its central principles is the
ability to have run-time ``plug-in'' modules that can extend LAM's
native capabilities.  As such, many of LAM's intrinsic functions have
been moved into SSI modules, and new modules have been added that
provide capabilities that were not previously available in LAM/MPI.
Component modules are selected from each type at run-time and used to
effect the LAM run-time environment and MPI library.

There are currently four types of components used by LAM/MPI:

\begin{itemize}
\item \kind{boot}: Starting the LAM run-time environment, used mainly
  with the \cmd{lamboot} command.

\item \kind{coll}: MPI collective communications, only used within MPI
  processes.

\item \kind{cr}: Checkpoint/restart functionality, used both within
  LAM commands and MPI processes.

\item \kind{rpi}: MPI point-to-point communications, only used within
  MPI processes.  
\end{itemize}

The LAM/MPI distribution includes instances of each component type
referred to as modules.  Each module is an implementation of the
component type which can be selected and used at run-time to provide
services to the LAM run-time environment and MPI communications layer.

Each SSI module has its own configuration and build sub-system.  The
top-level LAM \cmd{configure} script will act as a coordinator and
attempt to configure and build every SSI module that it finds in the
LAM/MPI source tree.  Any module that fails to configure properly will
simply be skipped -- it will not cause the entire build to fail.  For
example, if a system has no Myrinet hardware (and therefore no
corresponding gm message passing software), the \rpi{gm} RPI module
will simply be skipped.

SSI modules may have requirements in addition to the general LAM
requirements.  Failure to meet the requirements listed below does not
prohibit using LAM/MPI, but does prohibit using the given SSI module.
A module may require additional libraries standard to using a
particular interface (for example, the \boot{bproc} boot module
requires \file{libbproc.(a|so)} in order to build properly).  Such
dependencies are not explicitly listed here.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{BProc Boot Module}

\index{bproc boot module@\boot{bproc} \kind{boot} module!versions supported}

\changebegin{7.1}

The LAM/MPI \boot{bproc} boot module has only been tested with the
3.2.x series of BProc starting with v3.2.5.  Prior versions of the
3.2.x series are known to have bugs that may cause LAM/MPI to fail.
The \boot{bproc} module has also been tested with BProc v4.0.

\changeend{7.1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{SLURM Boot Module}

\index{slurm boot module@\boot{slurm} \kind{boot} module}

\changebegin{7.1}

The LAM/MPI \boot{slurm} boot module supports running in SLURM
run-time environments.  No additional configuration or build
parameters are needed to include support for SLURM.  The \cmd{laminfo}
command can be used to verify that the \boot{slurm} boot module is
available in a LAM/MPI installation.

\changeend{7.1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{BLCR Checkpoint/Restart Module}

\index{blcr checkpoint/restart module@\crssi{blcr} checkpoint/restart module}

The BLCR checkpoint/restart SSI module for LAM/MPI requires the
Berkeley Laboratory Checkpoint/Restart package for operation.
%
BLCR has its own limitations (e.g., BLCR does not yet support saving
and restoring file descriptors); see the documentation included in
BLCR for further information.  Check the project's main web
site\footnote{\url{http://ftg.lbl.gov/}} to find out more about BLCR.

\changebegin{7.1}
If the BLCR module(s) are compiled dynamically, the \ienvvar{LD\_\-PRELOAD} 
environment variable must include the location of the \ifile{libcr.so} library.  
This is to ensure that \ifile{libcr.so} is loaded before the PThreads library.
\changeend{7.1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Myrinet (gm) RPI Module}
\label{sec:requirements-gm}
\index{ptmalloc package}
\index{pinned memory}

\changebegin{7.0.1}

LAM/MPI supports both the 1.6.x and 2.0.x series of the Myrinet native
communication library.

Note that LAM/MPI does not need Myrinet hardware installed on the node
where LAM is configured and built to include Myrinet support; LAM/MPI
only needs the appropriate gm libraries and header files installed.
Running MPI applications with the \rpi{gm} RPI module requires Myrinet
hardware, of course.  

\changeend{7.0.1}

The Myrinet native communication library (gm) can only communicate
through ``registered'' or ``pinned'' memory.  In most operating
systems, LAM/MPI handles this automatically by pinning user-provided
buffers when required.  This allows for good message passing
performance, especially when re-using buffers to send/receive multiple
messages.  However, user applications may choose to free memory at any
time.  Hence, LAM/MPI must intercept calls to functions such as
\func{sbrk()} and \func{munmap()} in order to guarantee that pinned
memory is unpinned before it is released back to the operating system.

To this end, the ptmalloc v2 memory allocation
package\footnote{\url{http://www.malloc.de/}} is included in LAM/MPI.
Use of ptmalloc will effectively overload all memory allocation
functions (e.g., \func{malloc()}, \func{calloc()}, \func{free()},
etc.) for all applications that are linked against the LAM/MPI
libraries (potentially regardless of whether they are using the gm RPI
module or not).

\changebegin{7.1}

Please see Section \ref{release-notes:os-bypass} for important notes
about this functionality.

\changeend{7.1}

Note that on Solaris, the gm library does not have the ability to pin
arbitrary memory and auxiliary buffers must be used.  Although LAM/MPI
controls all pinned memory (since the user application cannot free
pinned memory, the ptmalloc package is not necessary, and therefore is
not used), this has a detrimental effect on performance of large
messages: LAM/MPI must copy all messages from the application-provided
buffer to an auxiliary buffer before it can be sent (and vice versa
for receiving messages).  As such, users are strongly encouraged to
use the \mpifunc{MPI\_\-ALLOC\_\-MEM} and \mpifunc{MPI\_\-FREE\_\-MEM}
functions instead of \func{malloc()} and \func{free()}.  Using these
functions will allocate ``pinned'' memory such that LAM/MPI will not
have to use auxiliary buffers and an extra memory copy.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Infiniband (ib) RPI Module}
\label{sec:requirements-ib}
\index{ptmalloc package}
\index{pinned memory}

\changebegin{7.1}

LAM/MPI supports the Infiniband RPI using the Mellanox Verbs API
(VAPI).

Note that LAM/MPI does not need Infiniband hardware installed on the
node where LAM is configured and built to include Infiniband support;
LAM/MPI only needs the appropriate VAPI libraries and header files
installed.  Running MPI applications with the \rpi{ib} RPI module
requires Infiniband hardware, of course.

The Infiniband communication library can only communicate
through ``registered'' or ``pinned'' memory.  In most operating
systems, LAM/MPI handles this automatically by pinning user-provided
buffers when required.  This allows for good message passing
performance, especially when re-using buffers to send/receive multiple
messages.  However, user applications may choose to free memory at any
time.  Hence, LAM/MPI must intercept calls to functions such as
\func{sbrk()} and \func{munmap()} in order to guarantee that pinned
memory is unpinned before it is released back to the operating system.

To this end, the ptmalloc v2 memory allocation package is included in
LAM/MPI.  Use of ptmalloc will effectively overload all memory
allocation functions (e.g., \func{malloc()}, \func{calloc()},
\func{free()}, etc.) for all applications that are linked against the
LAM/MPI libraries (potentially regardless of whether they are using
the ib RPI module or not).

Please see Section \ref{release-notes:os-bypass} for important notes
about this functionality.

\changeend{7.1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Shared Memory RPI Modules}
\label{sec:requirements-shared-memory}

The \rpi{sysv} and \rpi{usysv} shared memory RPI modules require both
System V shared memory and System V semaphore support.

\changebegin{7.1.2}

The \rpi{usysv} RPI module requires both cache coherency and ordered
writes.  On most platforms LAM supports, both conditions are met.  The
PowerPC CPU does allow the system to enforce a weak consistency model,
meaning that writes into shared memory writes are not necessarily
ordered.\footnote{Platforms known to have unordered writes between
  processors are the IBM Power4 ``Regatta'' and the Apple G5 Power Mac
  / XServe platforms.}  The \rpi{usysv} RPI will force the CPU and
memory controller to synchronize writes on the PowerPC where
necessary.

\changeend{7.1.2}

The \rpi{sysv} module allocates a semaphore set (of size 6) for each
process pair communicating via shared memory.  On some systems, it may
be necessary to reconfigure the system to allow for more semaphore
sets if running tasks with many processes communicating via shared
memory.

The operating system may run out of shared memory and/or semaphores
when using the shared memory RPI modules.  This is typically indicated
by failing to run an MPI program, or failing to run more than a
certain number of MPI processes on a single node.  To fix this
problem,  operating system settings need to be modified to
increase the allowable shared semaphores/memory.

For Linux, reconfiguration can only be done by building a new kernel.
First modify the appropriate constants in
\file{include/asm-<arch>/shmparam.h}.  Increasing \var{SHMMAX} will
allow larger shared segments and increasing \const{\_SHM\_ID\_BITS}
allows for more shared memory identifiers.\footnote{This information
  is likely from 2.0 Linux kernels; it may or may not have changed in
  more recent versions.}

For Solaris, reconfiguration can be done by modifying
\file{/etc/system} and then rebooting.  See the Solaris
\file{system(4)} manual page for more information.
%
For example to set the maximum shared memory segment size to 32 MB put
the following in \file{/etc/system}:

\lstset{style=lam-shell}
\begin{lstlisting}
set shmsys:shminfo_shmmax=0x2000000
\end{lstlisting}

If you are using the \rpi{sysv} module and are running out of
semaphores, the following parameters can be set:

\lstset{style=lam-shell}
\begin{lstlisting}
set semsys:seminfo_semmap=32
set semsys:seminfo_semmni=128
set semsys:seminfo_semmns=1024
\end{lstlisting}

\changebegin{7.1}
For Microsoft Windows\trademark\ (Cygwin), the IPC services are provided
by the CygIPC module.  It is necessary to have this module installed 
to build the \rpi{sysv} and \rpi{usysv} RPIs.  Specifically, these RPIs 
are built if and only if \file{libcygipc.a} is found and \cmd{ipc-daemon2.exe}
is running when configuring LAM/MPI.  Furthermore, \cmd{ipc-daemon2.exe} should 
be running on all the nodes if the RPI modules are to be used. 
\changeend{7.1}

Consult your system documentation for help in determining the correct
values for your system(s).

