1 Overview WebJob downloads a program or script from a remote WebJob server and executes it in one unified operation. Any output produced by the program/script is packaged up and sent to a remote, possibly different, WebJob server. WebJob is useful because it provides a mechanism for running known good programs on damaged or potentially compromised systems. This makes it ideal for remote diagnostics, incident response, and evidence collection. WebJob also provides a framework that is conducive to centralized management. Therefore, it can support and help automate a large number of common administrative tasks and host-based monitoring scenarios such as periodic system checks, file updates, integrity monitoring, patch/package management, and so on. This document explains how to build WebJob, configure an Apache Web server to handle WebJob requests, and verify client-server connectivity. WebJob is available at: http://webjob.sourceforge.net 1.1 Revision $Id: README.INSTALL,v 1.17 2006/05/31 15:14:48 mavrik Exp $ 1.2 Table of Contents Section 1 .................... Overview Section 1.1 .................. Revision Section 1.2 .................. Table of Contents Section 1.3 .................. Caveats Section 1.3.1 ................ Windows NT/2K Specific Caveats Section 2 .................... WebJob Installation Section 2.1 .................. Requirements Section 2.2 .................. Unpack the Source Archive Section 2.3 .................. Verify Installation of OpenSSL Section 2.4 .................. UNIX/Cygwin Build Instructions Section 2.4.1 ................ UNIX/Cygwin Build Gotchas Section 2.5 .................. Windows NT/2K Build Instructions Section 2.5.1 ................ Windows NT/2K Build Gotchas Section 3 .................... Web Server Configuration Section 3.1 .................. Requirements Section 3.2 .................. CGI Script Setup Section 3.2.1 ................ Environment Variables Section 3.2.2 ................ CGI Directory Setup Section 3.2.3 ................ Script Setup Section 3.2.4 ................ Access Control Setup Section 3.3 .................. Verify Client-Server Connectivity Section 3.3.1 ................ Server Configuration Section 3.3.2 ................ Client Configuration Section 3.3.3 ................ Test Client-Server Connectivity 1.3 Caveats The various programs and scripts referenced throughout this document are generally specified using relative paths. To avoid potential complications, you can either make sure that your PATH environment variable is configured properly or provide full path information as needed. 1.3.1 Windows NT/2K Specific Caveats The native NT/2K build only supports execution of .exe, .com, and .bat files. In particular, scripts such as .pl and .sh that require a file association to run will not work and are not supported at this time. This limitation, however, can be overcome by packaging the target script as a PaD executable. 2 WebJob Installation This section describes the process for building and installing WebJob. 2.1 Requirements Building WebJob in UNIX or Cygwin environments requires at least the following tools: make, gcc, and possibly autoconf. For Windows NT/2K environments Visual Studio 6.0 is required. By default, SSL support is enabled in WebJob. This means that OpenSSL must be be installed and/or available on the build system. 2.2 Unpack the Source Archive The source archive needs to be unpacked in a suitable location before the build process can commence. All commands presented in this section attempt to unpack the source archive in the current working directory. On UNIX and Cygwin platforms this can be accomplished as follows: $ tar -zxvf If your version of tar doesn't support the 'z' flag, then try the following command pipeline: $ gunzip -c | tar -xvf - On Windows platforms WinZip may be used to unpack the archive. Since the source archive is actually a gzipped tar file, WinZip will ask if you wish to decompress and open it -- answer yes. The command line syntax for WinZip is given here: > winzip32.exe -min -e 2.3 Verify Installation of OpenSSL By default, SSL support is enabled in WebJob. This means that OpenSSL must be installed and/or available on the build system. The OpenSSL project is located at http://www.openssl.org. Refer to OpenSSL's documentation for details on how to do this. 2.4 UNIX/Cygwin Build Instructions Typically, the build process is simply a matter of running configure and make. In general, you are not required to specify any extra options/arguments when invoking either command. The most basic build sequence is as follows: $ ./configure && make By default, SSL support is enabled. This means that the configure script will attempt to locate OpenSSL on your system. If OpenSSL is not found in one of several common locations, the script will abort. To override this behavior and explicitly specify OpenSSL's location, use the --with-ssl=DIR option. For example, if your copy of OpenSSL was installed in /usr/local/my_openssl, the build sequence would be as follows: $ ./configure --with-ssl=/usr/local/my_openssl && make To disable SSL support, use the --without-ssl option as follows: $ ./configure --without-ssl && make WebJob's default installation directory is /usr/local/webjob. If you want to install in a different location, set the --prefix configure option. For example, if you wanted to install in the directory /usr/local/my_webjob, the build sequence would be as follows: $ ./configure --prefix=/usr/local/my_webjob && make To see a listing of all available configure options execute the following command: $ ./configure --help The following command configures and builds the software with server-side components and all supporting utilities. $ ./configure --with-server-side --with-all-tools && make When the software has been built to your satisfaction, it can be tested and installed with the following commands: # make test # make install 2.4.1 UNIX/Cygwin Build Gotchas When experimenting with different build/configure options, be sure to do a "make clean-all" between builds. This ensures that any intermediate files are removed. In certain situations, build problems could arise if this is not done. Do not use the native version of make on Solaris platforms as the build/test process will, in all likelihood, fail. Use GNU Make 3.80 or higher instead (i.e., gmake). Executing the built-in tests (i.e., the test target) can take a considerable amount of time -- especially on older/slower systems. Also, these tests may fail if you don't have the necessary Perl modules installed. 2.5 Windows NT/2K Build Instructions WebJob has compiled successfully on Windows NT/2K using Microsoft's Visual Studio 6.0. The build process is simply a matter of setting up a build environment and running nmake. Windows NT/2K binaries are built using the following Makefile: Makefile.vs. Note that there is no equivalent configure process for NT/2K builds. When building WebJob from the command line, you may need to run the batch script VCVARS32.BAT to establish a build environment. One quick test is to check the LIB and INCLUDE variables with: > set LIB > set INCLUDE If neither variable is defined or you are in doubt, run VCVARS32.BAT. Usually, this script is kept in Visual Studio's bin directory. Once you have located the script, simply execute it. Because SSL support is enabled by default, you need to make sure that the SSL_DIR macro in Makefile.vs has an appropriate value. Basically, Makefile.vs was written with the assumption that your version of OpenSSL was installed in C:\OpenSSL. Adjust SSL_DIR as necessary, if this is not the case. Assuming that you are required to run VCVARS32.BAT and all the required libraries and header files are installed in their default locations, the build sequence would be as follows: > VCVARS32.BAT > nmake -f Makefile.vs If you don't want SSL support, set the USE_SSL macro to 'N'. This can be done from the command line as follows: > nmake -f Makefile.vs USE_SSL=N If you want to turn on debugging, set the BUILD_TYPE macro to 'DEBUG'. This can be done from the command line as follows: > nmake -f Makefile.vs BUILD_TYPE=DEBUG The various macros described in this section may be combined in any given build. For example, to build a debug release without SSL support, invoke nmake as follows: > nmake -f Makefile.vs BUILD_TYPE=DEBUG USE_SSL=N When the software has been built to your satisfaction, it can be tested and installed with the following commands: > nmake -f Makefile.vs test > nmake -f Makefile.vs install Note: Testing require ActivePerl. WebJob's default installation directory is C:\WebJob. If you want to install in a different location, adjust the INSTALL_DIR macro as appropriate -- either in the makefile or on the command line. For example, to install WebJob in the 'Program Files' directory, invoke nmake as follows: > nmake -f Makefile.vs install INSTALL_DIR="C:\Program Files\WebJob" 2.5.1 Windows NT/2K Build Gotchas You may have trouble compiling with nmake if your environment settings aren't correct. When this happens, try running the batch file VCVARS32.BAT provided with Visual Studio. If all else fails, try creating a Visual Studio project and import the source code. When experimenting with different build options (e.g., BUILD_TYPE, USE_SSL, etc.), be sure to do a "make clean-all" between builds. > nmake -f Makefile.vs clean-all This will ensure that everything is rebuilt correctly. In certain situations, build problems could arise if this is not done. You may have trouble linking against static OpenSSL libraries. If that happens check to see if the OpenSSL Makefile (ms\nt.mak) is using /MT -- by default it uses /MD (as of openssl-0.9.7f). Executing the built-in tests (i.e., the test target) can take a considerable amount of time -- especially on older/slower systems. Also, these tests may fail if you don't have ActivePerl and the necessary Perl modules installed. 3 Web Server Configuration This section describes how to configure an Apache Web server running on a UNIX platform to handle WebJob requests. It is assumed that Apache is installed and running correctly. Throughout this section, the terms username and clientid are used interchangeably. By default, nph-webjob.cgi expects a clientid to conform to the following Perl regular expression: ^(?:[A-Za-z](?:(?:[0-9A-Za-z]|[_-](?=[^.]))){0,62})(?:[.][A-Za-z](?:(?:[0-9A-Za-z]|[_-](?=[^.]))){0,62}){0,127}$ 3.1 Requirements To configure a UNIX-based Apache Web server to handle WebJob requests you will need nph-webjob.cgi, Apache, and Perl. The name of the WebJob CGI script must begin with the string "nph-". This naming convention, which stands for Non Parsed Headers, informs the Web server that the CGI application is responsible for generating the entire HTTP response. Non Parsed Headers are used by this application because they allow it to take full advantage of HTTP status code extensions. This simplifies client-side error checking without giving up the ability to generate application specific responses. The tree structure depicted here is used by the server to support WebJob request processing. Each top-level directory is explained in the following paragraphs. webjob | + archives | + config | | | + nph-webjob | | | - nph-webjob.cfg | | | + clients | | | | | - nph-webjob.cfg | | | | | + | | | | | - nph-webjob.cfg | | | | | + | | | | | - nph-webjob.cfg | | | + commands | | | - nph-webjob.cfg | | | + | | | - nph-webjob.cfg | | | + | | | - nph-webjob.cfg | + incoming | | | - .out | - .err | - .env | - .rdy | + logfiles | | | - nph-webjob.log | - nph-webjob-trigger.log | + outcasts | + pidfiles | + pipeline | + profiles | + | | | + commands | | | - | + | + baseline | | | + | | | | | - | | | + | | | - | + config | + commands | - The archives directory is a area where uploaded data may be placed for long-term storage. It is not currently used by the CGI script. The config directory contains sub-directorys for each server-side processing application. For nph-webjob.cgi, there is a sub-directory called nph-webjob. This directory contains the main CGI configuration file. The directories below this level are optional, and they are used in conjunction with the EnableConfigOverrides and ConfigSearchOrder controls. The incoming directory is a holding tank. This is where uploaded data is stored prior to being processed. Files in this directory are created automatically by nph-webjob.cgi as clients post their data to the server. The names of these files may be customized (see PutNameFormat property in nph-webjob.cfg). The logfiles directory is self-explanatory. The pidfiles directory is reserved for future use. The pipeline directory is reserved for future use. The outcasts directory is reserved for future use. The profiles directory contains one or more shared directories (e.g., common, shared, sysadmin, etc.) and one directory for each client that is managed by the server. Client directories contain three sub-directories: baseline, config, and commands. The baseline tree is intended to be a mirror or backup (perhaps sparsely populated) of the client file system. Trusted files are placed in this tree just as they would appear on the client system relative to the root directory. The config tree is reserved for future use. The commands tree is intended to contain programs or scripts that you want the client to execute. This directory may also be populated with symbolic links that point to native files in the correspondig baseline tree. WebJob automatically looks in the commands tree for programs and scripts that may be served to the client. Shared directories (e.g., common, shared, webadmin, etc.) are used to store programs common to multiple clients. These directories are searched when the requested program or script can't be found in the client's commands directory. Shared directories are enabled by adding their name to the folder list (see FolderList property in nph-webjob.cfg). 3.2 CGI Script Setup This section describes how to install nph-webjob.cgi. It also explains how to modify Apache's configuration to recognize the script's location and restrict access to the script through basic authentication. 3.2.1 Environment Variables The variable WEBJOB_DIR refers to the location where the WebJob server-side files will reside. This document assumes that WEBJOB_DIR is set to the value "/var/webjob". You may set this variable on your system as follows: Bourne shell variants: # export WEBJOB_DIR=/var/webjob C shell variants: # setenv WEBJOB_DIR /var/webjob The variable APACHE_DIR refers to the location where Apache is installed. This document assumes that APACHE_DIR is set to the value "/usr/local/apache". You may set this variable on your system as follows: Bourne shell variants: # export APACHE_DIR=/usr/local/apache C shell variants: # setenv APACHE_DIR /usr/local/apache The variable CGI_DIR refers to the location where nph-webjob.cgi will be installed. This document assumes that CGI_DIR is set to the value "${APACHE_DIR}/cgi-client". You may set this variable on your system as follows: Bourne shell variants: # export CGI_DIR=${APACHE_DIR}/cgi-client C shell variants: # setenv CGI_DIR ${APACHE_DIR}/cgi-client The variable CFG_DIR refers to the location where nph-webjob.cfg will be installed. This document assumes that CFG_DIR is set to the value "${WEBJOB_DIR}/config/nph-webjob". You may set this variable on your system as follows: Bourne shell variants: # export CFG_DIR=${WEBJOB_DIR}/config/nph-webjob C shell variants: # setenv CFG_DIR ${WEBJOB_DIR}/config/nph-webjob For the remainder of this document, it is assumed that these variables have been set in your shell. 3.2.2 CGI Directory Setup The script nph-webjob.cgi should be installed in a directory that is strictly dedicated to one purpose: access control. The major factor here is that this application is designed to write data to the server's local disk. Whenever access of this kind is allowed by potentially hostile clients, you want to maintain a tight rein. If your environment is set up as described in section 3.2.1, you may create a dedicated CGI directory as follows: # mkdir -p -m 755 ${CGI_DIR} # chown 0:0 ${CGI_DIR} Next, assuming that you have just created a script directory that Apache is not yet aware of, you will need to update httpd.conf. Locate this file and place the following snippet in the alias section. The alias section may be located by searching for the following string: Note: You must manually replace ${CGI_DIR} and ${CFG_DIR} with their literal values. --- snippet --- SetEnv WEBJOB_PROPERTIES_FILE ${CFG_DIR}/nph-webjob.cfg ScriptAlias /cgi-client/ "${CGI_DIR}/" AllowOverride AuthConfig Options None Order allow,deny Allow from all --- snippet --- After modifying httpd.conf, you should verify its syntax and, if that is successful, signal Apache to restart. To verify syntax use the following command: # apachectl configtest If necessary, correct syntax errors. Otherwise, tell Apache to restart with: # apachectl restart At this point, you are ready to install nph-webjob.{cgi,cfg}. 3.2.3 Script Setup Locate nph-webjob.cgi in the WebJob distribution tree. It should be in the cgi directory. Change to this directory, and install the script in ${CGI_DIR} as follows: # cp nph-webjob.cgi ${CGI_DIR} # chmod 755 ${CGI_DIR}/nph-webjob.cgi # chown 0:0 ${CGI_DIR}/nph-webjob.cgi Verify that your Perl interpreter is located at /usr/bin/perl. If this is not the case, modify the first line of the script to reflect the true location of the interpreter. Typically, that line should look like this: --- snippet --- #!/usr/bin/perl --- snippet --- Next, you need to create the following basic tree structure: ${WEBJOB_DIR} | - config | - incoming | - logfiles | - profiles By default, nph-webjob.cgi expects to find this tree in the WEBJOB_DIR directory (i.e., /var/webjob). If a different location is desired, modify the BaseDirectory property in nph-webjob.cfg as appropriate. Otherwise, use the following command sequence to create the tree. However, before doing this, you need to determine the UID and/or username that Apache uses. This document assumes that Apache runs in the context of the www user. If your configuration is different, replace www with the appropriate username as necessary. # mkdir -p -m 755 ${WEBJOB_DIR} # chown 0:0 ${WEBJOB_DIR} # cd ${WEBJOB_DIR} # mkdir -p -m 755 config incoming logfiles profiles # chown www:0 incoming Locate nph-webjob.cfg in the WebJob distribution tree. It should be in the etc directory. Change to this directory, and install the config file in ${CFG_DIR} as follows: # mkdir -p -m 755 ${CFG_DIR} # cp nph-webjob.cfg ${CFG_DIR} # chmod 644 ${CFG_DIR}/nph-webjob.cfg # chown 0:0 ${CFG_DIR}/nph-webjob.cfg Finally, you need to create the nph-webjob log files. This is done as follows: # touch ${WEBJOB_DIR}/logfiles/nph-webjob.log # chown www:0 ${WEBJOB_DIR}/logfiles/nph-webjob.log # chmod 644 ${WEBJOB_DIR}/logfiles/nph-webjob.log # touch ${WEBJOB_DIR}/logfiles/nph-webjob-trigger.log # chown www:0 ${WEBJOB_DIR}/logfiles/nph-webjob-trigger.log # chmod 644 ${WEBJOB_DIR}/logfiles/nph-webjob-trigger.log The final configuration objective is to establish an access control mechanism. 3.2.4 Access Control Setup By default, nph-webjob.cgi expects Apache's basic authentication to be enabled. Therefore, you'll need to create and install an access file in ${CGI_DIR}. To determine the name of access files on your system, check the AccessFileName directive in httpd.conf. Typically, this directive contains the value .htaccess. The following snippet may be used to create a suitable access file. Note: Be sure to replace ${APACHE_DIR} with its literal value. --- snippet --- AuthType Basic AuthName "WebJob Client Realm" AuthUserFile ${APACHE_DIR}/htusers require valid-user --- snippet --- Assuming that Apache's AccessFileName directive contains the value .htaccess, the following command sequence may be used to create and install a suitable access file in ${CGI_DIR}. # echo "AuthType Basic" > ${CGI_DIR}/.htaccess # echo "AuthName 'WebJob Client Realm'" >> ${CGI_DIR}/.htaccess # echo "AuthUserFile ${APACHE_DIR}/htusers" >> ${CGI_DIR}/.htaccess # echo "require valid-user" >> ${CGI_DIR}/.htaccess # chmod 644 ${CGI_DIR}/.htaccess # chown 0:0 ${CGI_DIR}/.htaccess To complete access control setup, you must create a password file and insert a new user. The purpose of the AuthUserFile directive is to bind a particular access file to a particular password file. Therefore, be sure to use the filename assigned to AuthUserFile when creating the password file. The following command will create a password file and insert an entry for the client_1 user. You will be prompted to enter the user's password. # htpasswd -c ${APACHE_DIR}/htusers client_1 Set the permissions and ownership on this file as follows: # chmod 644 ${APACHE_DIR}/htusers # chown 0:0 ${APACHE_DIR}/htusers Finally, verify that the password file is not accessible to remote Web browsers. Once you have established a working authentication scheme, you are ready to begin testing client-server connectivity. 3.3 Verify Client-Server Connectivity Before tackling this section, read the WebJob man page. Pay particular attention to Example 1. The purpose of this section is to test client-server connectivity. This will be done by configuring WebJob to download and run a script -- output will be displayed on stdout/stderr. Get the password for client_1 handy -- i.e, the user you created in section 3.2.4. This password will be used in the coming sections to complete the verification process. 3.3.1 Server Configuration On the server you need to build client_1's directory structure and create the following scripts: host.bat and host.sh. The directory structure for client_1 can be created as follows: # umask 22 # mkdir -p -m 755 ${WEBJOB_DIR}/profiles/client_1/commands # chown -R 0:0 ${WEBJOB_DIR}/profiles/client_1 Create the host.{bat|sh} scripts using the content shown below. Put these scripts in client_1's commands directory, and set their file permissions to 644 -- Note that the mode is 644 because programs and scripts in the commands directory should not, in general, be run on the server. --- host.bat --- hostname.exe --- host.bat --- --- host.sh --- #!/bin/sh hostname --- host.sh --- # chmod 644 ${WEBJOB_DIR}/profiles/client_1/commands/host.bat # chmod 644 ${WEBJOB_DIR}/profiles/client_1/commands/host.sh 3.3.2 Client Configuration On the client you need to install WebJob (see section 2) and create a config file. Go to /tmp (\temp for NT/2K) or some other suitable location and create a file called webjob.cfg using the content provided below. Make sure you replace with its real value. Also, don't forget to modify URLGetURL as appropriate. --- webjob.cfg --- ClientId=client_1 URLGetURL=http://your.server.net/cgi-client/nph-webjob.cgi URLUsername=client_1 URLPassword= URLAuthType=basic TempDirectory=/tmp # Use \temp for NT/2K OverwriteExecutable=Y --- webjob.cfg --- The TempDirectory control is optional. By default, it is set to /tmp (\temp for NT/2K). TempDirectory is used to store downloaded programs/scripts and any generated output. The OverwriteExecutable control is also optional. Here, it was specified to eliminate a potential problem that occurs when the same program or script is run more than once -- by default WebJob will not overwrite any program or script that already exists in TempDirectory. 3.3.3 Test Client-Server Connectivity Now, it's time to test client-server connectivity. Execute one of the commands listed below from a shell on the client. Output will be written to stdout/stderr. Don't forget to update your PATH environment variable, if necessary, to include the newly installed WebJob binary. $ webjob --execute --file webjob.cfg host.sh or (for NT/2K) > webjob --execute --file webjob.cfg host.bat If that worked, modify the config file as shown below and rerun the previous command. This time, output will be written to three files located in TempDirectory having the extensions: .env, .err, and .out. After the host.{bat|sh} script has completed, WebJob will attempt to upload the three output files to the server. If the upload succeeds, there will be four files in the server's incoming directory having the extensions: .env, .err, .out, and .rdy. If any of these files doesn't exist, double check WebJob's return status. Also, check Apache and WebJob log files for any signs of trouble. --- webjob.cfg --- ClientId=client_1 URLGetURL=http://your.server.net/cgi-client/nph-webjob.cgi URLPutURL=http://your.server.net/cgi-client/nph-webjob.cgi URLUsername=client_1 URLPassword= URLAuthType=basic RunType=snapshot TempDirectory=/tmp # Use \temp for NT/2K OverwriteExecutable=Y UnlinkOutput=N UnlinkExecutable=N --- webjob.cfg ---