http-analyze 2.01 - A WWW log analyzer ====================================== -------------------------------------------------------------------------- Changes in 2.01 patchlevel 15 / April, 2nd -------------------------------------------------------------------------- Bug fixes: o Fixed a bug introduced in pl14 where Country list disappeared. o Fixed a bug with relative pathname for executing http-analyze in ha-setup in the branding function. -------------------------------------------------------------------------- Changes in 2.01 patchlevel 14 / March, 31th -------------------------------------------------------------------------- Changes: o Added counters for HTTP/1.1 request methods other than GET and POST which cause no document to be transmitted by the server. Those requests are now separated from the file/pageview/session accounting, although it is still accounted for in the hits count. o Added support for Windows NT/Netware by Andre Schild: Changed the names max/avg into max_day/avg_day to avoid name clashes. Added macro ISFULLPATH to test for full pathnames on different platforms. Use faked uname(2) entries for determining platform/OS. Andre also contributed a manual in PDF format for Windows users. o Added a bug reporting form which fills in the required registration information automatically if the software has been branded. NOTE: This requires usernames and passwords for access to our support site at http://support.netstore.de/ (http://www.netstore.de/support/ for older browsers) which are sent to you in a following private email soon. o Added links to the main pages in the two navigation windows. o Added missing link to the short statistics in the frames UI. o Added new TLD CD for Democratic Republic of Congo (former Zaire). The old TLD ZR should still remain present for a transition period. Bug fixes: o Fixed bug where links to previous months were created, although there was no record for this month in the history. o Fixed a bug where links were insereted for empty files although the number of total hits for items/URLs, sites/domains, agents/agent types, referrer hosts/referrer URLs was zero. Now, no empty files are created if no hits have been detected in the appropriate category. o Check for zero denominator in PERCENT macro to avoid FPE DIVZERO exception when counters for totals remain zero as can happen with the new HTTP/1.1 server responses. This caused a core-dump on certain platforms. o Fixed a bug with not strict-ANSI compatible handling of varargs parameter list in function dfpr(). o Fixed a bug where empty lines were seen as binary garbagge. o Fixed a bug in the NavWin configuration setting. -------------------------------------------------------------------------- Changes in 2.01 patchlevel 13 / March, 8th -------------------------------------------------------------------------- Bug fix: o Fixed bug in new code for determining version number. -------------------------------------------------------------------------- Changes in 2.01 patchlevel 12 / March, 8th -------------------------------------------------------------------------- Changes: o Now determines the version number automatically. o Cosmetic changes to keep gcc on Solaris 2.5 happy (warning-free). Bug fix: o Fixed wrong data object type in debugging message. -------------------------------------------------------------------------- Changes in 2.01 patchlevel 11 / March, 6th -------------------------------------------------------------------------- Bug fixes: o The 'totals' page now is created everytime, not only when option -f (create a frames UI) was given. o A bug has been fixed which caused the analyzer to dump core on certain platforms if no frames UI was specified. -------------------------------------------------------------------------- Changes in 2.01 patchlevel 10 / March, 2nd -------------------------------------------------------------------------- Bug fix: o The link to the 3D model on the summary page in the statistics summary didn't work when JavaScript was enabled. -------------------------------------------------------------------------- Changes in 2.01 patchlevel 9 / February, 24th -------------------------------------------------------------------------- Enhancements: o The user name and password is now removed in referrer URLs. o The new option '-O virtnames,...' allows to set virtual names as with the VirtualNames directive on the command line. o The analyzer now tries to detect and to skip binary data. Bug fix: o The option '-R docroot' did not work correctly. -------------------------------------------------------------------------- Changes in 2.01 patchlevel 8 / February, 9th -------------------------------------------------------------------------- Enhancements: o The directives in the configuration files have been changed sligthly to avoid mis-interpretations. This seemed to be necessary because some people did get confused about the meaning of certain directives. For example, they casually used "DocRoot" to specify the place for the statistics report instead of "HTMLDir" therefore overwriting their hompage with the statistics report, so this particular directive now has been renamed "OutputDir". Of course, the directive names from previous versions of http-analyze are also still valid. o The conventional (non-frames) interface navigation has been simplified. The idea is that if you have enabled JavaScript in your browser anyway, there will be a small navigation panel which contains all links to the reports and updates the main window (instead of popping up a new one). If you have disabled JavaScript, all reports are displayed in the same window except for the detailed lists, which in every case (JavaScript disabled or enabled) are still displayed in a separate window to minimize loading time when following links in the overview reports. o The analyzer now has a pageview rating which is enabled by default. All files ending with a particular suffix are rated as text files. The suffix ".html" is pre-defined already. You can add more pageview suffixes with the PageView directive in the configuration file. You can suppress pageviews with the "Pageviews" suppress option (-s or "Suppress" directive). If disabled, the analyzer displays 304's (Not Modified) instead. The pageview feature requires an additional field in the history file. http-analyze will automatically convert an older history file into the new format. o The duration, after which accesses from the same host are counted as "Sessions", is now configurable between 0 seconds and 31 days. If for example, the time-window is set to one hour, all accesses from the same host are counted only once until this hosts accesses your server again at least one hour after the first access. In previous releases, the time window was a hardcoded default of one day. o The handling of top-level domains (TLDs) has been enhanced. First, the code now handles TLDs up to 6 characters in length (not counting the dot). The TLD file now contains all valid TLDs, not only the ISO two-letter codes. This has been changed to cover the new CORE domains (.firm, .info, .shop, .arts, .web, .rec, .nom) which will become active in March 1998. Second, the country table now is looked up for using a binary search thus greatly reducing the number of steps to find a given country. (The table of "virtual" countries made up with the AddDomain directive still uses a linear search.) o The average load now includes a bar chart of the average hits per weekday, a bar chart of the avergae hits for the last week, and a table with the top 5 days in addition to the hits per hour chart and top N hours, minutes, and seconds. o The response codes supported by http-analyze have been changed to fully support the HTTP/1.1 standard (RFC2068). o Added new directives NavWinSize and VirtualNames, see the online manual (included as pre-formatted ASCII text and PostScript booklet) for more information. Bug fixes: o A bug has been fixed where http-analyze would core-dump when collecting referrer URLs. o A bug has been fixed where the hits for the previous day were accumulated over the invocations of the analyzer in the short statistics report. o A bug in cntrycode.c where the code for countries from the TLD file was not computed correctly has been fixed. o The ha-setup utility did not copy all required files. It now asks for the LIBDIR so it will install all required files instead of the buttons only. -------------------------------------------------------------------------- Changes in 2.01 patchlevel 7 / January, 24th -------------------------------------------------------------------------- Enhancements: o Added 3DWinSize directive to configuration file to allow for definition of the 3D window. Bug fixes: o Now prints an error message if registration ID couldn't be saved in the registration ID file if -r was given. o Fixed bug which caused Netscape Navigator 3.X to show only two buttons in the frames-based UI. o Fixed bug which caused the analyzer to crash when no custom logos where defined in the registered version. o Fixed bug in JavaScript (frames-based UI) where the year button did select the wrong statisitics subdirectory for the last 12 months report and the yearly 3D model. -------------------------------------------------------------------------- Changes in 2.01 patchlevel 2-6 / January, 17th -------------------------------------------------------------------------- New features: o The manpage has been updated. o Tailoring http-analyze to your Corporate Identity: Customers which registered for a commercial service license now can add their logo to the reports and suppress the Copyright note (except for the main page). o Now handles user-agent and referrer URL: The logfile format is automatically determined by looking at the first 100 logfile entries. If you analyze mixed logfiles, the analyzer uses the format detected in the first file. o Improved "conventional" (non-frames) interface: Some of our customers did request splitting up the output files to make them smaller. We now have three levels of detail for the list of URLs, sites, agents, and referrer URLs. Please see the description about the window management in the manpage if you are confused by the windows popping up the first time you look at the new reports. As far as possible, the output format is compatible to not only 2.0beta, but also to 1.9e. o Better consistency in reports, suppressing "noise": The reports now are structured equally into a Top Ten list, an overview and a detailed list. There is one such report for filenames, sites, user-agents and referrer URLs. The Top N URLs, sites, agents, and referrer URLs now have a pie chart each. A "noise level" parameter has been introduced to suppress detailed reports for all files, sites, agents, and referrer URLs below a certain hit rate. This way, the size of the detailed reports will be reduced by a significant amount. o Sites are now listed under their second-level domain, which is determined automatically. Similar, the algorithm to determine the browser type and the referrer hostname has been enhanced again. o The AddAgent/AddRefer directives in the configuration file has been renamed to HideAgent/HideRefer for consistency with the other directives. The old directives are also recognized. o The HideRefer directive has a mechanism to extract keywords from CGI parameters. With an appropriate setup in the configuration file (see the sample.conf file), the keywords used in search engines are now extracted in the list of referrer URLs. o All tables are now built using dynamic memory allocation. There are no limits for those tables other than the total virtual memory of your system. o Lookups for the URLs, sites, agents, and referrer URLs are done using a binary search algorithm rather than a linear search to speed up those lookups. Bug fixes: o A bug which caused core-dumps on 486 platforms due to byte-wise alignment has been fixed. o A bug with signed comparison of unsigned types in isHidden* functions has been fixed. o Comparison of referrer URLs and browser types now is case-insensitive. o A bug which did cause the graphs going crazy on sites with huge amount of data sent ( >100,000,000 KBytes) has been fixed. -------------------------------------------------------------------------- Changes in 2.01beta (patchlevel 1) -------------------------------------------------------------------------- This is a completely rewritten version of http-analyze. There are lot of new data structures, many enhancements, bug fixes, etc. The most important new features are: - Optional frames-based interface Allows for easy navigation and faster load time of the full statistics report. - Optional VRML model Integrated 3Dstats functionality generates a 3D model of the statistics summary (requires a VRML plug-in such as CosmoPlayer for Netscape/MSIE from SGI, available for IRIX and Windows). - Improved handling of logfile and history Recognizes Extended and Combined Logfile Format, generates user-agent and referrer statistics. Allows specification of start and end date of a period. Allows use of history to skip logfile entries which have been processed already now also in full statistics ("monthly") mode. - Support for (software) virtual servers Restricts statistics for given document root. Certain URLs and/or sites now can be ignored (slows http-analyze down by a considerable amount). - New directory structure Separate locations for global files, which now may be shared between periods, and yearly summaries for WWW, FTP and RealMedia access statistics. - Improved handling of cmdline interface Made options more consistent, allow more than only one logfile. -m is now default if neither -d nor -m is given. -3 creates a VRML model if the code is compiled in. -a ignores requests which required authentication. -e allows to skip previous periods already processed -I igndate and -E expdate allow to skip for certain periods. -f creates a Frames-based user interface. -V prints version and platform information. -F format defines the format of the logfile (auto/clf/dlf/elf) -P prolog creates a VRML model for a whole year. -R docroot allows you to restrict analysis to this DocRoot. -T TLD uses TLD for the two-letter ISO Top-Level-Domains. -U srvurl defines the suffix for hot links in the URL lists. -s allows to switch off parts of the statistics. -N allows to define the size of the Top Ten lists. -------------------------------------------------------------------------- Changes in 1.9e -------------------------------------------------------------------------- Bug fixes: o Fixed a bug which caused the size of documents to be added to the total amount of data transferred for HEAD requests. But why the hell do servers fill in the `reqsize' field in logfile entries if actually no data has been transferred?!? o Removed the colors for text in table headings due to popular demand. If you use older navigators, which do not support background colors in table cells, you will again see black colored text on white background (as it happened to be in older versions of ha). New features: o Added provisions to skip the first line in the logfile which is used by Netscape's new server generation to determine the format which should be used for log entries. To enable this functionality, define the macro NS_FASTTRACK in the Makefile. -------------------------------------------------------------------------- Changes in 1.9d -------------------------------------------------------------------------- New features: o Added a new entry "DefaultMode" in the configuration file to allow specification of the default operation mode of http-analyze (daily/monthly mode). o Now prints a detailed list for hidden items in the "Transfers by files" summary. o Added code for time measuring which can be activated by compiling http-analyze with the macro TIME_STATS defined. The undocumented option `-y' turns on time measurement. Bug fixes: o Fixed bugs in sort_by_item() and c_chart(), which caused http-analyze to dump core sometimes. o Now prints a warning if a third parameter is missing in an HideSys/HideURL entry in the configuration file instead of just dumping core :-) o Fixed a bug in addCountry() which caused .int and .nato being accounted for as unresolved sites. The function now uses separate counters for unresolved and unknown hostnames. o Raised size of input buffer to 10KB (again) to process even huge logfile entries like CGI scripts with thousands of parameters. o Now trailing slashes are not removed from URLs anymore. Since an incomplete URL causes a Code 302 (Redirect) response sent by the server, it does not get saved in the urltab. Therefore we do not need to take care for URLs without trailing slashes. -------------------------------------------------------------------------- Changes in 1.9c -------------------------------------------------------------------------- New features: o Added key colors to the header cells of the tables. Bug fixes: o Fixed a bug in the calculation of totals on last 12 month page (again). Fixed a bug where the graph on the same page always started at origin 0,0 regardless of the results for previous summary periods. o Changed the image functions to reserve a fix amount of space for the padding between the upper border of the image and the graphs and bars (4 pixels). -------------------------------------------------------------------------- Changes in 1.9b -------------------------------------------------------------------------- Bug fixes: o Now computes the correct totals in the last 12 month page, which got corrupted when code to smooth out the graph on the same page had been added. o Fixed a bug which caused core dumps on some platforms due to invalid text addresses. -------------------------------------------------------------------------- Changes in version 1.9 -------------------------------------------------------------------------- New features: o Added code for computation of average/max hits per day/hour. o Moved the list of countries from the list by Client domain into the monthly statistics page. Added a pie chart for the 12 top countries accessing the server. o Added code for accounting of user agents. See the note below. o The URLs in the top 30 URL list are now hot links, except for hidden items. o The macro PR_ROW has been splitted into independent macros to not exceed the ANSI C limit for a compilation unit anymore. o Now ignores noise resulting from entries of the next month's period caused by rotating the logfiles at 00:00 on the first day of a new month. New options: o The number of sites and URLs in the top lists now can be changed with an option (-N) or an entry in the config file. o The detailed lists of sites/URLs can now be created in a separate, private directory. You can deny access for the public by turning on authorization for this private directory. A private directory may be specified with the option `-p' or an entry in the config file. Speed enhancements mostly: o Removed all calls to functions in time-critical sections to gain maximum speed on RISC-based platforms. Added re- placements of more general, but slower stdlib functions. o To avoid redundant function calls, values which are computed already are now remembered. This avoids a lot of calls to strlen(3) and consorts. o Completely rewrote the logfile parsing algorithm to avoid strcmp(3), strtol(3) and friends (special thanks to for his suggestions to optimize the parser algorithm and the code he provided as an example). Bug fixes: o Fixed a bug which caused the list of countries to disappear when analyzing more than one month at once. o Fixed a bug in the calculation of total unique URLs. Items are now hidden after the computation of unique URLs and therefore each hidden item will now be accounted for separately. Thus, the number of unique URLs does not decrease with an increasing number of hidden items anymore. On the other hand, you can't comprise URLs under another URL anymore (this was an undocumented feature). o Fixed a bug in the calculation for the total hits of unresolved hosts. Instead of the number of total hits, http-analyze did summarize the unique number of sites for all unresolved hostnames. o Fixed a bug in the calculation of total kilobytes saved by cache. Previously, the amount of data saved by caches wasn't computed correctly for hidden items. o Fixed a bug where images (.gif, .jpg etc.) have not been added into the list of hidden items by default as intended. Images are now hidden by default under the item "[All images]". The `[' prevents the display in the top 30 URL list. You can turn off this feature with the new option `-x' (cmd line only). o Fixed a bug where homepages like "/some_dir/index.html" have been reduced to "/some_dir/" and didn't therefore merge with their equivalent "/some_dir". o Fixed a lot of minor bugs, added many explicit type casts to make lint shut up and all compiler warnings go away. -------------------------------------------------------------------------- Changes in 1.8 (first official release) -------------------------------------------------------------------------- o http-analyze now correctly obeys the -H homepage option. o The -D option has gone, since it doesn't make much sense to ignore logfile entries which are already ignored because http-analyze keeps track of the recent history. o Fixed a bug in the code which caused creation of the small icons with a wrong aspect ratio. o Fixed a bug in the conversion of hex codes in URLs which caused repetition of last few characters in URL lists. o Creates a monthly summary for a new month after the data for the first day has been collected. Previously, http-analyze did create a monthly summary not before the 4th of a new month. o http-analyze now conforms fully to the System V command line syntax rules (see intro(1)). The name of the logfile now may be specified as an optional argument (without -i). If no name is given, http-analyze reads stdin. If a default name is defined in the configuration file, a `-' may be used to ignore the default name and process the standard input. -------------------------------------------------------------------------- Version 1.7 was the first alpha version which had been released to testers outside RENT-A-GURU(TM). I think that there is no need to list the history of releases prior to this. --------------------------------------------------------------------------