## README for superstat (c)1999 Pete Nelson http://www.serversolved.com/superstat/ A Special Note: This program is no longer maintained by the author. This program may soon be replaced - no sooner did I get it posted online, when I started to notice it's shortcomings. For example -- once the meta file is generated, you have a list of pages visited and their number of hits; a list of clients and their number of hits; a list of 404's and their number of hits -- but none of these lists relate back to each other in any way. Also, it has been brought to my attention that very large log files (such as a gigabyte) can cause the application to run out of memory -- not too shocking, really. Superstat was one of the first perl programs I had written, and felt worth distributing. While it has it's limitations, I'm hoping that if nothing else, the program might give someone insite on ways perl & regular expresions can be utilized. COPYRIGHT INFO ============== You may freely use, edit, and distribute this program free of charge, under the condition that the copyrite information remains in tact. The sale of this program is expressly forbidden. WHAT DOES IT DO? ================ Superstat takes a server's access log, and generates statistics including top requests, most frequently visited directories, top clients, and even and hourly breakdown of activity. The most common way to use superstat is with the command: superstat --infile=access.archive --outfile=stats.html (If you don't have the 'Getopt::Long' module for perl, you would use the command 'superstat access.archive stats.html' to achive the same effect) It is highly advised that you NOT run superstat on an active log file! This is just a bad idea in general. You should alway rotate (archive) the access log or at least make a copy of the log to analyze. If you run this program on an active log file, it is quite possible your server will stop writing to it! Superstat also creates a metafile through the 'Storable' module for perl. (It's a very strait-forward usage of Storable - see Storable's documentation for more details.) The metafile is handy for changing the formatting of the output file without having to re-parse the log file again. To my knowledge, you should be able to change any option (with the exception of '--infile') and get the same end results from a metafile as you would from re-parsing the log. A NOTE ON MODULES ================= This program uses three modules by default. You can still analyze access logs and generate a web page without these modules, but you'll have to make a few simple changes, and you'll miss out on a few features! Storable - used to create a 'meta' file for faster loading. comment out the 'use Storable;' line, and add '$NO_STOR = 1;' Socket - used for performing hostlookups. comment out the 'use Socket;' line, and add '$NO_SOCK = 1;' Getopt::Long - used to parse command line options comment out the 'use Getopt::Long;' and add '$NO_OPT = 1;' (you can still use all the options by including them in the config file) INSTALLATION ============ This should be a cakewalk. Set '$DEFAULT_CONFIG_FILE' to where ever you keep superstat.cfg, and have '$null_dev' point to '/dev/null' (Unix) or 'nul' (DOS). USAGE ===== If you have the Getopt::Long module available, you have the following switches: --infile=filename the access log you want to analyze. If the name ends with '.meta', the file is imported as a metafile --outfile=filename The HTML output file. (You can specify 'STDOUT') --[no]graph use --graph to generate an hourly breakdown --[no]hostlookup do hostlookups for Top Clients (off if $NO_SOCK is set) --use_meta_default if {filename}.meta exists, use the meta file --silent | --quiet Does not send messages to STDOUT. (Setting 'outfile=STDOUT' automatically sets --silent, sending only the HTML to STDOUT instead) --hidden_dirs=list a quoted list of directorys that you don't want displayed on the generated page. '/cgi-bin/' is a good example. Note that the command line option does not overwrite the list in the config file, but appends them. There is no way to display directories listed as 'hidden' in the config file. --least_hits=n Don't show URLs that got less than n hits --max_dirs=n Show the top n dirs --max_clients=n Show the top n clients --server=servername The name of your server --body_def=s Anything here is placed between '' in the generated HTML (eg: bgcolor=white text=black) --table_def=s Like --body_def, but for generated tables --table_header_def=s Like --body_def, but between '' --title=s Sets the title for the generated page. Variables in this string are interpolated right be for the title is used, so you can use almost any variable in the program. See susperstat.cfg for more details. --img_dir=dir the directory containing 'just_green.gif' and 'just_red.gif' (realative to the generated page or DocumentRoot if preceded by a '/' --config=configfile the path to the config file If you don't have the Getopt::Long module, you will have to place any settings in the config file (isn't that better than typing them everytime, anyway?) superstat will also accept command line arguments for the infile & outfile - superstat access.110699 stats.html The command line switches take precedence over everything else. superstat --infile=access.110699 --outfile=stats.html HISTORY ======= I started writing this program for the City of Saint Paul, Minnesota (http://www.ci.saint-paul.mn.us/) for analyzing server access logs. We had several different servers - Netscape FastTrack, Apache, etc., and had a perl script that I found myself hacking for each different log file. This program was written to avoid having different programs for different logs. The logs we were analyzing were often 500k lines long, so it took some time to parse. The first versions sent the HTML output to STDOUT, but too many times the program would run for hours without any indication of what it was doing, and eventually I'd have to give it the ol' Ctrl-C, not knowing if it was still going, or had gotten stuck somewhere. This is when I decided that it needs to send messages to STDOUT so the operator can see what it's doing. The hostlookup feature was added since our servers have hostlookups turned off. Originally, I had the program doing hostlookups for EVERY CLIENT that was in the log. Andrew Moravec had the good sense to point out "Hey, why don't you only do it for the ten-or-so clients that you're going to display?" Thanks Andy! (and Duh! - why didn't I think of that?!?). This means that the hostnames are NOT stored in the metafile, and have to be looked up every time, but it's still WAY better than looking up all of them. The hourly breakdown graph I added mostly because I wanted to see if I could do it. 'just_red.gif' & 'just_green.gif' are just that - one pixel of red (or green). If you loose the original image (or never got it), it shouldn't be too hard to make a new one. The width of the image for the graph is deduced by this formula: $imgwidth = int((hits-this-hour) * (400 / (most-hits-an-hour) )); So the maximum length is 400 pixels. If the number of hits is greater than average, it uses red, otherwise, green. If there are no hits for a given hour, it's not displayed. The User-Agent info was added for curiousity. Being a City Government website, we wanted to make our pages accessable to everyone, not just those with the latest & greatest browsers. We used this information to show other depatments how important it was we not alienate people just for using an older browser. As it turned out, about 25% of our customers were still using level 3 browsers at the end of 1999. I also wanted to see what browser strings were out there besides Netscape & IE. So, I added an 'other_browsers' array. When I found one that was pretty common (like AOL), I added it to the list. I was a little supprised when I first ran the program with the other_browsers feature to find a browser named 'F**k You/2.0'. So I added a little regrep substitution to clean things like this up.