Log Files Overview Last Updated: Thu, 02 Aug 2018 14:36:21 GMT
Log files overview Last updated: Thu, 02 Aug 2018 14:36:21 GMT
EZproxy offers institutions multiple customizable log formats to record user activity. These logs can be helpful in identifying levels of proxy usage, security concerns, and general operational details. Customization through directives allows EZproxy administrators to refine what information they receive and how often they receive it. Log types
The log files generated by EZproxy provide information only about remote users’ activity.
The types of logs that can be generated by EZproxy are defined below. The settings described are the default settings for standalone EZproxy that are set in the config.txt file when EZproxy is first downloaded. Many of these options can be customized to fit the needs of your institution. For more information about customization options, see the tabs for each individual log and the related directives for these log types.
Audit logs
Audit logs are daily logs that contain information about your users’ access to EZproxy as specified by the conditions you set with the Audit directive. By default, the config.txt file you download with EZproxy is set to Audit Most, which will record most login events, usage limits, and others (see the Audit logs tab for more details). These log files are retained in a directory named audit which is a subdirectory of the EZproxy installation directory; you cannot redirect these files to be saved in another directory. However, you can customize your audit logs by specifying which events you would like EZproxy to record and how long you would like EZproxy to retain these files. Audit events can be viewed by logging in to the EZproxy administration page where you can search all files in audit directory. They can also be viewed in your operating system by opening the audit folder within the EZproxy installation directory. The information included in these logs can be helpful in monitoring and resolving security issues.
EZproxy logs
EZproxy logs are monthly logs that contain large amounts of data about the information sent between EZproxy and all the database providers you have configured in your config.txt file. These logs are retained by default in the EZproxy installation directory and named ezpyyyymm.log. You can change the name of this log, the directory where the information is retained, designate the periods of time over which an individual file collects information, and limit the type of information recorded in these logs by including certain directives in your config.txt file. The active log can be viewed and searched from your EZproxy administration page, and all ezproxy logs can be accessed directly from the EZproxy installation directory. The information included in these logs can be used to assess EZproxy usage and evaluate and resolve potential security threats.
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 1 Printed: Wed, 21 Nov 2018 20:12:48 GMT message.txt logs
The messages.txt file records daily operational information about each time EZproxy was started or stopped and fatal or non-fatal errors. This log also contains any messages resulting from debugging scripts generated by the Debug directive when included in your config.txt file. The messages.txt file is retained by default in the EZproxy installation directory and named messages.txt. You can change the name of this log, the directory where the information is retained, and designate periods of time over which an individual file collects information by editing the messages.txt log itself. The active log can be viewed and searched from your EZproxy administration page, and all messages.txt files can be accessed from your EZproxy installation directory. These logs can be useful in troubleshooting problems with EZproxy and verifying system details.
SPU logs
Starting Point URL logs record information about each time a starting point URL is clicked. Standalone EZproxy is not configured to create SPU logs by default when downloaded. When configured without specifying directory locations where SPU logs should be saved, EZproxy stores these files in the EZproxy installation directory with the name spu.log. You can specify different locations for these logs to be stored and refine the information they collect with the same configurations as EZproxy logs. These log files can only be viewed through your operating system viewer. This log can be used to determine how often remote users access certain resources.
Audit logs
Adding the Audit directive to your config.txt file will command EZproxy to create audit logs when specified events occur and save these files in the EZproxy installation directory. Individual files will be named the year, month, and day that the event occurred (e.g. 20140512.txt). The most common use of the Audit directive to command EZproxy to create audit logs is as follows:
Audit Most
This directive statement will create a log when any of the following events occur:
• A user is denied accessed EZproxy • A user successfully logs in to EZproxy • A user has a failed attempt to log in to EZproxy • An intrusion attempt based on the IntruderIPAttempts or IntruderUserAttempts directive occurs • General system activities, such as system startup, occur • An unauthorized user attempts to access the administrative features of EZproxy • An event resulting from the UsageLimit directive occurs This is the default configuration in the config.txt file that is downloaded with EZproxy. For more details about further customization of the audit logs, see Audit.
Audit logs can be accessed from the EZproxy admin page at any time. Logs that have been purged as a result of the AuditPurge directive will not be accessible.
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 2 Printed: Wed, 21 Nov 2018 20:12:48 GMT Suggested Configuration
By retaining the default Audit Most directive statement in your config.txt file, you will have the most commonly assessed security events recorded to your audit logs. To limit the amount of storage space your audit logs take up, the AuditPurge directive is also configured by default to 7 following the Audit statement so EZproxy will delete files after a specified period of time. This will keep only the audit logs for the current day plus the previous week and delete any files older than 7 days. OCLC suggests increasing the AuditPurge time period to 180, so that your config.txt file looks as follows:
Audit Most AuditPurge 180
This change to the AuditPurge directive will cause EZproxy to retain the audit file for the current day plus the audit files of the previous 180 days, and delete any file older than 180 days. Retaining audit files for longer periods of time will provide you with a larger pool of information for review if you should need to access it. You can increase or decrease the purge number as you see fit to save disc space or ensure that you have the data you need should reporting requests or a security breach require you to reference it. When determining your how long to set your AuditPurge, consult with your IT department to ensure that your retention schedule complies with institutional policies for security and reporting purposes.
Ezproxy.log
EZproxy will automatically generate EZproxy logs and save them in a file named ezproxy.log in the directory where EZproxy is installed. The default command used to format data collected in the EZproxy log is as follows:
LogFormat %h %l %u %t “%r” %s %b
This will generate the following data in your EZproxy log:
132.174.1.1 - - [14 /Mar/2014:09:39:18 -0700] “GET http://www.somedb.com:80/index.html HTTP/1.0” 200 1234
The following table breaks down this line of data:
CORRESPONDING VALUE IN FIELD DESCRIPTION EXAMPLE
The IP address of the Host %h 132.174.1.1 accessing EZproxy
The remote username obtained by %l - identd, ifidentd is not used, the a - will be recorded in your EZproxy log
The username or session identifier, %u - based on other config.txt options
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 3 Printed: Wed, 21 Nov 2018 20:12:48 GMT CORRESPONDING VALUE IN FIELD DESCRIPTION EXAMPLE
[14/Mar/ The date and time the request was %t 2014:09:39:18-0700] made
The complete http request sent to the remote server; this field is “GET contained in quotation marks so it is "%r" http://www.somedb.com:80/ parsed as one piece of data even index.html HTTP/1.0” though it contains spaces, since spaces are generally a signal that a new field of data is beginning
The HTTP numeric status of the request (see LogFormat Status %s 200 Codes for more information about these numbers)
%b 1234 The number of bytes transferred
Note: If EZproxy is not able to collect the data denoted by a particular field, it will insert a dash for the missing information.
For more details about these and additional fields that can be used to further customize this directive, see LogFormat.
Suggested Configuration
The default configuration in the standalone EZproxy config.txt file will provide you with monthly log files that contain data about all data transfers and requests sent through your instance of EZproxy. Depending on use levels, you might want to configure EZproxy to maintain daily instead of monthly log files so you can more quickly identify the location of the information you may need by date. This suggested configuration does not contain any LogFilters, but you may also want to consider whether the inclusion of filters to limit the volume of data collected would make your ezproxy logs more manageable and provide more focused information. See LogFilter for more details about this option. The following configuration will change your EZproxy logs to daily files instead of monthly:
LogFormat %h %l %u %t "%r" %s %b LogFile -strftime ezp%Y%m%d.log
Messages.txt
EZproxy will automatically generate messages.txt logs and save them in files of the same name in the EZproxy installation directory. While there are fewer dedicated directives to customize the information recorded in messages.txt, many directives used to set maximum limits will record information to messages.txt when these limits are reached. A list
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 4 Printed: Wed, 21 Nov 2018 20:12:48 GMT of the directives that record this type of information can be found in the Related Directives tab, and clicking the link will provide you with additional information.
The data contained in the messages.txt file, will look similar to the following when you start or restart your instance of EZproxy, if you have no errors:
2018-01-12 12:41:07 EZproxy 5.7.42 GA [SOURCE:5.7.41_SOURCE] [Windows] [2017-06-26T19:46:13Z] 2018-01-12 12:41:07 For further information or updates, visit http://www.oclc.org/ezproxy/ or contact [email protected]. 2018-01-12 12:41:07 MaxVirtualHost (MV) changed from 200 to 1000 2018-01-12 12:41:07 Server name localhost 2018-01-12 12:41:07 The use of a server name with no periods will cause some databases to fail. 2018-01-12 12:41:07 Server http URL http://localhost:2048 2018-01-12 12:41:07 Proxy by port 2018-01-12 12:41:07 SSL is not configured so https proxying will not be available 2018-01-12 12:41:07 To configure EZproxy as a service that runs in the background and 2018-01-12 12:41:07 starts when this system is booted, press CTRL/C and issue the commands: 2018-01-12 12:41:07 C:\ezproxy\ezproxy.exe -si 2018-01-12 12:41:07 net start EZproxy 2018-01-12 12:41:07 You may not see any additional messages, but EZproxy is running. 2018-01-12 12:41:07 If you press CTRL/C or close this window, EZproxy will stop running. 2018-01-12 12:41:07 Thank you for your purchase of this licensed copy of EZproxy. 2018-01-12 12:41:07 EZproxy was last able to validate the license on 2018-01-12 12:41:07. 2018-01-12 12:45:53 Guardian starting EZproxy
Much of the data in messages.txt is written in a message format to convey to you details about or problems with your config.txt or user.txt that may prevent your EZproxy from working. If you have made changes to configurations in either of these files, but EZproxy is not behaving as expected, messages.txt is a good place to check to determine if any of those changes were entered incorrectly and caused the error.
Suggested Configuration
As with the other log file configurations, OCLC suggests that you rename the messages.txt files to rotate daily so your information is retained in logs named with the date. This will make it easier to identify the location of information you might need by date. Enter the following line as the first line of your messages.txt file to create messages logs named with the year, month, and day on which they are created:
MessagesFile -strftime messages-%Y-%m-%d.txt
Spu.log
Adding the LogSPU directive to your config.txt file will provide you with data to determine which starting point URL’s remote users have accessed. Adding additional configuration fields, the same used in LogFormat, allows you to refine the information recorded along with each starting point URL and include data such as date and time accessed and user group.
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 5 Printed: Wed, 21 Nov 2018 20:12:48 GMT There is no default for LogSPU in the config.txt file you download with EZproxy, so you must determine which information you would like to gather in this log before you add it to your config.txt file. You can customize your spu.log using the same fields accepted by the LogFormat directive.
Suggested Configuration
OCLC suggests, and implements for Hosted EZproxy sites, the following LogSPU configuration:
LogSPU -strftime spu%Y%m%d.log %{%Y/%m/%d:%H:%M:%S}t %h %u %{ezproxy-spuaccess}i %v %U
This configuration will provide you with the following information:
FIELD DESCRIPTION
The date and time the request was made in the format: %{%Y/%m/%d:%H:%M:%S}t year/month/day:hours:minutes:seconds
%h The IP address of the host accessing EZproxy
Inserts “proxy” if the remote user’s access to the URL was proxied; “local” if user was redirected due to %{ezproxy-spuaccess}i ExcludeIP; or “unknown” if the URL is not known to EZproxy and Option Redirect appears in the config.txt
The virtual web server’s hostname (e.g. %v www.somedb.com)
%U URL requested (e.g. http://www.somedb.com)
This will provide information to help you determine which resources are accessed most frequently and by whom. While this information can be useful for making collection decisions, we suggest that you also configure your ezproxy.log to record additional data that could be valuable in assessing your remote patrons’ use of proxied resources because the spu.log will contain only information about the user’s first access to a resource, not what additional information they access while at that URL.
Log file analysis
While EZproxy has the ability to record large amounts of usage data in the three logs described above, it does not provide a way to analyze that information at this time. Many users have discussed the ways in which they analyze their log files on the EZproxy e-mail list. Archives of this list can be searched at ls.suny.edu/read/?forum=ezproxy. A good search term is “analysis” to locate messages on this topic.
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 6 Printed: Wed, 21 Nov 2018 20:12:48 GMT Tools for Analysis
OCLC does not endorse any particular web server log file analysis program; however, the following links provide quick access to a variety of programs that have been discussed on the e-mail list.
SERVICE URL
AWStats http://awstats.sourceforge.net/
EzPAARSE http://www.ezpaarse.org/
FastStats Analyzer http://www.mach5.com/products/analyzer
Sawmill http://www.sawmill.net
The Webalizer http://www.mrunix.net/webalizer
XpoLog http://www.xpolog.com
If you would like to recommend that a program be added to this list, please send an email message to [email protected].
EZproxy-specific status codes
The LogFormat field %s records the HTTP numeric status of the request. This field can return standard HTTP status codes, but may also record special status codes under the circumstances specified in the table below.
CODE MEANING
Recorded on access attempts after the IntruderReject 597 threshold has been exceeded.
Attempt made to access an unauthorized EZproxy 598 administration function.
Starting point URL referenced a host that EZproxy is not 599 configured to proxy.
Error occurred receiving the request from the remote 900-905 & 907 user's browse.
Error occurred forwarding the request from the user's 906 browser to the remote server.
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 7 Printed: Wed, 21 Nov 2018 20:12:48 GMT CODE MEANING
Error occurred interpreting an administration request 950 from the remote user.
Shibboleth login failure due to metadata misconfiguration 997 (recorded as 999 in early beta releases of Shibboleth support).
Access denied based on a DenyIfRequestHeader 998 directive in config.txt.
Access attempted from an IP address that is in a 999 RejectIP address range.
Related directives
The following directives can be used to customize your log files. For more details about each, click on the directive name.
Note: If the directive is not linked to a page dedicated to this directive, it should be listed in EZproxy config.txt directives.
Commonly used data directives related to EZproxy and SPU logs
DIRECTIVE PURPOSE
Designates the name of the EZproxy log and whether to LogFile sort information by date into dated files
Designates that certain types of data NOT be collected in LogFilter EZproxy logs
Used with EZproxy logs to customize the type of LogFormat information collected in log (the same fields can be used to customize SPU logs)
LogSPU Logs usage information of Starting Point URLs
Specifies that the session identifier be logged in the Option LogSession EZproxy or SPU log for every EZproxy session opened by a remote user
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 8 Printed: Wed, 21 Nov 2018 20:12:48 GMT DIRECTIVE PURPOSE
Specifies that the username provided when logging in to Option LogUser EZproxy be logged in the EZproxy or SPU log for each user who logs in to EZproxy
Commonly used security directives related to audit and messages logs
The directives in this table generally impact security or debugging settings in EZproxy and record details about those settings to either the audit or messages logs. This information can be useful when troubleshooting problems with EZproxy or dealing with security issues.
DIRECTIVE PURPOSE
Enables intruder detection and records login attempts to IntruderIPAttempts the audit logs (with Audit Most) from IP addresses using invalid credentials; qualifiers define the limits
Specifies the maximum number of times to record in messages.txt failed log in attempts from the same IP IntruderLog address; the Audit directive will override this and failed login attempts will be recorded to the audit logs
Enables intruder detection and records login attempts to IntruderUserAttempts the audit logs (with Audit Most) from usernames using invalid passwords; qualifiers define the limits
Specifies the maximum number of HTTP transfers that MaxConcurrentTransfers can be active at the same time; if the limit is reached, an error will be recorded in messages.txt
Specifies the maximum number of user sessions that MaxSessions can be active at the same time; if the limit is reached, an error will be recorded in messages.txt
Specifies the maximum number of virtual web servers that EZproxy can create for proxying remote web MaxVirtualHosts servers; if the limit is reached, an error will be recorded in messages.txt
Designates a new name for the messages.txt file when MessagesFile placed as the first line in the messages.txt file
Logs the number of times EZproxy performs referring Option LogReferer URL authentication; should only be enabled during Referring URL Testing
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 9 Printed: Wed, 21 Nov 2018 20:12:48 GMT DIRECTIVE PURPOSE
Enables the recording of new peak maximum values for Option RecordPeaks operational functions such as number of user sessions, concurrent transfers, and virtual hosts
Logs any transformations attempted or performed by the LogSPUEdit SPUEdit directive to the messages.txt
Note: Option LogSAML is a directive that is used during Shibboleth authentication for debugging. This directive doe not record information to the messages.txt log as other debugging directives do. Instead it creates and logs those details in a separate XML file that is stored in the EZproxy installation directory.
Tags recommended by the template: article:topic
https://help.oclc.org/Library_Management/EZproxy/Manage_EZproxy/Log_files_overview 10 Printed: Wed, 21 Nov 2018 20:12:48 GMT