Monitoring Web server logs (Apache, NGINX) with Netdata.

Collects webserver/caching proxy metrics via access log file

ps: we assume Netdata is fully installed, active, and running.

Web server logs have been around for years now, generating log files that record every accessed website and API in real-time. All these web servers' log files are mostly just filling your disks, rotating every night without any use whatsoever. That's where the magic starts. Netdata turns this "useless" log file, into a powerful performance and health monitoring tool, capable of detecting, in real-time, the most common web server problems as such: too many internal server errors, too many bad requests, too many redirect unreasonably slow responses, or too few successful responses just to name few.

Netdata supports Apache, Nginx, Lighttpd, and Tomcat and as we've mentioned earlier, in order to get real-time data from a web server API, your web server needs to provide those.

Configuration

Edit the python.d/web_log.conf configuration file using edit-config from the Netdata config directory, which is usually located at /etc/netdata.

$ cd /etc/netdata    # Replace this path with your Netdata config directory, if any different

sudo ./edit-config python.d/web_log.conf

Restart the service: sudo systemctl restart netdata

Remember, Netdata has a powerful web_log plugin, capable of incrementally parsing any number of web server log files. This plugin is automatically started with Netdata and comes, pre-configured, for finding web server log files on popular distributions. Its configuration is located at /etc/netdata/python.d/web_log.conf and will look like this:

#
##
###
nginx_log:
  name  : 'nginx_log'
  path  : '/var/log/nginx/access.log'

apache_log:
  name  : 'apache_log'
  path  : '/var/log/apache/other_vhosts_access.log'
  #
  ##
  ###

Separately monitoring each access log?

Again the idea here is to change the names nginx_log, apache_log to whatever you wish them to be. The goal is to have different names as such:

#
##
###
# nginx log on various distros

nginx_log_1:
  name: 'Saturn'
  path: '/var/log/nginx/saturn_access.log'
#
#
nginx_log_2:
  name: 'Uranus'
  path: '/var/log/nginx/uranus_access.log'
##
#
#
nginx_log_3:
  name: 'Neptune'
  path: '/var/log/nginx/neptune_access.log'
#
#
nginx_log_4:
  name: 'Pluto'
  path: '/var/log/nginx/pluto_access.log'
  #
  ##
  ###

Important
Keep in mind that Netdata runs as user netdata. So, make sure user netdata has access to the logs directory and can read the log file.

Charts

Once you have all log files configured and Netdata restarted, for each log file you will get a section at the Netdata dashboard, with the following charts.

That's a wrap!!!
We hope you have found this post as useful and informative as we do.
Keep on learning and sharing knowledge.