Using Monit to Monitor Linux Daemon, CPU, Filesystem, Network and Files on Ubuntu Server.
Services on any server must be operational and perform their duties effectively. It is critical to monitor the overall state of your server, including processes, programs, directories, files, and so on. This is known as system monitoring.
The goal is to tell the application what to monitor, and it will perform regular checks, sending alerts, or even run a script if any errors occur. These checks can range from monitoring CPU load to checking a file's MD5 or SHA1 checksum providing a plethora of options and careful monitoring. Automated monitoring is necessary since it ensures and maximizes the uptime and availability of system services on a server.
It enables system administrators to be notified of problems and resolve them as soon as possible with minimal disruption or downtime.
Install Monit from source code
GIT
To checkout Monit source code, use clone https://tildeslash@bitbucket.org/tildeslash/monit.git
Development Environment
You need to install some packages to be able to compile Monit.
$ sudo apt install \
zlib1g-dev \
libpam0g-dev \
libssl-dev \
libtool \
bison \
flex \
autoconf \
gcc \
make
Bootstrapping
If you build Monit from the git repository you should call ./bootstrap first, before continuing with the installation steps below. The bootstrap script requires that automake, autoconf, and libtool are installed on your system.
Installation
Monit utilizes the GNU auto-tools and provided the requirements above are satisfied, building Monit is conducted via the standard;
$ ./configure
make
make installThis will install Monit and the Monit man-file in /usr/local/bin and /usr/local/man/man1 respectively. If you want another location than /usr/local, run configure with the --prefix options and specify the install directory.
After installation Monit would start as a background service (if not, run the following commands.
$ sudo systemctl start monit
$ sudo systemctl enable monitand can be verified with the following command:
$ sudo systemctl status monit
OUTPUT
[sudo] password for snubmonkey:
● monit.service - Pro-active monitoring utility for unix systems
Loaded: loaded (/etc/systemd/system/monit.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-08-12 18:23:05 UTC; 20h ago
Docs: man:monit(1)
https://mmonit.com/wiki/Monit/HowTo
Main PID: 1804 (monit)
Tasks: 2 (limit: 4265)
CGroup: /system.slice/monit.service
└─1804 /usr/local/bin/monit -IConfigure Monit
The following step is to configure Monit to monitor specific things. On Ubuntu and Debian-based systems, the Monit configuration file is located at /etc/monit/monitrc. We recommend that you make a copy of this file before applying any change – $ sudo cp monitrc monitrc.bck
A. Run as a daemon:
The first directive declared in the file is:
##
#
set daemon 120
with start delay 120
#
##It is used to indicate the refresh time for checking the services, that is to say in this case that the check will take place every 120 seconds, or 2 minutes. This command also indicates that Monit is running in daemon mode. with start delay 120 delays the first check by 2-minutes (by default Monit check immediately at start up)
B. Logs:
The second directive concerns Monit' logs, which by default are stored in the file "monit.log" in "/var/log/". By changing the value of this parameter, you change the log file.
##
#
set log /var/log/monit.log
#
##C. Identification and status files:
The third directive specifies the path to the file containing Monit's unique identifier. Do not change this setting.
##
#
# set idfile /var/.monit.id
#
##The fourth directive specifies the path to Monit's state file, which is a temporary file where Monit stores information. Do not change this setting.
##
#
# set statefile /var/.monit.state
#
##
D. SMTP server (s) for alerts:
Since Monit offers email alerts, it must be configured by adding this to the monitrc file; here we are using Gmail as a Relay.
##
#
set mailserver smtp.gmail.com port 587
username "auth_user" password "auth_password"
using tls
#
##Note: Replace "auth_user" and "auth_password" with your Gmail user name and password. To specify multiple SMTP servers, separate them with a comma.
Remark: On version 5.28.1 adding V1 to TLS will not work; avoid adding any TLS version.
E. Customize emails format:
Sending alert emails is good, personalizing them is even better!
Monit allows you to do this using the "mail-format" directive:
##
#
set mail-format {
from: Monit <monit@$HOST>
subject: Monit alert ⚠️ -- $EVENT $SERVICE
message: $EVENT Service $SERVICE
Date: $DATE
Action: $ACTION
Host: $HOST
Description: $DESCRIPTION
}
#
##
You notice the presence of different variables, like "$SERVICE", "$EVENT", "$DATE", "$ACTION" and "$HOST". They make it possible to ensure that the alert message adapts to the detected error.
• $SERVICE: Name of the service indicated in monitrc.
• $EVENT: Type of event.
• $DATE: The current date.
• $ACTION: The name of the action performed, which corresponds to the action defined in monitrc for this service.
• $HOST: The name of the host where monit is installed.
"From" corresponds to the sending address, "reply-to" corresponds to the address to reply, "subject" to the subject of the message, and "message" to the message contained in the email.
F. Email recipient:
Do not forget to specify to whom the alerts should be sent. We use the "alert" directive as such:
##
#
set alert admin@Xxx.com
#
##G. Alert storage:
If you do not want to receive any email alert, you can tell Monit to store the alerts in a directory so that you can access them later. The "eventqueue" directive should be used for this and it is already enabled by default.
##
#
#set eventqueue
#basedir /var/lib/monit/events # set the base directory where events #will be stored
#slots 100 # optionally limit the queue size
#
##H. Web interface Activation :
Monit offers a web interface for viewing the data collected. However, it must be enabled and configured in the monitrc file since this is not done by default.
##
#
# set the port Monit should bind to and listen on.
set httpd port 2812 and # Monit is usually setup on port 2812.
# use address localhost # only accept connection from localhost (drop if you use M/Monit)
allow 192.168.1.201 # allow localhost to connect to the server and
allow <auth_user>:<auth_password> # require user 'admin' with password 'monit'
# allow @monit # allow users of group 'monit' to connect (rw)
# allow @users readonly # allow users of group 'users' to connect readonly
with ssl { # enable SSL/TLS and set path to server certificate
pemfile: /etc/ssl/certs/monit.pem
selfsigned: allow
}
#
##
#
You can now use http://localhost:2812/ to access Monit's web interface after providing your username and password as credentials. You might need to use double quotes around the password if it contains special chars such as "p@ssw:r#".
PORT set a port so Monit should be bound to and listened on. Monit is usually setup on port 2812 but you can pretty much use any available port.
set httpd port 2812 and
allow <auth_user>:<auth_password> # require user 'admin' with password 'monit'
ADDRESS make Monit listen on a specific interface only. For example, if you don't want to expose Monit's web interface to the network, bind it to localhost only. Monit will accept connections on any addresses if the ADDRESS option is not used.
set httpd port 2812 and
use address 127.0.0.1
allow <auth_user>:<auth_password> # require user 'admin' with password 'monit'
Host and network allow list allow Monit to keep track of which hosts and networks are authorized to connect. You can add as many hosts as you want to, but only hosts with a valid domain name or IP address are allowed. Monit will query a name server to check any hosts trying to connect. If a host (client) is trying to connect, but cannot be found in the access list or cannot be resolved, Monit will shutdown the connection to the client promptly.
Control file example:
set httpd port 2812 and
use address localhost
allow localhost
allow my.quantium.server.com
allow 10.1.1.1
allow 192.168.1.0/255.255.255.0
allow 10.0.0.0/8Clients, not mentioned in the allow list and trying to connect to Monit will be denied access and their IP address will be logged.
Notes: To access Monit from an outside network, just comment out or remove any allow option, such as below:
set httpd port 2812 and
#allow localhost
#allow my.quantium.server.com
#allow 10.1.1.1
#allow 192.168.1.0/255.255.255.0
#allow 10.0.0.0/8
allow <auth_user>:<auth_password>a. Create New Certificate
Selfsigned self-signed certificates are rejected by default. Use this option to allow self-signed certificates.
Warning: Using a self-signed certificate in production is not recommended, as in such case the client cannot verify that it talks to the correct server (vulnerable to man-in-the-middle attacks and DNS-hijacking).
Run these commands to generate the pemfile :
# Generates the private key and the certificate
openssl req -new -x509 -days 365 -nodes \
-config ./monit.cnf -out /etc/ssl/certs/monit.pem \
-keyout /etc/ssl/certs/monit.pem
# Generates the Diffie-Hellman Parameters
openssl dhparam -2 2048 >> /etc/ssl/certs/monit.pem
# Set mode
chmod 600 /etc/ssl/certs/monit.pem
# Prints out the certificate information
openssl x509 -text -noout -in /etc/ssl/certs/monit.pemBefore Monit is started the first time, we need to test the control file for syntax errors:
$ sudo monit -t
[sudo] password for snubmonkey:
OUPUT
Control file syntax OKMonit will output an error message to the console if an error occurs, along with the line number in the control file where the issue has happened.
Finally, restart your Monit service to take the changes into account.
$ sudo systemctl restart monit
or
$ sudo monit restart all
After you logged into Monit, you should see the Monit default dashboard as shown below.
Click on any item to view its system status on your server.
Here, we monitoring the network link.
G. Monitor services:
Monit is now set up and ready to send you alerts whenever an issue occurs, but you must first tell it what to look for. You'll see it's really straightforward and intuitive. Of course, we will not be able to observe all types of checks in this tutorial because they are unlimited and dependent on your needs.
a. The different types of action:
You need to tell Monit what to do when it detects an error, using the following actions:
• alert: Send an alert message,
• start: Start the service and send an alert message,
• stop: Stops the service and sends an alert message,
• restart: Restarts the service and sends an alert message,
• exec: Execute a script and send an alert message,
• unmonitor: Disables monitoring of the service and sends an alert message,
• monitor: Activate monitoring.
H. Configuration Examples
After changing configurations, make sure to 'sudo monit -t'
and run 'sudo monit reload' to check for errord and bring them in effect.
While testing you can manually turn services on/off
and check the monit report from the web interface.
1. System resources
Check general system resources such as load average, cpu and memory usage.
The goal is to send an alert by email if one of the following conditions is true:
check system $HOST
if loadavg (5min) > 3 then alert
if loadavg (15min) > 1 then alert
if memory usage > 80% for 4 cycles then alert
if swap usage > 20% for 4 cycles then alert
# Test the user part of CPU usage
if cpu usage (user) > 40% for 2 cycles then alert
# Test the system part of CPU usage
if cpu usage (system) > 31% for 2 cycles then alert
# Test the i/o wait part of CPU usage
if cpu usage (wait) > 80% for 2 cycles then alert
# Test CPU usage including user, system and wait. Note that
# multi-core systems can generate 100% per core
# so total CPU usage can be more than 100%
if cpu usage > 200% for 4 cycles then alert
2. Apache webserver - Process Monitoring
Consider the following scenario: our server runs an Apache2 web server, and we want to avoid any downtime, thus we'll need to monitor the service and restart it if it fails.
To do so, we need to check the "apache2" process by specifying its PID file after "with pidfile" so that Monit knows which process to monitor.
To find out if the Apache service is still available on port 80 for our web server, we'll use the HTTP protocol to test the webserver again on port 80.
In the event of a problem, apache2 will be restarted.
# apache2 is the name of the service
check process apache2 with pidfile /var/run/apache2.pid
# Start/Stop action
start program = "/bin/systemctl start"
stop program = "/bin/systemctl stop"
# Test server on port 80
if failed host snunbmonkey.com port 80 protocol http then restart
3. Monitor SSHD service
Here is how to check that the sshd process corresponding to the SSH service is active:
# apache2 is the name of the service
check process apache2 with pidfile /var/run/apache2.pid
# Start/Stop action
start program = "/etc/init.d/ssh start"
stop program = "/etc/init.d/ssh stop"
# Test server on port 22
if failed port 22 protocol ssh then restart
4. Monitor checksum SHA1
Another type of monitoring that can be performed is checking the MD5 or SHA1 sum of a file, which is a unique fingerprint of the file in its current state. As soon as there is the slightest modification within the monitored file, the sum will necessarily change and therefore an alert will be triggered by Monit.
With this, you can ensure that no changes are made to sensitive system files, especially configuration files. We will take the example of the "sshd_config" file which is the configuration file for the SSH service.
There are two ways to accomplish this using Monit. The first is to calculate the SHA1 sum first and then tell Monit to check that it is equal to that all of the time. The second is to check if the total has changed and if so, you'll get notified.
We will see both approaches.
### 1
# sshd_config is the name of the service
check file apache_bin with path /etc/ssh/sshd_config
if failed checksum and
expect the sum 8f7f419955cefa0b33a2ba316cba3659 then alert
### 2
# sshd_config is the name of the service
check file sshd_config path /etc/ssh/sshd_config
#Send an aletr if the SHA1 sum has changed
if changed sha1 checksum then alert
5. Monitor permissions
It is also possible to control the permissions on a file, which can be interesting to do on configuration files, in order to be sure that nobody makes changes thanks to the calculation of the sum and that nobody modifies the permissions thanks to the control over permissions.
# sshd_config is the name of the service
check file sshd_config path /etc/ssh/sshd_config
# Send an alert is the SHA1 sum has changed
if changed sha1 checksum then alert
# Send an aletr if the file' permissions if different fromn 644
if failed permission 644 then alert
6. Network ping test
Monit can perform a network ping test by sending ICMP echo request datagram packets to a host and wait for the reply. Here we will watch a host, that is supposed to be offline.
check host VirtualServer#445 with address 192.168.1.111
if succeeded ping then alert
7. Alert repeatedly
By default, Monit generates an alert only twice. Once when a test fails and other when it succeeds.
If you wish to get repeated alerts for the same failing test, then use the following configuration.
set alert email@example.com with reminder on 10 cyclesThe above configuration is a global setting, that sends all alerts after every 10 checks/cycles.
To repeat alerts only for specific tests, specify the alert directive within the particular test criteria like the following:
##
#
...
alert email@example.com with reminder on 1 cycles
if space usage > 80% for 5 times within 5 cycles then alertIf you would like to be reminded fewer times but repeatedly then use something like "reminder on 10 cycles".
Check Monit status
You can also check the status of all the services using the following command:
$ sudo monit status
[sudo] password for snubmonkey:
Monit 5.28.2 uptime: 2h 46m
OUTPUT
System 'WebServer'
status Resource limit matched
monitoring status Monitored
monitoring mode active
on reboot start
load average [0.49] [0.77] [1.60]
cpu 9.0%usr 3.5%sys 0.0%nice 1.0%iowait 0.0%hardirq 0.1%softirq 0.0%steal 0.0%guest 0.0%guestnice
memory usage 747.3 MB [40.0%]
swap usage 0 B [0.0%]
uptime 3d 0h 4m
boot time Thu, 12 Aug 2021 18:49:11
filedescriptors 2400 [0.0% of 2147483647 limit]
data collected Sun, 15 Aug 2021 18:53:00
Process 'sshd'
status OK
monitoring status Monitored
monitoring mode active
on reboot start
pid 1881
parent pid 1
uid 0
effective uid 0
gid 0
uptime 3d 0h 3m
threads 1
children 6
cpu 0.0%
cpu total 0.0%
memory 0.2% [3.5 MB]
memory total 1.3% [24.9 MB]
security attribute unconfined
filedescriptors 5 [0.5% of 1024 limit]
total filedescriptors 65
read bytes 290.2 B/s [262.9 MB total]
disk read bytes 0 B/s [51.1 MB total]
disk read operations 0.8 reads/s [487283 reads total]
write bytes 3.7 B/s [12.0 MB total]
disk write bytes 0 B/s [2.4 MB total]
disk write operations 0.0 writes/s [33180 writes total]
port response time 34.991 ms to localhost:22 type TCP/IP protocol SSH
data collected Sun, 15 Aug 2021 18:53:00
Process 'OpenVPN'
status OK
monitoring status Monitored
monitoring mode active
on reboot start
pid 1814
parent pid 1
uid 65534
effective uid 65534
gid 65534
uptime 3d 0h 3m
threads 1
children 0
cpu 2.0%
cpu total 2.0%
memory 0.2% [4.4 MB]
memory total 0.2% [4.4 MB]
security attribute unconfined
filedescriptors 7 [0.7% of 1024 limit]
total filedescriptors 7
read bytes 148.2 kB/s [14.0 GB total]
disk read bytes 0 B/s [2.6 MB total]
disk read operations 119.2 reads/s [13515753 reads total]
write bytes 4.4 kB/s [1.5 GB total]
disk write bytes 0 B/s [0 B total]
disk write operations 33.4 writes/s [8974587 writes total]
data collected Sun, 15 Aug 2021 18:53:00
Yeeeeahhh!!!
You have successfully installed and configured Monit monitoring tool.
Monit is capable of doing much complex things like pinging network hosts, running external scripts, and test the output for a certain value. We cannot urge you strongly enough to check the full documentation at -
https://mmonit.com/monit/documentation/monit.html