Using Monit to Monitor Linux Daemon, CPU, Filesystem, Network and Files on Ubuntu Server.

The goal is to tell the application what to monitor, and it will perform regular checks, sending alerts, or even run a script if any errors occur. These checks can range from monitoring CPU load to checking a file's MD5 or SHA1 checksum providing a plethora of options and careful monitoring. Automated monitoring is necessary since it ensures and maximizes the uptime and availability of system services on a server.
It enables system administrators to be notified of problems and resolve them as soon as possible with minimal disruption or downtime.

Install Monit from source code

GIT

To checkout Monit source code, use
clone https://tildeslash@bitbucket.org/tildeslash/monit.git

Development Environment

You need to install some packages to be able to compile Monit.

$ sudo apt install \
      zlib1g-dev \
      libpam0g-dev \
      libssl-dev \
      libtool \
      bison \
      flex \
      autoconf \
      gcc \
      make

Bootstrapping

If you build Monit from the git repository you should call ./bootstrap first, before continuing with the installation steps below. The bootstrap script requires that automake, autoconf, and libtool are installed on your system.

Installation

Monit utilizes the GNU auto-tools and provided the requirements above are satisfied, building Monit is conducted via the standard;

$ ./configure
make
make install

This will install Monit and the Monit man-file in /usr/local/bin and /usr/local/man/man1 respectively. If you want another location than /usr/local, run configure with the --prefix options and specify the install directory.

After installation Monit would start as a background service (if not, run the following commands.

$ sudo systemctl start monit
$ sudo systemctl enable monit

and can be verified with the following command:

$ sudo systemctl status monit

OUTPUT 

[sudo] password for snubmonkey:
● monit.service - Pro-active monitoring utility for unix systems
     Loaded: loaded (/etc/systemd/system/monit.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-08-12 18:23:05 UTC; 20h ago
       Docs: man:monit(1)
             https://mmonit.com/wiki/Monit/HowTo
   Main PID: 1804 (monit)
      Tasks: 2 (limit: 4265)
     CGroup: /system.slice/monit.service
             └─1804 /usr/local/bin/monit -I

Configure Monit

The following step is to configure Monit to monitor specific things. On Ubuntu and Debian-based systems, the Monit configuration file is located at /etc/monit/monitrc. We recommend that you make a copy of this file before applying any change – $ sudo cp monitrc monitrc.bck

A. Run as a daemon:

The first directive declared in the file is:

##
#
set daemon 120
with start delay 120
#
##

It is used to indicate the refresh time for checking the services, that is to say in this case that the check will take place every 120 seconds, or 2 minutes. This command also indicates that Monit is running in daemon mode. with start delay 120 delays the first check by 2-minutes (by default Monit check immediately at start up)

B. Logs:

The second directive concerns Monit' logs, which by default are stored in the file "monit.log" in "/var/log/". By changing the value of this parameter, you change the log file.

##
#
set log /var/log/monit.log
#
##

C. Identification and status files:

The third directive specifies the path to the file containing Monit's unique identifier. Do not change this setting.

##
#
# set idfile /var/.monit.id
#
##

The fourth directive specifies the path to Monit's state file, which is a temporary file where Monit stores information. Do not change this setting.

##
#
# set statefile /var/.monit.state
#
##

D. SMTP server (s) for alerts:

Since Monit offers email alerts, it must be configured by adding this to the monitrc file; here we are using Gmail as a Relay.

##
#
set mailserver smtp.gmail.com port 587
  username "auth_user" password "auth_password"
  using tls
#  
##

Note: Replace "auth_user" and "auth_password" with your Gmail user name and password. To specify multiple SMTP servers, separate them with a comma.
Remark: On version 5.28.1 adding V1 to TLS will not work; avoid adding any TLS version.

E. Customize emails format:

Sending alert emails is good, personalizing them is even better!
Monit allows you to do this using the "mail-format" directive:

##
#
set mail-format {
 from:    Monit <monit@$HOST>
 subject: Monit alert ⚠️  --  $EVENT $SERVICE
 message: $EVENT Service $SERVICE
                 Date:        $DATE
                 Action:      $ACTION
                 Host:        $HOST
                 Description: $DESCRIPTION
 }
#  
##

You notice the presence of different variables, like "$SERVICE", "$EVENT", "$DATE", "$ACTION" and "$HOST". They make it possible to ensure that the alert message adapts to the detected error.

• $SERVICE: Name of the service indicated in monitrc.
• $EVENT: Type of event.
• $DATE: The current date.
• $ACTION: The name of the action performed, which corresponds to the action defined in monitrc for this service.
• $HOST: The name of the host where monit is installed.

"From" corresponds to the sending address, "reply-to" corresponds to the address to reply, "subject" to the subject of the message, and "message" to the message contained in the email.

F. Email recipient:

Do not forget to specify to whom the alerts should be sent. We use the "alert" directive as such:

##
#
  set alert admin@Xxx.com
#  
##

G. Alert storage:

If you do not want to receive any email alert, you can tell Monit to store the alerts in a directory so that you can access them later. The "eventqueue" directive should be used for this and it is already enabled by default.

##
#
#set eventqueue
#basedir /var/lib/monit/events  # set the base directory where events #will be stored
#slots 100           # optionally limit the queue size
#  
##

H. Web interface Activation :

Monit offers a web interface for viewing the data collected. However, it must be enabled and configured in the monitrc file since this is not done by default.

##
#
# set the port Monit should bind to and listen on. 

set httpd port 2812 and  # Monit is usually setup on port 2812.

# use address localhost  # only accept connection from localhost (drop if you use M/Monit)
  allow 192.168.1.201  	 # allow localhost to connect to the server and
  allow <auth_user>:<auth_password>  # require user 'admin' with password 'monit'

# allow @monit           # allow users of group 'monit' to connect (rw)
# allow @users readonly  # allow users of group 'users' to connect readonly

   with ssl {            # enable SSL/TLS and set path to server certificate
        pemfile: /etc/ssl/certs/monit.pem
        selfsigned: allow
}
#
##
# 

You can now use http://localhost:2812/ to access Monit's web interface after providing your username and password as credentials. You might need to use double quotes around the password if it contains special chars such as "p@ssw:r#".

PORT set a port so Monit should be bound to and listened on. Monit is usually setup on port 2812 but you can pretty much use any available port.

set httpd port 2812 and
    allow <auth_user>:<auth_password>  # require user 'admin' with password 'monit'

ADDRESS make Monit listen on a specific interface only. For example, if you don't want to expose Monit's web interface to the network, bind it to localhost only. Monit will accept connections on any addresses if the ADDRESS option is not used.

set httpd port 2812 and
    use address 127.0.0.1
    allow <auth_user>:<auth_password>  # require user 'admin' with password 'monit'

Host and network allow list allow Monit to keep track of which hosts and networks are authorized to connect. You can add as many hosts as you want to, but only hosts with a valid domain name or IP address are allowed. Monit will query a name server to check any hosts trying to connect. If a host (client) is trying to connect, but cannot be found in the access list or cannot be resolved, Monit will shutdown the connection to the client promptly.

Control file example:

set httpd port 2812 and
    use address localhost
    allow localhost
    allow my.quantium.server.com
    allow 10.1.1.1
    allow 192.168.1.0/255.255.255.0
    allow 10.0.0.0/8

Clients, not mentioned in the allow list and trying to connect to Monit will be denied access and their IP address will be logged.

Notes: To access Monit from an outside network, just comment out or remove any allow option, such as below:

set httpd port 2812 and
    #allow localhost
    #allow my.quantium.server.com
    #allow 10.1.1.1
    #allow 192.168.1.0/255.255.255.0
    #allow 10.0.0.0/8
    allow <auth_user>:<auth_password>

a. Create New Certificate

Selfsigned self-signed certificates are rejected by default. Use this option to allow self-signed certificates.
Warning: Using a self-signed certificate in production is not recommended, as in such case the client cannot verify that it talks to the correct server (vulnerable to man-in-the-middle attacks and DNS-hijacking).
Run these commands to generate the pemfile :

# Generates the private key and the certificate
  openssl req -new -x509 -days 365 -nodes \
    -config ./monit.cnf -out /etc/ssl/certs/monit.pem \
    -keyout /etc/ssl/certs/monit.pem

# Generates the  Diffie-Hellman Parameters
  openssl dhparam -2 2048 >> /etc/ssl/certs/monit.pem

# Set mode
  chmod 600 /etc/ssl/certs/monit.pem

# Prints out the certificate information
   openssl x509 -text -noout -in /etc/ssl/certs/monit.pem

Before Monit is started the first time, we need to test the control file for syntax errors:

$ sudo monit -t
[sudo] password for snubmonkey: 

OUPUT

Control file syntax OK

Monit will output an error message to the console if an error occurs, along with the line number in the control file where the issue has happened.

Finally, restart your Monit service to take the changes into account.

$ sudo systemctl restart monit
or
$ sudo monit restart all

After you logged into Monit, you should see the Monit default dashboard as shown below.

Click on any item to view its system status on your server.
Here, we monitoring the network link.

G. Monitor services:

Monit is now set up and ready to send you alerts whenever an issue occurs, but you must first tell it what to look for. You'll see it's really straightforward and intuitive. Of course, we will not be able to observe all types of checks in this tutorial because they are unlimited and dependent on your needs.

a. The different types of action:

You need to tell Monit what to do when it detects an error, using the following actions:

• alert: Send an alert message,
• start: Start the service and send an alert message,
• stop: Stops the service and sends an alert message,
• restart: Restarts the service and sends an alert message,
• exec: Execute a script and send an alert message,
• unmonitor: Disables monitoring of the service and sends an alert message,
• monitor: Activate monitoring.

H. Configuration Examples

After changing configurations, make sure to 'sudo monit -t'
and run 'sudo monit reload' to check for errord and bring them in effect.
While testing you can manually turn services on/off 
and check the monit report from the web interface.

1. System resources

Check general system resources such as load average, cpu and memory usage.
The goal is to send an alert by email if one of the following conditions is true:

 check system $HOST
   if loadavg (5min) > 3 then alert
   if loadavg (15min) > 1 then alert
   if memory usage > 80% for 4 cycles then alert
   if swap usage > 20% for 4 cycles then alert
# Test the user part of CPU usage 
   if cpu usage (user) > 40% for 2 cycles then alert
# Test the system part of CPU usage 
   if cpu usage (system) > 31% for 2 cycles then alert
# Test the i/o wait part of CPU usage 
   if cpu usage (wait) > 80% for 2 cycles then alert
# Test CPU usage including user, system and wait. Note that 
# multi-core systems can generate 100% per core
# so total CPU usage can be more than 100%
   if cpu usage > 200% for 4 cycles then alert

2. Apache webserver - Process Monitoring

Consider the following scenario: our server runs an Apache2 web server, and we want to avoid any downtime, thus we'll need to monitor the service and restart it if it fails.
To do so, we need to check the "apache2" process by specifying its PID file after "with pidfile" so that Monit knows which process to monitor.
To find out if the Apache service is still available on port 80 for our web server, we'll use the HTTP protocol to test the webserver again on port 80.
In the event of a problem, apache2 will be restarted.

# apache2 is the name of the service
 check process apache2 with pidfile /var/run/apache2.pid
# Start/Stop action
   start program = "/bin/systemctl start"
   stop program = "/bin/systemctl stop"
# Test server on port 80
   if failed host snunbmonkey.com port 80 protocol http then restart

3. Monitor SSHD service

Here is how to check that the sshd process corresponding to the SSH service is active:

# apache2 is the name of the service
 check process apache2 with pidfile /var/run/apache2.pid
# Start/Stop action
   start program = "/etc/init.d/ssh start"
   stop program = "/etc/init.d/ssh stop"
# Test server on port 22
   if failed port 22 protocol ssh then restart

4. Monitor checksum SHA1

Another type of monitoring that can be performed is checking the MD5 or SHA1 sum of a file, which is a unique fingerprint of the file in its current state. As soon as there is the slightest modification within the monitored file, the sum will necessarily change and therefore an alert will be triggered by Monit.

With this, you can ensure that no changes are made to sensitive system files, especially configuration files. We will take the example of the "sshd_config" file which is the configuration file for the SSH service.

There are two ways to accomplish this using Monit. The first is to calculate the SHA1 sum first and then tell Monit to check that it is equal to that all of the time. The second is to check if the total has changed and if so, you'll get notified.
We will see both approaches.

 ### 1
 # sshd_config is the name of the service
 check file apache_bin with path /etc/ssh/sshd_config
   if failed checksum and
   expect the sum 8f7f419955cefa0b33a2ba316cba3659 then alert
### 2
# sshd_config is the name of the service
  check file sshd_config path /etc/ssh/sshd_config
#Send an aletr if the SHA1 sum has changed
   if changed sha1 checksum then alert

5. Monitor permissions

It is also possible to control the permissions on a file, which can be interesting to do on configuration files, in order to be sure that nobody makes changes thanks to the calculation of the sum and that nobody modifies the permissions thanks to the control over permissions.

 # sshd_config is the name of the service
check file sshd_config path /etc/ssh/sshd_config
# Send an alert is the SHA1 sum has changed
if changed sha1 checksum then alert
# Send an aletr if the file' permissions if different fromn 644
if failed permission 644 then alert

6. Network ping test

Monit can perform a network ping test by sending ICMP echo request datagram packets to a host and wait for the reply. Here we will watch a host, that is supposed to be offline.

check host VirtualServer#445 with address 192.168.1.111
   if succeeded ping then alert

7. Alert repeatedly

By default, Monit generates an alert only twice. Once when a test fails and other when it succeeds.
If you wish to get repeated alerts for the same failing test, then use the following configuration.

set alert email@example.com with reminder on 10 cycles

The above configuration is a global setting, that sends all alerts after every 10 checks/cycles.
To repeat alerts only for specific tests, specify the alert directive within the particular test criteria like the following:

##
#
...
alert email@example.com with reminder on 1 cycles
  if space usage > 80% for 5 times within 5 cycles then alert

If you would like to be reminded fewer times but repeatedly then use something like "reminder on 10 cycles".

Check Monit status

You can also check the status of all the services using the following command:

$ sudo monit status

[sudo] password for snubmonkey: 
Monit 5.28.2 uptime: 2h 46m

OUTPUT

System 'WebServer'
  status                       Resource limit matched
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  load average                 [0.49] [0.77] [1.60]
  cpu                          9.0%usr 3.5%sys 0.0%nice 1.0%iowait 0.0%hardirq 0.1%softirq 0.0%steal 0.0%guest 0.0%guestnice 
  memory usage                 747.3 MB [40.0%]
  swap usage                   0 B [0.0%]
  uptime                       3d 0h 4m
  boot time                    Thu, 12 Aug 2021 18:49:11
  filedescriptors              2400 [0.0% of 2147483647 limit]
  data collected               Sun, 15 Aug 2021 18:53:00


Process 'sshd'
  status                       OK
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  pid                          1881
  parent pid                   1
  uid                          0
  effective uid                0
  gid                          0
  uptime                       3d 0h 3m
  threads                      1
  children                     6
  cpu                          0.0%
  cpu total                    0.0%
  memory                       0.2% [3.5 MB]
  memory total                 1.3% [24.9 MB]
  security attribute           unconfined
  filedescriptors              5 [0.5% of 1024 limit]
  total filedescriptors        65
  read bytes                   290.2 B/s [262.9 MB total]
  disk read bytes              0 B/s [51.1 MB total]
  disk read operations         0.8 reads/s [487283 reads total]
  write bytes                  3.7 B/s [12.0 MB total]
  disk write bytes             0 B/s [2.4 MB total]
  disk write operations        0.0 writes/s [33180 writes total]
  port response time           34.991 ms to localhost:22 type TCP/IP protocol SSH
  data collected               Sun, 15 Aug 2021 18:53:00
  
  
  Process 'OpenVPN'
  status                       OK
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  pid                          1814
  parent pid                   1
  uid                          65534
  effective uid                65534
  gid                          65534
  uptime                       3d 0h 3m
  threads                      1
  children                     0
  cpu                          2.0%
  cpu total                    2.0%
  memory                       0.2% [4.4 MB]
  memory total                 0.2% [4.4 MB]
  security attribute           unconfined
  filedescriptors              7 [0.7% of 1024 limit]
  total filedescriptors        7
  read bytes                   148.2 kB/s [14.0 GB total]
  disk read bytes              0 B/s [2.6 MB total]
  disk read operations         119.2 reads/s [13515753 reads total]
  write bytes                  4.4 kB/s [1.5 GB total]
  disk write bytes             0 B/s [0 B total]
  disk write operations        33.4 writes/s [8974587 writes total]
  data collected               Sun, 15 Aug 2021 18:53:00

Yeeeeahhh!!!
You have successfully installed and configured Monit monitoring tool.
Monit is capable of doing much complex things like pinging network hosts, running external scripts, and test the output for a certain value. We cannot urge you strongly enough to check the full documentation at -
https://mmonit.com/monit/documentation/monit.html