Getting Started with Prometheus and Grafana

# What is Prometheus ?

Prometheus is a tool that watches over your computer systems, collects data about how they're doing, and alerts you if anything goes wrong.

STEP 1 :- Create an AWS EC2 instance of t2.micro or t3.micro with Ubuntu 20.04 LTS .

STEP2:- Connect to the instance ( I use Git Bash and I connect using the command --> ssh -i mypemfile.pem ubuntu@ip_address_of_instance )

STEP3:- The below code will install 2 services , namely Prometheus and the Prometheus Node Exporter.

sudo apt update
sudo apt install prometheus

STEP4:- We can verify the status of Prometheus and the Prometheus Node Exporter using the below commands.

sudo service prometheus status
sudo service prometheus-node-exporter status

We will see --> Active: active (running)

STEP5:- Edit the inbound rules of the security groups of the EC2 instance to include port 9090.

STEP6:- We can visit Prometheus at --> http://[IP address of the EC2 instance]:9090

Similarly if we include port 9100 in the inbound rules of security groups, then we can visit Prometheus Node Exporter at -->*http://[IP address of the EC2 instance]:9100*

We can see which processes are running by using the command :-

ps -u prometheus

It will give us details like PID , TTY , TIME , CMD.

#Reverse proxying Prometheus with Nginx

It helps to improve security, manage traffic, and provide additional features like adding SSL , basic Authentication ,etc.

sudo apt install nginx
sudo service nginx status

On visiting the ipaddress_of_ec2instance :- the default Nginx welcome page opens.

#Providing Domain Name for our Prometheus Server

STEP1 :- Visit the link https://freedns.afraid.org/signup/ . After signing up, create your own Domain there for FREE.

STEP2 :- Then follow the following set of commands :-

cd /etc/nginx/sites-enabled
sudo nano prometheus

Then copy-paste the below text :-

server {

listen 80;

listen [::]:80;

server_name YOUR-DOMAIN-NAME;

location / {

proxy_passlocalhost:9090;

}

}

sudo nginx -t
sudo service nginx restart

So we can visit Prometheus Server at --> YOUR-DOMAIN-NAME

#Adding SSL to Prometheus Reverse Proxy

After adding SSL, accessing the server via HTTPS ensures secure communication by encrypting data transmitted between the server and clients, enhancing privacy and security.

sudo snap install core; sudo snap refresh core
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
sudo certbot --nginx

Now we can also visit the Server at --> YOUR-DOMAIN-NAME

#Add Basic Authentication

cd /etc/nginx
sudo apt install apache2-utils
htpasswd -c /etc/nginx/.htpasswd admin #creating user called "admin"

Then we need to enter a password.

Then we need to open the Nginx Prometheus config file using the below command :- sudo nano /etc/nginx/sites-enabled/prometheus

server { ...

auth_basic "Protected Area";

auth_basic_user_file /etc/nginx/.htpasswd;

...

}

sudo nginx -t
sudo service nginx restart

So now we need to enter Username and Password :-

#Monitoring Jenkins with Prometheus and Grafana :-

STEP1:- Install JENKINS by following this video https://youtu.be/lRpS2CovMrs?si=2tju9q0HrD43HaIn

STEP2:- Install the Prometheus metrics plugin in Jenkins

STEP3:- Use the following command :-

sudo nano /etc/prometheus/prometheus.yml

Then copy-paste the following portion in the .yml file

- job_name: 'jenkins'
    metrics_path: '/prometheus'
    static_configs:
      - targets: ['url_of_the_jenkins_server:8080']

STEP4:- Then I restarted the prometheus server

sudo service prometheus stop
sudo systemctl start prometheus

Then in the Prometheus server , I monitored the number of plugins active in my Jenkins server :-

In my case , I see that 92 plugins are active right now .

Step5 :- INSTALL GRAFANA

Grafana is a data visualization tool that works seamlessly with Prometheus, allowing users to create dashboards and visualize the metrics collected by Prometheus in a user-friendly and customizable way.

sudo apt update
sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_9.3.2_amd64.deb
sudo dpkg -i grafana_9.3.2_amd64.deb
sudo service grafana-server start
sudo service grafana-server status #Check the status

Grafana server will be hosted at :-http://[Grafana server IP address]:3000

Default Grafana login is -> Username : admin Password : admin

After Logging in , it is recommended to change the Password.

Then follow the below steps :-

Add Data Sources --> Prometheus --> Paste the URL of prometheus server in the HTTP(URL) Section. --> Save & Test

Then create a new dashboard , then add a new panel . In visualizations I opted forStat. Search for jenkins_plugins_activein the Query section.

So we see the result as 92.

Then I saved the Dashboard with the title --> How many active plugins in Jenkins

Now , suppose I want to monitor the results of my last builds in jenkins , then we see:-

a) Result from my Jenkins Dashboard

b) Result from Prometheus

c) Result from Grafana

So all these results confirm the fact that my last build of the jenkins jobs "first_project" and "kkc_builders" are successful but "project_test_hook" has failed.