Monitoring Docker Containers using Docker stats, Prometheus and cAdvisor

Ticker

6/recent/ticker-posts

Monitoring Docker Containers using Docker stats, Prometheus and cAdvisor

will walk through how to monitor containers using Docker stats, Prometheus and cAdvisor.


1. Log into the Docker server and then become the root user with a sudo su -

    I already have two containers  running in  live environment: nginx and redis.

 [root@ip-10-0-1-212 ~]# docker ps

CONTAINER ID   IMAGE     COMMAND                  CREATED       STATUS       PORTS                                      NAMES

e80fef4dcb1d   nginx     "/docker-entrypoint.…"   2 hours ago   Up 2 hours   0.0.0.0:80->80/tcp, :::80->80/tcp           nginx

753447b96a14   redis     "docker-entrypoint.s…"   2 hours ago   Up 2 hours   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp   redis

[root@ip-10-0-1-212 ~]#

2. In root's home directory, create prometheus.yml:

[root@ip-10-0-1-212 ~]# vi prometheus.yml

[root@ip-10-0-1-212 ~]# cat prometheus.yml

scrape_configs:

- job_name: cadvisor

  scrape_interval: 5s

  static_configs:

  - targets:

    - cadvisor:8080

 [root@ip-10-0-1-212 ~]#

3. To set up Prometheus services, we've got to create configuration files. 

We're going to make docker-compose.yml, right here in the same directory

[root@ip-10-0-1-212 ~]# vi docker-compose.yml

[root@ip-10-0-1-212 ~]# cat docker-compose.yml

version: '3'

services:

  prometheus:

    image: prom/prometheus:latest

    container_name: prometheus

    ports:

      - 9090:9090

    command:

      - --config.file=/etc/prometheus/prometheus.yml

    volumes:

      - ./prometheus.yml:/etc/prometheus/prometheus.yml

    depends_on:

      - cadvisor

 

  cadvisor:

    image: google/cadvisor:latest

    container_name: cadvisor

    ports:

      - 8080:8080

    volumes:

      - /:/rootfs:ro

      - /var/run:/var/run:rw

      - /sys:/sys:ro

      - /var/lib/docker:/var/lib/docker:ro

 [root@ip-10-0-1-212 ~]#

4. To create the cadvisor & Prometheus container run 

[root@ip-10-0-1-212 ~]# docker-compose up -d

Creating network "root_default" with the default driver

Pulling cadvisor (google/cadvisor:latest)...

latest: Pulling from google/cadvisor

ff3a5c916c92: Pull complete

44a45bb65cdf: Pull complete

0bbe1a2fe2a6: Pull complete

Digest: sha256:815386ebbe9a3490f38785ab11bda34ec8dacf4634af77b8912832d4f85dca04

Status: Downloaded newer image for google/cadvisor:latest

Pulling prometheus (prom/prometheus:latest)...

latest: Pulling from prom/prometheus

aa2a8d90b84c: Pull complete

b45d31ee2d7f: Pull complete

7579d86a00c9: Pull complete

8583d0bc7e17: Pull completeb32caf1c5e65: Pull complete

e53f205885a2: Pull complete6366df248f46: Pull complete

a63db3af7b6e: Pull complete

94cd9f02fa61: Pull complete

2511fa13a76c: Pull complete

50c2584d9f31: Pull complete

22749d939f03: Pull complete

Digest: sha256:e9620d250b16ffe0eb9e3eac7dd76151848424c17d577632ae9ca61d1328687e

Status: Downloaded newer image for prom/prometheus:latest

Creating cadvisor ... done

Creating prometheus ... done

[root@ip-10-0-1-212 ~]#

5. Verify by Docker ps additional new containers should be running status.

[root@ip-10-0-1-212 ~]# docker ps

CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS         PORTS            NAMES

d9b2f7d472da   prom/prometheus:latest   "/bin/prometheus --c…"   8 seconds ago    Up 6 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus

3d11890e1cd1   google/cadvisor:latest   "/usr/bin/cadvisor -…"   12 seconds ago   Up 7 seconds   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   cadvisor

e80fef4dcb1d   nginx                    "/docker-entrypoint.…"   2 hours ago      Up 2 hours     0.0.0.0:80->80/tcp,     :::80->80/tcp       nginx

753447b96a14   redis                                                       "docker-entrypoint.s…"   2 hours ago      Up 2 hours     0.0.0.0:6379->6379/tcp, :::6379->6379/tcp   redis

[root@ip-10-0-1-212 ~]#


6. Viewing the Prometheus Web Interface using the correct port number:                       http://<IP_ADDRESS>:9090/graph/

7. Viewing the cAdvisor Web Interface using the correct port number:                           http://<IP_ADDRESS>:8080/docker/nginx/

       ( Refer Below Video for web interface )

8. Stats in Docker various aspects (CPU and memory usage, etc.) of  containers.
 
     Press crtl + c to exit from stats 

[root@ip-10-0-1-212 ~]# vi  ~/stats.sh

[root@ip-10-0-1-212 ~]# cat  ~/stats.sh

docker stats --format "table {{.Name}} {{.ID}} {{.MemUsage}} {{.CPUPerc}}"

[root@ip-10-0-1-212 ~]# ./stats.sh

NAME CONTAINER ID MEM USAGE / LIMIT CPU %

prometheus d9b2f7d472da 37.7MiB / 962.8MiB 0.00%

cadvisor 3d11890e1cd1 54.3MiB / 962.8MiB 1.56%

nginx e80fef4dcb1d 2.215MiB / 962.8MiB 0.00%

redis 753447b96a14 7.945MiB / 962.8MiB 0.12%

^C[root@ip-10-0-1-212 ~]#


9. We got Prometheus running, and we're able to monitor our containers with it. Congratulations!

Happy Learning!!!!


Apart from this if you need more clarity,  I have made a  tutorial video on this

please go through and share your comments. Like and share the knowledge




Post a Comment

0 Comments

Latest Posts

KodeKloud Kubernetes Security CKS  Lab Challenge 4 |  Audit-policy | Install & configure falco utility | Inspect the API server audit logs and identify the user