Linux Process Troubleshooting KodeKloud Engineer Task Success

 

Question:   The production support team of xFusionCorp Industries has deployed some of the latest monitoring tools to keep an eye on every service, application, etc. running on the systems. One of the monitoring systems reported about Apache service unavailability on one of the app servers in Stratos DC.

Identify the faulty app host and fix the issue. Make sure Apache service is up and running on all app hosts. They might not hosted any code yet on these servers so you need not to worry about if Apache isn't serving any pages or not, just make sure service is up and running. Also, make sure Apache is running on port 8087 on all app servers.

Please Note :-  Perform the below commands based on your question server,  user name & other details that might differ. So please read the task carefully before executing it. 

All the Best 👍

Solution:  

1. At first identify the app server which has issue by running the below commands

thor@jump_host ~$ telnet stapp01 8087 Trying 172.16.238.10... telnet: connect to address 172.16.238.10: Connection refused thor@jump_host ~$ telnet stapp02 8087 Trying 172.16.238.11... Connected to stapp02. Escape character is '^]'. ^] telnet> quit Connection closed. thor@jump_host ~$ telnet stapp03 8087 Trying 172.16.238.12... Connected to stapp03. Escape character is '^]'. ^] telnet> quit Connection closed. thor@jump_host ~$

2. login on app server which has connection refuse error & switch to root  

thor@jump_host ~$ ssh tony@stapp01 The authenticity of host 'stapp01 (172.16.238.10)' can't be established. ECDSA key fingerprint is SHA256:SySamszyWhhLGFiybhGBqfrr8g55wS/3e37ZpBOvICs. ECDSA key fingerprint is MD5:6d:31:18:2a:f9:07:f3:29:dd:0a:d3:1f:6e:04:0a:db. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'stapp01,172.16.238.10' (ECDSA) to the list of known hosts. tony@stapp01's password: [tony@stapp01 ~]$ sudo su - We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things:     #1) Respect the privacy of others.     #2) Think before you type.     #3) With great power comes great responsibility. [sudo] password for tony: [root@stapp01 ~]#

3. Run the Below command to check the existing Apache HTTPd service     

[root@stapp01 ~]# systemctl start httpd Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details. [root@stapp01 ~]# systemctl status httpd ● httpd.service - The Apache HTTP Server    Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)    Active: failed (Result: exit-code) since Fri 2021-08-13 15:02:27 UTC; 7s ago      Docs: man:httpd(8)            man:apachectl(8)   Process: 693 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)   Process: 692 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)  Main PID: 692 (code=exited, status=1/FAILURE)   Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com httpd[692]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using stap... message Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com httpd[692]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:8087 Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com httpd[692]: no listening sockets available, shutting down Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com httpd[692]: AH00015: Unable to open logs Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com kill[693]: kill: cannot find process "" Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com systemd[1]: httpd.service: control process exited, code=exited status=1 Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com systemd[1]: Failed to start The Apache HTTP Server. Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com systemd[1]: Unit httpd.service entered failed state. Aug 13 15:02:27 stapp01.stratos.xfusioncorp.com systemd[1]: httpd.service failed. Hint: Some lines were ellipsized, use -l to show in full. [root@stapp01 ~]#

4. With the help of  netstat check which service is using your port and whats is PID. In below snipping ( refer to  below video for more details)

[root@stapp01 ~]# netstat -tulnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    tcp        0      0 127.0.0.11:45287        0.0.0.0:*               LISTEN      -                   tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/init              tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      529/sshd            tcp        0      0 127.0.0.1:8087          0.0.0.0:*               LISTEN      615/sendmail: accep tcp6       0      0 :::111                  :::*                    LISTEN      528/rpcbind         tcp6       0      0 :::22                   :::*                    LISTEN      529/sshd            udp        0      0 127.0.0.11:40100        0.0.0.0:*                           -                   udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/init              udp        0      0 0.0.0.0:687             0.0.0.0:*                           528/rpcbind         udp6       0      0 :::111                  :::*                                528/rpcbind         udp6       0      0 :::687                  :::*                                528/rpcbind         [root@stapp01 ~]#

5. Confirm the PID as per  the netstat output 

[root@stapp01 ~]# ps -ef  |grep sendmail root       615     1  0 14:52 ?        00:00:00 sendmail: accepting connections root       721   674  0 15:08 pts/0    00:00:00 grep --color=auto sendmail [root@stapp01 ~]# [root@stapp01 ~]# kill 615 [root@stapp01 ~]# ps -ef  |grep sendmail root       723   674  0 15:08 pts/0    00:00:00 grep --color=auto sendmail [root@stapp01 ~]#

6.  Start the httpd service & check the status.

       

[root@stapp01 ~]# systemctl start httpd [root@stapp01 ~]# systemctl status  httpd ● httpd.service - The Apache HTTP Server    Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)    Active: active (running) since Fri 2021-08-13 15:10:15 UTC; 7s ago      Docs: man:httpd(8)            man:apachectl(8)   Process: 693 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)  Main PID: 727 (httpd)    Status: "Processing requests..."    CGroup: /docker/0e93815bb34ff1c4e85f922182c9b3d480dab672ea1330f85a3c6e9c286a52ca/system.slice/httpd.service            ├─727 /usr/sbin/httpd -DFOREGROUND            ├─729 /usr/sbin/httpd -DFOREGROUND            ├─730 /usr/sbin/httpd -DFOREGROUND            ├─731 /usr/sbin/httpd -DFOREGROUND            ├─732 /usr/sbin/httpd -DFOREGROUND            └─733 /usr/sbin/httpd -DFOREGROUND  Aug 13 15:10:15 stapp01.stratos.xfusioncorp.com systemd[1]: Starting The Apache HTTP Server... Aug 13 15:10:15 stapp01.stratos.xfusioncorp.com httpd[727]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using stap... message Aug 13 15:10:15 stapp01.stratos.xfusioncorp.com systemd[1]: Started The Apache HTTP Server. Hint: Some lines were ellipsized, use -l to show in full. [root@stapp01 ~]#

7.  Validate Apache HTTPd running  as per the task request

thor@jump_host ~$ telnet stapp01 8087 Trying 172.16.238.10... Connected to stapp01. Escape character is '^]'.

8.  Click on Finish & Confirm to complete the task successfully

Happy Learning!!!!

Apart from this if you need more clarity,  I have made a  tutorial video on this , please go through and share your comments. Like and share the knowledge