zabbix故障预处理和报警升级

实验效果：实现httpd停止服务，zabbix执行远程命令进行恢复，1分钟之后如果恢复失败，发邮件给运行，1分钟之后，问题没解决发邮件给领导

1. 环境规划

2.zabbix-server安装

略

3：zabbix-agent安装

略

4：zabbix-agent上httpd安装

xxxxxxxxxx
1
1
yum install httpd -y

5：自定义监控项

xxxxxxxxxx
4
1
vim /etc/zabbix/zabbix_agentd.conf
2
UserParameter=httpd,ps -ef|grep -v grep|grep -c '/usr/sbin/httpd'
3
​
4
systemctl  restart zabbix-agent

6：创建自定义触发器

7: 创建邮件报警

a：创建发件人 b：创建两个收件人

boss用户需要超级管理权限

8：在zabbix-server上使用ansible远程重启httpd

xxxxxxxxxx
7
1
yum install ansible -y
2
​
3
vim /etc/ansible/hosts
4
[web]
5
192.168.100.12  ansible_ssh_user='root' ansible_ssh_pass='123456'
6
​
7
ansible web -m shell -a 'systemctl restart httpd.service'

客户端效果

xxxxxxxxxx
32
1
[root@oldboy ~]# systemctl status httpd
2
● httpd.service - The Apache HTTP Server
3
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
4
   Active: activating (start) since Tue 2019-08-06 22:20:08 CST; 4s ago
5
     Docs: man:httpd(8)
6
           man:apachectl(8)
7
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
8
 Main PID: 2894 (httpd)
9
   CGroup: /system.slice/httpd.service
10
           └─2894 /usr/sbin/httpd -DFOREGROUND
11
​
12
Aug 06 22:20:08 oldboy systemd[1]: Starting The Apache HTTP Server...
13
Aug 06 22:20:08 oldboy httpd[2894]: AH00558: httpd: Could not reliably determine the server's fully qua...ssage
14
Hint: Some lines were ellipsized, use -l to show in full.
15
​
16
[root@oldboy ~]# systemctl status httpd
17
● httpd.service - The Apache HTTP Server
18
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
19
   Active: active (running) since Tue 2019-08-06 22:20:13 CST; 295ms ago
20
     Docs: man:httpd(8)
21
           man:apachectl(8)
22
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
23
 Main PID: 2894 (httpd)
24
   Status: "Processing requests..."
25
   CGroup: /system.slice/httpd.service
26
           ├─2894 /usr/sbin/httpd -DFOREGROUND
27
           ├─2898 /usr/sbin/httpd -DFOREGROUND
28
           ├─2899 /usr/sbin/httpd -DFOREGROUND
29
           ├─2900 /usr/sbin/httpd -DFOREGROUND
30
           ├─2901 /usr/sbin/httpd -DFOREGROUND
31
           └─2902 /usr/sbin/httpd -DFOREGROUND
32
​

9：zabbix-server上的zabbix-agent启用远程命令功能：

xxxxxxxxxx
5
1
vim /etc/zabbix/zabbix_agentd.conf 
2
EnableRemoteCommands=1
3
AllowRoot=1
4
​
5
systemctl restart zabbix-agent.service

10:在zabbix-web上配置触发器动作

整体效果如下：

远程命令细节：

xxxxxxxxxx
1
1
命令：ansible web -m shell -a 'systemctl restart httpd.service'

11: 测试zabbix-agent报警升级

测试1：

停止httpd服务

xxxxxxxxxx
10
1
[root@oldboy ~]# systemctl status httpd
2
● httpd.service - The Apache HTTP Server
3
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
4
   Active: activating (start) since Tue 2019-08-06 22:20:08 CST; 4s ago
5
     Docs: man:httpd(8)
6
           man:apachectl(8)
7
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
8
 Main PID: 2894 (httpd)
9
   CGroup: /system.slice/httpd.service
10
           └─2894 /usr/sbin/httpd -DFOREGROUND

查看httpd服务状态

xxxxxxxxxx
16
1
[root@oldboy ~]# systemctl status httpd
2
● httpd.service - The Apache HTTP Server
3
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
4
   Active: active (running) since Tue 2019-08-06 22:20:13 CST; 295ms ago
5
     Docs: man:httpd(8)
6
           man:apachectl(8)
7
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
8
 Main PID: 2894 (httpd)
9
   Status: "Processing requests..."
10
   CGroup: /system.slice/httpd.service
11
           ├─2894 /usr/sbin/httpd -DFOREGROUND
12
           ├─2898 /usr/sbin/httpd -DFOREGROUND
13
           ├─2899 /usr/sbin/httpd -DFOREGROUND
14
           ├─2900 /usr/sbin/httpd -DFOREGROUND
15
           ├─2901 /usr/sbin/httpd -DFOREGROUND
16
           └─2902 /usr/sbin/httpd -DFOREGROUND

测试2：

停止httpd服务

xxxxxxxxxx
14
1
[root@oldboy ~]# systemctl status httpd
2
● httpd.service - The Apache HTTP Server
3
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
4
   Active: activating (start) since Tue 2019-08-06 22:20:08 CST; 4s ago
5
     Docs: man:httpd(8)
6
           man:apachectl(8)
7
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
8
 Main PID: 2894 (httpd)
9
   CGroup: /system.slice/httpd.service
10
           └─2894 /usr/sbin/httpd -DFOREGROUND
11
​
12
#使用nc监听80端口，模拟httpd重启失败
13
yum install nc  -y
14
nc -l 80

至此,已经实现httpd停止服务，zabbix执行远程命令进行恢复，1分钟之后如果恢复失败，发邮件给运行，1分钟之后，问题没解决发邮件给领导