保姆级教程：从零搭建Prometheus+Grafana监控全家桶（含Linux/MySQL/Redis/Windows）

从零构建企业级监控系统PrometheusGrafana全栈实战指南当服务器数量突破两位数时凌晨三点被报警电话吵醒的运维人员总会思考同一个问题如何提前发现潜在问题监控系统就像IT基础设施的神经系统而今天我们要部署的这套组合——Prometheus负责数据采集Grafana实现可视化呈现——正是当前最流行的开源监控解决方案。不同于零散的教程本文将带您从单一控制点出发构建覆盖Linux主机、MySQL、Redis以及Windows系统的完整监控体系。1. 环境规划与核心组件部署在开始安装前我们需要明确架构设计的基本原则。典型的监控系统采用中心辐射模型Prometheus作为中央服务器各被监控节点运行对应的exporter作为数据采集代理。假设我们有以下环境角色IP地址需安装组件开放端口监控服务器192.168.1.10PrometheusGrafana9090,3000Linux应用服务器192.168.1.20node_exporter9100MySQL数据库服务器192.168.1.30mysqld_exporter9104Windows文件服务器192.168.1.40windows_exporter9182Prometheus安装只需三步# 下载最新稳定版以2.37.0为例 wget https://github.com/prometheus/prometheus/releases/download/v2.37.0/prometheus-2.37.0.linux-amd64.tar.gz # 解压到系统目录 tar xf prometheus-*.tar.gz -C /usr/local mv /usr/local/prometheus-* /usr/local/prometheus # 验证版本 /usr/local/prometheus/prometheus --version提示生产环境建议配置systemd服务管理示例配置[Unit] DescriptionPrometheus Monitoring System Afternetwork.target [Service] Userprometheus ExecStart/usr/local/prometheus/prometheus \ --config.file/usr/local/prometheus/prometheus.yml \ --storage.tsdb.path/var/lib/prometheus/data \ --web.listen-address:9090 Restarton-failure [Install] WantedBymulti-user.target2. 多平台数据采集实战2.1 Linux系统监控深度配置node_exporter的默认配置可能无法满足精细化监控需求我们需要启用特定收集器/usr/local/node_exporter/node_exporter \ --collector.systemd \ --collector.processes \ --collector.netdev \ --collector.diskstats关键指标说明node_load11分钟平均负载node_memory_MemFree_bytes空闲内存node_disk_io_time_seconds_total磁盘I/O耗时node_network_receive_bytes_total网络接收量Prometheus配置示例追加到scrape_configs- job_name: linux_nodes scrape_interval: 30s static_configs: - targets: - 192.168.1.20:9100 - 192.168.1.21:9100 relabel_configs: - source_labels: [__address__] regex: (.*):9100 target_label: instance replacement: $12.2 MySQL监控关键指标采集mysqld_exporter需要数据库只读账号权限建议创建专用账户CREATE USER exporterlocalhost IDENTIFIED BY StrongPassword WITH MAX_USER_CONNECTIONS 3; GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO exporterlocalhost;配置.my.cnf文件[client] userexporter passwordStrongPassword启动参数优化nohup /usr/local/mysqld_exporter/mysqld_exporter \ --config.my-cnf/usr/local/mysqld_exporter/.my.cnf \ --web.listen-address:9104 \ --collect.info_schema.processlist \ --collect.info_schema.innodb_metrics \ --collect.engine_innodb_status \ /var/log/mysqld_exporter.log 21 3. Grafana可视化高级技巧3.1 数据源配置最佳实践在Grafana的Configuration Data Sources中添加Prometheus时建议设置Scrape interval与Prometheus配置保持一致默认15sHTTP MethodGET避免POST可能导致的413错误Custom query parameterstimeout30增加查询超时时间注意对于大型环境可启用Prometheus的external_labels区分不同数据中心global: external_labels: region: east-1 env: production3.2 仪表板模板智能应用推荐的核心仪表板模板监控对象模板ID关键指标Linux8919CPU/Memory/Disk/NetworkMySQL7362QPS/连接数/慢查询/缓冲池Redis11835内存/命中率/阻塞客户端/命令统计Windows10467服务状态/CPU/内存/磁盘队列导入模板的三种方式直接输入模板ID需联网下载JSON文件本地导入复制模板内容到剪贴板变量使用技巧{ __inputs: [ { name: DS_PROMETHEUS, label: Prometheus, description: , type: datasource, pluginId: prometheus, pluginName: Prometheus } ], __elements: { server: { selected: true, text: 192.168.1.20, value: 192.168.1.20 } } }4. 生产环境调优策略4.1 Prometheus性能优化当监控目标超过50个时需要调整这些参数# prometheus.yml优化项 global: scrape_interval: 1m evaluation_interval: 1m scrape_timeout: 30s storage: tsdb: retention: 15d wal_compression: true out_of_order_time_window: 1h query: lookback_delta: 5m max_concurrency: 20内存占用优化命令# 限制内存使用示例4GB /usr/local/prometheus/prometheus \ --config.file/usr/local/prometheus/prometheus.yml \ --storage.tsdb.retention.time15d \ --web.config.file/usr/local/prometheus/web.yml \ --storage.tsdb.max-block-duration2h \ --storage.tsdb.min-block-duration2h \ --query.max-samples50000000 \ --query.timeout2m4.2 告警规则设计模式在/usr/local/prometheus/rules目录下创建告警规则文件groups: - name: host-alerts rules: - alert: HighCPUUsage expr: 100 - (avg by(instance)(irate(node_cpu_seconds_total{modeidle}[5m])) * 100) 80 for: 10m labels: severity: warning annotations: summary: High CPU usage on {{ $labels.instance }} description: {{ $labels.instance }} CPU usage is {{ $value }}% - alert: DiskWillFillIn4Hours expr: predict_linear(node_filesystem_free_bytes{mountpoint/}[1h], 4*3600) 0 for: 30m labels: severity: critical annotations: summary: Disk on {{ $labels.instance }} will fill in 4 hours5. 跨平台监控特殊处理5.1 Windows系统监控要点wmi_exporter安装后需要检查的服务Get-Service windows_exporter | Select Status, StartType常用性能计数器windows_cpu_time_total处理器时间百分比windows_memory_available_bytes可用物理内存windows_logical_disk_free_bytes磁盘剩余空间windows_net_bytes_total网络流量5.2 Redis监控深度配置启动redis_exporter时建议添加认证参数./redis_exporter \ -redis.addr redis://192.168.1.50:6379 \ -redis.password yourpassword \ -web.listen-address :9121 \ -namespace redis \ -include-system-metrics关键监控指标redis_connected_clients当前客户端连接数redis_memory_used_bytesRedis内存使用量redis_commands_processed_total命令执行总数redis_keyspace_hits_total键空间命中次数6. 系统集成与批量管理6.1 使用Ansible批量部署exporter创建ansible playbook文件deploy_exporters.yml--- - hosts: linux_servers tasks: - name: Download node_exporter get_url: url: https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz dest: /tmp/node_exporter.tar.gz - name: Install node_exporter unarchive: src: /tmp/node_exporter.tar.gz dest: /usr/local/ remote_src: yes notify: - restart node_exporter handlers: - name: restart node_exporter systemd: name: node_exporter state: restarted enabled: yes6.2 服务发现动态配置基于文件的服务发现配置示例scrape_configs: - job_name: linux_servers file_sd_configs: - files: - /etc/prometheus/targets/linux_servers.yml refresh_interval: 5m对应的目标文件/etc/prometheus/targets/linux_servers.yml- labels: env: production role: app_server targets: - 192.168.1.20:9100 - 192.168.1.21:9100 - labels: env: staging role: db_server targets: - 192.168.1.30:91007. 故障排查与日常维护7.1 常见问题诊断流程指标收集失败排查步骤检查exporter进程状态systemctl status node_exporter验证端口连通性telnet 192.168.1.20 9100查看原始指标curl http://192.168.1.20:9100/metrics检查Prometheus日志journalctl -u prometheus -f验证配置语法promtool check config prometheus.yml7.2 数据存储优化策略TSDB维护命令# 查看块数据状态 /usr/local/prometheus/promtool tsdb analyze /var/lib/prometheus/data # 清理过期数据 /usr/local/prometheus/promtool tsdb clean --retention.keep-last 10 /var/lib/prometheus/data # 备份重要数据 aws s3 sync /var/lib/prometheus/data s3://your-bucket/prometheus-backup/8. 安全加固方案8.1 网络层防护建议的防火墙规则# 只允许监控服务器访问exporter端口 iptables -A INPUT -p tcp --dport 9100 -s 192.168.1.10 -j ACCEPT iptables -A INPUT -p tcp --dport 9100 -j DROP # Grafana启用HTTPS [server] protocol https http_port 3000 domain yourdomain.com cert_file /etc/grafana/server.crt cert_key /etc/grafana/server.key8.2 认证授权配置Prometheus基础认证配置web.ymlbasic_auth_users: admin: $2y$10$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGrafana多租户管理[auth.anonymous] enabled false [auth.basic] enabled true [users] allow_sign_up false auto_assign_org true auto_assign_org_role Viewer9. 扩展监控能力9.1 黑盒监控实践安装blackbox_exporterwget https://github.com/prometheus/blackbox_exporter/releases/download/v0.22.0/blackbox_exporter-0.22.0.linux-amd64.tar.gz配置示例监控HTTP服务modules: http_2xx: prober: http timeout: 5s http: valid_status_codes: [200] method: GET preferred_ip_protocol: ip49.2 自定义指标暴露Python应用示例使用Prometheus客户端库from prometheus_client import start_http_server, Counter REQUEST_COUNT Counter(app_requests_total, Total HTTP requests) API_TIME Gauge(app_api_response_seconds, API response time) app.route(/metrics) def metrics(): return generate_latest() app.route(/api) def handle_api(): start_time time.time() REQUEST_COUNT.inc() # API处理逻辑 API_TIME.set(time.time() - start_time)10. 性能基准测试10.1 资源消耗参考值典型环境中的资源占用组件CPU占用内存占用磁盘IO网络流量Prometheus2核4GB50IOPS5Mbpsnode_exporter0.1核50MB2IOPS100KbpsGrafana1核1GB10IOPS1Mbps10.2 极限压力测试使用prombench工具模拟高负载docker run -it --rm \ -e PROMETHEUS_URLhttp://192.168.1.10:9090 \ -e QUERY_COUNT1000 \ -e CONCURRENT20 \ prom/prombench测试结果分析要点查询延迟P99应2s内存增长速率应50MB/min样本接收速率应50k samples/sTSDB压缩不应出现阻塞