Docker构建ELK Docker集群日志收集系统

当我们搭建好Docker集群后就要解决如何收集日志的问题 ELK就提供了一套完整的解决方案 本文主要介绍使用Docker搭建ELK 收集Docker集群的日志

ELK简介

ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成

Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。

Logstash是一个完全开源的工具，他可以对你的日志进行收集、过滤，并将其存储供以后使用

Kibana 也是一个开源和免费的工具，它Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。

使用Docker搭建ELK平台

首先我们编辑一下 logstash的配置文件 logstash.conf

input {   udp {  port => 5000  type => json }}filter {  json {   source => "message"  }}output {  elasticsearch {       hosts => "elasticsearch:9200" #将logstash的输出到 elasticsearch 这里改成你们自己的host   }}

然后我们还需要需要一下Kibana 的启动方式

编写启动脚本 等待elasticserach 运行成功后启动

#!/usr/bin/env bash# Wait for the Elasticsearch container to be ready before starting Kibana.echo "Stalling for Elasticsearch" while true; do  nc -q 1 elasticsearch 9200 2>/dev/null && breakdoneecho "Starting Kibana"exec kibana

修改Dockerfile 生成自定义的Kibana镜像

FROM kibana:latestRUN apt-get update && apt-get install -y netcatCOPY entrypoint.sh /tmp/entrypoint.shRUN chmod +x /tmp/entrypoint.shRUN kibana plugin --install elastic/senseCMD ["/tmp/entrypoint.sh"]

同时也可以修改一下Kibana 的配置文件 选择需要的插件

# Kibana is served by a back end server. This controls which port to use.port: 5601# The host to bind the server to.host: "0.0.0.0"# The Elasticsearch instance to use for all your queries.elasticsearch_url: "http://elasticsearch:9200"# preserve_elasticsearch_host true will send the hostname specified in `elasticsearch`. If you set it to false,# then the host you use to connect to *this* Kibana instance will be sent.elasticsearch_preserve_host: true# Kibana uses an index in Elasticsearch to store saved searches, visualizations# and dashboards. It will create a new index if it doesn't already exist.kibana_index: ".kibana"# If your Elasticsearch is protected with basic auth, this is the user credentials# used by the Kibana server to perform maintence on the kibana_index at statup. Your Kibana# users will still need to authenticate with Elasticsearch (which is proxied thorugh# the Kibana server)# kibana_elasticsearch_username: user# kibana_elasticsearch_password: pass# If your Elasticsearch requires client certificate and key# kibana_elasticsearch_client_crt: /path/to/your/client.crt# kibana_elasticsearch_client_key: /path/to/your/client.key# If you need to provide a CA certificate for your Elasticsarech instance, put# the path of the pem file here.# ca: /path/to/your/CA.pem# The default application to load.default_app_id: "discover"# Time in milliseconds to wait for elasticsearch to respond to pings, defaults to# request_timeout setting# ping_timeout: 1500# Time in milliseconds to wait for responses from the back end or elasticsearch.# This must be > 0request_timeout: 300000# Time in milliseconds for Elasticsearch to wait for responses from shards.# Set to 0 to disable.shard_timeout: 0# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying# startup_timeout: 5000# Set to false to have a complete disregard for the validity of the SSL# certificate.verify_ssl: true# SSL for outgoing requests from the Kibana Server (PEM formatted)# ssl_key_file: /path/to/your/server.key# ssl_cert_file: /path/to/your/server.crt# Set the path to where you would like the process id file to be created.# pid_file: /var/run/kibana.pid# If you would like to send the log output to a file you can set the path below.# This will also turn off the STDOUT log output.log_file: ./kibana.log# Plugins that are included in the build, and no longer found in the plugins/ folderbundled_plugin_ids: - plugins/dashboard/index - plugins/discover/index - plugins/doc/index - plugins/kibana/index - plugins/markdown_vis/index - plugins/metric_vis/index - plugins/settings/index - plugins/table_vis/index - plugins/vis_types/index - plugins/visualize/index