特别说明,Mac系统需要关闭SIP才可以正常使用!即便开启了,某些位置也可能导致你无法正常使用。去他妈的,本教程不适合Mac Apple Chip安装的Docker!!

创建网络

docker network create elk

安装ElasticSearch

docker pull elasticsearch:7.7.1

Docker配置ElasticSearch时,无法直接挂载不方便导出elasticsearch.yml配置文件。我们可以在启动的时候,进入容器修改。(我们也可以从容器拷贝出来配置文件,到我们本地,我们本地修改了elasticsearch.yml配置文件后再在塞入容器。)

docker run --privileged=true --network elk --name elasticsearch -d -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -e "discovery.type=single-node" -p 9200:9200 -p 9300:9300 -v /home/docker/elk/es/data/:/usr/share/elasticsearch/data/ -v /home/docker/elk/es/logs/:/usr/share/elasticsearch/logs/ elasticsearch:7.7.1

进入容器修改文件

docker exec -it 容器ID /bin/bash

修改文件

vi config/elasticsearch.yml

添加如下配置

cluster.name: "docker-cluster"
network.host: 0.0.0.0

配置完毕,记得退出ElasticSearch容器。exit

重启ElasticSearch

docker restart 容器ID

等待一会,查看ElasticSearch是否启动 curl ip:9200

{
    "name":"dbc23c79f8ca",
    "cluster_name":"docker-cluster",
    "cluster_uuid":"SNSkQPWPREKJg_MAxXB30A",
    "version":{
        "number":"7.7.1",
        "build_flavor":"default",
        "build_type":"docker",
        "build_hash":"ad56dce891c901a492bb1ee393f12dfff473a423",
        "build_date":"2020-05-28T16:30:01.040088Z",
        "build_snapshot":false,
        "lucene_version":"8.5.1",
        "minimum_wire_compatibility_version":"6.8.0",
        "minimum_index_compatibility_version":"6.0.0-beta1"
    },
    "tagline":"You Know, for Search"
}

Kibana

拉取Kibana

docker pull kibana:7.7.1

启动Kibana

docker run --privileged=true -d --net elk --name kibana -v /home/docker/elk/kibana:/config -p 5601:5601 kibana:7.7.1

修改Kibana配置文件

进入kibana容器

docker exec -it 容器ID /bin/bash

修改配置文件,实现可视化界面中文

vim /home/docker/elk/kibana/kibana.yml
docker exec -it 容器ID /bin/bash
vi config/kibana.yml

添加代码 或修改 添加中文(与下文保持一直就行)

server.host: "0"
server.shutdownTimeout: "5s"
#elasticsearch.hosts: [ "http://elasticsearch:9200" ]
elasticsearch.hosts: [ "http://192.168.10.13:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
i18n.locale: "zh-CN"

logstash配置

logstash介绍

Logstash 配置文件有两个必需元素,输入(inputs)和输出(ouputs),以及一个可选元素 filters。输入插件配置来源数据,过滤器插件在你指定时修改数据,输出插件将数据写入目标。

看到上图,就能了解logstash的作用,他就是中间文件处理的作用。有filebeat上传文件到Logstash,然后处理一下发往Elasticsearch。然后Kibana就展示Elasticsearch的数据。

我们首先需要创建一个配置文件,配置内容如下图所示:

Logstash – 安装 – 启动

创建文件夹(用于放logstash配置文件)

mkdir /home/docker/elk/logstash

创建2个文件(一共需要创建2个文件,作用不一样)

文件1:vim /home/docker/elk/logstash/logstash.yml ,参考配置:https://www.elastic.co/guide/en/logstash/7.7/logstash-settings-file.html

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: 192.168.10.13:9200

文件2:vim /home/docker/elk/logstash/logstash.conf,参考配置:https://www.elastic.co/guide/en/logstash/7.7/configuration.html

input {
  beats {
    port => 4567
  }
}
filter {
  #Only matched data are send to output.
}
output {
  elasticsearch {
    hosts  => ["http://192.168.10.13:9200"]   #ElasticSearch host, can be array.
    index  => "logapp-%{+YYYY.MM.dd}"         #The index to write data to.
  }
}

启动Logstash

docker pull logstash:7.7.1

暴露出:4567端口

docker run --privileged=true -d -p 4567:4567 --name logstash -p 5044:5044 -p 5045:5045 -v /home/docker/elk/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf -v /home/docker/elk/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml logstash:7.7.1

小提示:配置文件中分为三个模块,input,filter,output很好理解,就是把东西通过input输入,通过filter过滤,通过output输出。其中input中的beats插件就是我们下面要启动的filebeat。filebeat通过4567端口将收集的日志发送给logstash,当然想用哪个端口随便你。

FileBeat 安装 – 启动

FileBeat日志采集器作用就是:收集好了日志,发往logstash。(需要在配置文件指定logstash的Host)

logstash保留出一个API。然后FileBeat在任何地方安装后,就可将采集的数据发送给logstash。然后交由logstash自己处理日志。

制作日志文件

注意这个文件,使我们创建的日志文件这是nginx的文件

mkdir -p /var/log/logapp && vim /var/log/logapp/app.info.log

塞入日志数据

数据点击展开,自己复制
121.5.135.177 - - [14/Nov/2022:21:14:35 +0800] "POST /wp-cron.php?doing_wp_cron=1668431675.5358579158782958984375 HTTP/1.1" 200 31 "https://www.zanglikun.com/wp-cron.php?doing_wp_cron=1668431675.5358579158782958984375" "WordPress/6.1; https://www.zanglikun.com"
185.191.171.10 - - [14/Nov/2022:21:14:35 +0800] "GET /page/15?big=tag%2Fdorico%2F HTTP/1.1" 200 22007 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
121.5.135.177 - - [14/Nov/2022:21:14:56 +0800] "POST /wp-cron.php?doing_wp_cron=1668431696.5862400531768798828125 HTTP/1.1" 200 31 "https://www.zanglikun.com/wp-cron.php?doing_wp_cron=1668431696.5862400531768798828125" "WordPress/6.1; https://www.zanglikun.com"
185.191.171.16 - - [14/Nov/2022:21:14:56 +0800] "GET /page/5?big=tag%2Foo-safeerase-download%2F HTTP/1.1" 200 21849 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
36.161.110.229 - - [14/Nov/2022:21:15:02 +0800] "POST /wp-admin/admin-ajax.php HTTP/2.0" 200 101 "https://www.zanglikun.com/wp-admin/post.php?post=14943&action=edit" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
185.191.171.23 - - [14/Nov/2022:21:15:22 +0800] "GET /robots.txt HTTP/1.1" 200 135 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
185.191.171.33 - - [14/Nov/2022:21:15:24 +0800] "GET /page/15?big=tag%2Fanytoiso-3-9-6-crack-2021%2F HTTP/1.1" 200 22020 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
180.211.97.53 - - [14/Nov/2022:21:15:36 +0800] "POST /xmlrpc.php HTTP/1.1" 503 18945 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96"
121.5.135.177 - - [14/Nov/2022:21:15:46 +0800] "POST /wp-cron.php?doing_wp_cron=1668431746.1603119373321533203125 HTTP/1.1" 200 31 "https://www.zanglikun.com/wp-cron.php?doing_wp_cron=1668431746.1603119373321533203125" "WordPress/6.1; https://www.zanglikun.com"
185.191.171.15 - - [14/Nov/2022:21:15:46 +0800] "GET /page/23?big=tag%2Fbitwig-studio-crack-download%2F HTTP/1.1" 200 21875 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
185.191.171.21 - - [14/Nov/2022:21:15:51 +0800] "GET /?big=tag%2Fcracks-software%2F HTTP/1.1" 200 22024 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
121.5.135.177 - - [14/Nov/2022:21:15:57 +0800] "POST /wp-cron.php?doing_wp_cron=1668431757.5445189476013183593750 HTTP/1.1" 200 31 "https://www.zanglikun.com/wp-cron.php?doing_wp_cron=1668431757.5445189476013183593750" "WordPress/6.1; https://www.zanglikun.com"
185.191.171.25 - - [14/Nov/2022:21:15:57 +0800] "GET /page/2?big=tag%2Fcpu-booster-windows-10%2F HTTP/1.1" 200 22000 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
36.161.110.229 - - [14/Nov/2022:21:16:01 +0800] "POST /wp-admin/admin-ajax.php HTTP/2.0" 200 101 "https://www.zanglikun.com/wp-admin/post.php?post=14943&action=edit" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
185.191.171.17 - - [14/Nov/2022:21:16:04 +0800] "GET /?big=tag%2Fmatebook-x-pro%2F HTTP/1.1" 200 22024 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
121.5.135.177 - - [14/Nov/2022:21:16:11 +0800] "POST /wp-cron.php?doing_wp_cron=1668431771.1240489482879638671875 HTTP/1.1" 200 31 "https://www.zanglikun.com/wp-cron.php?doing_wp_cron=1668431771.1240489482879638671875" "WordPress/6.1; https://www.zanglikun.com"
185.191.171.16 - - [14/Nov/2022:21:16:11 +0800] "GET /page/29?big=tag%2Fmalwarebytes-phone-number%2F HTTP/1.1" 200 21836 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
51.222.253.2 - - [14/Nov/2022:21:16:14 +0800] "GET /date/2021/02/?order=comment_count&price_type=0&cao_type=0 HTTP/2.0" 301 0 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
185.191.171.2 - - [14/Nov/2022:21:16:30 +0800] "GET /page/27?big=tag%2Ffreemake-video-converter-4-1-11-80-crack%2F HTTP/1.1" 200 22295 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
185.191.171.16 - - [14/Nov/2022:21:16:35 +0800] "GET /page/9?action=edit&big=wp-admin%2Fpost.php%3Fpost%3D318 HTTP/1.1" 200 21876 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
121.5.135.177 - - [14/Nov/2022:21:16:38 +0800] "POST /wp-cron.php?doing_wp_cron=1668431798.1814179420471191406250 HTTP/1.1" 200 31 "https://www.zanglikun.com/wp-cron.php?doing_wp_cron=1668431798.1814179420471191406250" "WordPress/6.1; https://www.zanglikun.com"
185.191.171.18 - - [14/Nov/2022:21:16:38 +0800] "GET /page/19?big=tag%2Fglary-utilities-free%2F HTTP/1.1" 200 22067 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
185.191.171.2 - - [14/Nov/2022:21:16:41 +0800] "GET /page/32?big=tag%2Fhelicon-remote-crack%2F HTTP/1.1" 200 22231 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
121.5.135.177 - - [14/Nov/2022:21:16:55 +0800] "POST /wp-cron.php?doing_wp_cron=1668431815.6226899623870849609375 HTTP/1.1" 200 31 "https://www.zanglikun.com/wp-cron.php?doing_wp_cron=1668431815.6226899623870849609375" "WordPress/6.1; https://www.zanglikun.com"
185.191.171.11 - - [14/Nov/2022:21:16:55 +0800] "GET /share/page/3?cao_type=4&order=views HTTP/1.1" 200 19954 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"

创建FileBeat配置文件

完整配置文件设置:https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html

mkdir -p /home/docker/elk/filebeat/ && vim /home/docker/elk/filebeat/filebeat.yml

往filebeat.yml填入:

filebeat.inputs:
- type: log
  paths:
    - /var/log/logapp/app.info.log
output.logstash:
    hosts: ["192.168.10.100:4567"]

启动FileBeat

docker pull store/elastic/filebeat:7.7.1

这里是制定了日志文件所在的位置/var/log/logapp

docker run --privileged=true -u root -v /var/log/logapp:/var/log/logapp:rw -v /home/docker/elk/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro -e setup.kibana.host=192.168.10.13:5601 -d --name filebeat docker.elastic.co/beats/filebeat:7.7.1

使用ELK

找到索引模式

配置日志索引

填写「索引模式」

点击下一步,配置其他设置

接下来,你就看到了ElasticSearch自动生成索引了

前往Discover查看日志分析

特别说明:安装程序如果遇到问题补充命令

容器启动时可查看容器日志

docker logs --since="2016-07-01" --tail=500 容器ID

查看容器内容地址

docker inspect --format '{{ .NetworkSettings.IPAddress }}' 容器ID