logstash7.8.0 and Kibana7.8.0

安装logstash

• 配置yum源

[root@iZ2ze2jlupmjlwwfuyg30gZ ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@iZ2ze2jlupmjlwwfuyg30gZ ~]# vim /etc/yum.repos.d/logstash.repo
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

• 安装

[root@iZ2ze2jlupmjlwwfuyg30gZ ~]# yum install logstash -y

• 或者下载rpm包

[root@iZ2ze2jlupmjlwwfuyg30gZ ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.8.0.rpm
[root@iZ2ze2jlupmjlwwfuyg30gZ ~]# rpm -ivh logstash-7.8.0.rpm

• 将执行路径写入环境变量

[root@iZ2ze2jlupmjlwwfuyg30gZ ~]# vim /etc/profile
PATH=$PATH:/usr/share/logstash/bin
export PATH
[root@iZ2ze2jlupmjlwwfuyg30gZ ~]# source /etc/profile
[root@iZ2ze2jlupmjlwwfuyg30gZ ~]# logstash -V
logstash 7.8.0

Logstash-条件判断

比较操作符：
相等: ==, !=, <, >, <=, >=
正则: =~(匹配正则), !~(不匹配正则)
包含: in(包含), not in(不包含)
布尔操作符：
and(与), or(或), nand(非与), xor(非或)
一元运算符：
!(取反)
()(复合表达式), !()(对复合表达式结果取反)

配置日志拉去路径

[root@iZ2ze2jlupmjlwwfuyg30iZ ~]#  cd /etc/logstash/conf.d/
[root@iZ2ze2jlupmjlwwfuyg30iZ conf.d]# vim logstash-to-es.conf
input {
    file {
        path => ["/home/xiaoxin/app/c-user-center/log/stdout.log"]
        type => "system"
        tags => ["c-user-center"]
        start_position => "beginning"
    }
    file {
        path => ["/home/xiaoxin/app/order-status-sync/log/stdout.log"]
        type => "system"
        tags => ["order-status-sync"]
        start_position => "beginning"
    }
}
filter {
 
}
output {
    if [type] == "system" {
        if [tags][0] == "c-user-center" {
            elasticsearch {
                hosts  => ["http://10.10.0.252:9200","http://10.10.0.253:9200","http://10.10.0.254:9200"]
                index  => "c-user-center-%{+YYYY.MM.dd}"
            }
            stdout { codec=> rubydebug }
        }
        else if [tags][0] == "order-status-sync" {
            elasticsearch {
                hosts  => ["http://10.10.0.252:9200","http://10.10.0.253:9200","http://10.10.0.254:9200"]
                index  => "order-status-sync-%{+YYYY.MM.dd}"
            }
            stdout { codec=> rubydebug }
        }
    }
}

• 测试是否正确（测试配置文件正确性，如下：-t表示测试，-f 指定文件位置）

[root@iZ2ze2jlupmjlwwfuyg30iZ conf.d]# logstash -t -f /etc/logstash/conf.d/logstash-to-es.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2020-07-03 17:53:28.589 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2020-07-03 17:53:28.611 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2020-07-03 17:53:29.100 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-07-03 17:53:30.942 [LogStash::Runner] Reflections - Reflections took 66 ms to scan 1 urls, producing 21 keys and 41 values
Configuration OK
[INFO ] 2020-07-03 17:53:32.345 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

• 直接运行，加-r参数

[root@iZ2ze2jlupmjlwwfuyg30iZ conf.d]#  logstash -r -f /etc/logstash/conf.d/logstash-to-es.conf

• 启动服务

[root@iZ2ze2jlupmjlwwfuyg30iZ ~]# systemctl start logstash
[root@iZ2ze2jlupmjlwwfuyg30iZ ~]# systemctl enable logstash
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.

Elasticsearch上查看

安装Kibana

• 在/etc/yum.repos.d/下新建kibana.repo  配置YUM源地址 内容如下：

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
[root@elk-es1 ~]# yum install -y kibana

• 或者使用rpm包

[root@elk-es1 ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.8.0-x86_64.rpm
[root@elk-es1 ~]# rpm -ivh kibana-7.8.0-x86_64.rpm

• 加入es集群任意节点

[root@elk-es1 ~]# vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://10.10.0.252:9200"]

• 启动服务

[root@elk-es1 ~]# systemctl start kibana.service
[root@elk-es1 ~]# systemctl enable kibana.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.

• 访问http://ip:5601/

• 可以看到有在es中生成的索引

• 不使用时间戳过滤

• 查看拉去过来的日志

• 根据所需，选择对应的日志索引即可。