前言

后续开发项目要用到 Filebeat 对日志做收集和处理。本文介绍了 ELK 技术中的 Filebeat，用于轻量级的日志收集和分析

参考资料：

视频教程：Elastic Stack（ELK）从入门到实践

官方文档：Filebeat overview | Filebeat Reference [8.14] | Elastic

一、安装部署

环境：

1
2
root@Andrew:~# cat /proc/version
Linux version 5.15.153.1-microsoft-standard-WSL2 (root@941d701f84f1) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP Fri Mar 29 23:14:13 UTC 2024

Linux环境下安装命令：

1
2
3
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.14.3-linux-x86_64.tar.gz

tar xzvf filebeat-8.14.3-linux-x86_64.tar.gz

解压后将文件夹重命名，进入目录：

1
2
3
mv filebeat-8.14.3-linux-x86_64 filebeat

cd filebeat/

压缩包解压安装的filebeat文件结构与deb/rpm等安装方式不同，详见：Directory layout | Filebeat Reference [8.14] | Elastic

二、启动示例

在解压后的根目录下新建测试的配置文件：

1
vi test.yml

从控制台输入的配置如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
filebeat.inputs:
- type: stdin
  enable: true  # default true
  # include_lines/exclude_lines support regular expresions
  include_lines: ["^andrew"]    # starts with "andrew"
  exclude_lines: ["www"]        # exclude which contains "www"
  tags: ["stdin"]                # define tags in output
  fields:                       # define specific fields in output
      author: andrew
      date: 2024-7-26
  fields_in_root: false        # default false

output.console:
  enable: true  # default true
  pretty: true  # default false

上述配置文件表示：

从标准输入读取输入
读取包括开头为”andrew”，且不包括”www”的行
为输出的数据添加tags为”stdin”
为输出的数据添加fields字段 author为”andrew” date为”2024-7-26”
fields添加的字段不位于根路径下，

输出到控制台，开启pretty美化输出

输出以下命令启动filebeat

1
./filebeat -e -c test.yml

-e 参数指定输出到stderr而不是配置文件指定的输出（仍然需要配置，否则无法启动）
-c 指定启动的配置文件，未指定时默认为filebeat.yml

在标准输入中输入，”andrew”、”andreww”、”andrewww”，观察输出情况

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
andrew
andreww
andrewww
{
  "@timestamp": "2024-07-26T03:15:21.470Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.14.3"
  },
  "log": {
    "file": {
      "path": ""
    },
    "offset": 0
  },
  "message": "andrew",
  "tags": [
    "stdin"
  ],
  "input": {
    "type": "stdin"
  },
  "fields": {
    "author": "andrew",
    "date": "2024-7-26"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "host": {
    "name": "Andrew"
  },
  "agent": {
    "id": "57b956ef-540a-45a4-bc04-dc192b75c801",
    "name": "Andrew",
    "type": "filebeat",
    "version": "8.14.3",
    "ephemeral_id": "b4c352d9-1ed1-4369-95ec-47455c21e272"
  }
}
{
  "@timestamp": "2024-07-26T03:15:22.628Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.14.3"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": ""
    }
  },
  "message": "andreww",
  "tags": [
    "stdin"
  ],
  "input": {
    "type": "stdin"
  },
  "fields": {
    "author": "andrew",
    "date": "2024-7-26"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "host": {
    "name": "Andrew"
  },
  "agent": {
    "ephemeral_id": "b4c352d9-1ed1-4369-95ec-47455c21e272",
    "id": "57b956ef-540a-45a4-bc04-dc192b75c801",
    "name": "Andrew",
    "type": "filebeat",
    "version": "8.14.3"
  }
}

观察到filebeat只输出了两个json，message分别为”andrew”、”andreww”，说明include_lines和exclude_lines生效，看到tags存在”stdin”，以及fields下有两个字段”author: “andrew”, “date”: “2024-7-26”

从日志文件输入的配置如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
filebeat.inputs:
  - type: log
    paths:
      - /root/filebeat/logs/*.log
    include_lines: ['^andrew']
    exclude_lines: ['ww']
    encoding: utf-8
    tags: ['log']
    fields:
      author: andrew
    fields_under_root: true
output.console:
  pretty: true
  enable: true

测试结果

1
2
3
root@Andrew:~/filebeat/logs# echo andrew >> a.log
root@Andrew:~/filebeat/logs# echo andreww >> a.log
root@Andrew:~/filebeat/logs# echo andrewww >> a.log

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
{
  "@timestamp": "2024-07-27T08:02:55.742Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.14.3"
  },
  "host": {
    "name": "Andrew"
  },
  "agent": {
    "id": "57b956ef-540a-45a4-bc04-dc192b75c801",
    "name": "Andrew",
    "type": "filebeat",
    "version": "8.14.3",
    "ephemeral_id": "428921c4-3a41-4225-80b5-3d2a3c6aa4bb"
  },
  "log": {
    "offset": 31,
    "file": {
      "path": "/root/filebeat/logs/a.log"
    }
  },
  "message": "andrew",
  "tags": [
    "log"
  ],
  "input": {
    "type": "log"
  },
  "author": "andrew",
  "ecs": {
    "version": "8.0.0"
  }
}

指定输出的配置（如Kafka、Elasticsearch等）参考官方文档

三、使用modules

通过 filebeat modules 命令操作模块化的配置

1
2
3
filebeat modules list          # 查看所有modules
filebeat modules enable nginx  # 启动nginx模块
filebeat modules disable nginx # 关闭nginx模块

编辑 modules.d 目录下的文件更改配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
- module: nginx
  # Access logs
  access:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Error logs
  error:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths: