Всем привет!
Есть лог-файл C:\Users\user\Documents\Adobe\Adobe Media Encoder\15.0\AMEEncodingLog.txt
, который я хочу отправить в ElasticSearch с помощью Filebeat. Проблема в том, что этот журнал не отправляется, и я не могу понять причину, почему это происходит.
Вот что я делал.
Установил Filebeat на Win 10 машине (внутри VirtualBox) согласно инструкции с официального сайта.
Изменил filebeat.yml (конфиг ниже)
Запустил из PowerShell команду
.\filebeat.exe -e --once
(журнал ниже).
Также запускал .\filebeat.exe -e
и ждал. Также делал uninstall для Filebeat, и устанавливал заново. Результат один - нет сообщения о явной ошибке, а лог-файл не отправляется. Причем я установил Winlogbeat похожим образом, и сообщения в Кибане появились. Также появились дашборды в Кибане для Filebeat, а вот собственно данных - нет.
Что я делаю не так? В чем ошибка?
filebeat.yml:
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# filestream is an input for collecting log messages from files.
- type: filestream
# Unique ID among all inputs, an ID is required.
id: media-encoder-log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- 'C:\Users\user\Documents\Adobe\Adobe Media Encoder\15.0\AMEEncodingLog.txt'
encoding: cp1251
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
exclude_lines: ['^--------', '^Файл регистрации создан']
scan_frequency: 60s
line_terminator: carriage_return_line_feed
parsers:
- multiline:
type: pattern
pattern: '^\r\n$'
negate: true
match: before
max_lines: 50
# ============================== Filebeat modules ==============================
filebeat.config.modules:
enabled: false
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
# =================================== Kibana ===================================
setup.kibana:
host: "http://192.168.56.1:5601"
# ================================== Outputs ===================================
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["http://192.168.56.1:9200"]
Результат запуска .\filebeat.exe -e --once
:
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.189+0300","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [C:\\Program Files\\filebeat-8.2.0-windows-x86_64] Config path: [C:\\Program Files\\filebeat-8.2.0-windows-x86_64] Data path: [C:\\Program Files\\filebeat-8.2.0-windows-x86_64\\data] Logs path: [C:\\Program Files\\filebeat-8.2.0-windows-x86_64\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.192+0300","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: 203f3c11-9fe7-4b51-8dbc-ed5ee4878073","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.193+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"C:\\Program Files\\filebeat-8.2.0-windows-x86_64","data":"C:\\Program Files\\filebeat-8.2.0-windows-x86_64\\data","home":"C:\\Program Files\\filebeat-8.2.0-windows-x86_64","logs":"C:\\Program Files\\filebeat-8.2.0-windows-x86_64\\logs"},"type":"filebeat","uuid":"203f3c11-9fe7-4b51-8dbc-ed5ee4878073"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.195+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1072},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"045da3a1bb89944373c33332c18ca99ef6192df2","libbeat":"8.2.0","time":"2022-04-19T23:31:04.000Z","version":"8.2.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.195+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1075},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":4,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.199+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-19T19:44:35.49+03:00","name":"win10-new","ip":["fe80::4cb1:59de:182a:1bd9/64","192.168.56.10/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.19041.985 (WinBuild.160101.0800)","mac":["08:00:27:76:48:0b"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"19043.985"},"timezone":"MSK","timezone_offset_sec":10800,"id":"053734c1-c000-4f7a-8286-8e3682fcd350"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.199+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"C:\\Program Files\\filebeat-8.2.0-windows-x86_64","exe":"C:\\Program Files\\filebeat-8.2.0-windows-x86_64\\filebeat.exe","name":"filebeat.exe","pid":5008,"ppid":7288,"start_time":"2022-05-19T19:55:14.984+0300"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.200+0300","log.origin":{"file.name":"instance/beat.go","file.line":325},"message":"Setup Beat: filebeat; Version: 8.2.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.204+0300","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: http://192.168.56.1:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.218+0300","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: win10-new","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.219+0300","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.222+0300","log.origin":{"file.name":"instance/beat.go","file.line":505},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.226+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.233+0300","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\Program Files\\filebeat-8.2.0-windows-x86_64\\data\\registry\\filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.237+0300","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.241+0300","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.241+0300","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.encoding filebeat.inputs.0.exclude_lines.0 filebeat.inputs.0.exclude_lines.1 filebeat.inputs.0.id filebeat.inputs.0.line_terminator filebeat.inputs.0.parsers.0.multiline.match filebeat.inputs.0.parsers.0.multiline.max_lines filebeat.inputs.0.parsers.0.multiline.negate filebeat.inputs.0.parsers.0.multiline.pattern filebeat.inputs.0.parsers.0.multiline.type filebeat.inputs.0.paths.0 filebeat.inputs.0.scan_frequency filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.243+0300","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 11643066496118975770)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.243+0300","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.243+0300","log.origin":{"file.name":"beater/filebeat.go","file.line":349},"message":"Running filebeat once. Waiting for completion ...","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.243+0300","log.origin":{"file.name":"beater/filebeat.go","file.line":351},"message":"All data collection completed. Shutting down.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.243+0300","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":111},"message":"Input filestream starting","service.name":"filebeat","id":"media-encoder-log","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.origin":{"file.name":"beater/crawler.go","file.line":155},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.origin":{"file.name":"beater/crawler.go","file.line":165},"message":"Stopping 1 inputs","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":170},"message":"Stopping input: 11643066496118975770","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":132},"message":"Input 'filestream' stopped","service.name":"filebeat","id":"media-encoder-log","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":138},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.origin":{"file.name":"beater/signalwait.go","file.line":88},"message":"Continue shutdown: All enqueued events being published.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":132},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":124},"message":"Input 'filestream' stopped","service.name":"filebeat","id":"media-encoder-log","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":166},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":137},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.246+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":192},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":93,"time":{"ms":93}},"total":{"ticks":202,"time":{"ms":202},"value":0},"user":{"ticks":109,"time":{"ms":109}}},"handles":{"open":187},"info":{"ephemeral_id":"90df14a7-0d88-4562-988e-df4685b284e5","uptime":{"ms":121},"version":"8.2.0"},"memstats":{"gc_next":19278240,"memory_alloc":10442272,"memory_sys":31759576,"memory_total":51802512,"rss":55164928},"runtime":{"goroutines":20}},"filebeat":{"events":{"active":0,"added":0,"done":0},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":4096}}},"registrar":{"states":{"cleanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":4}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.252+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":193},"message":"Uptime: 127.8201ms","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.252+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":160},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-19T19:55:15.252+0300","log.origin":{"file.name":"instance/beat.go","file.line":510},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
Результат запуска .\filebeat.exe test config
:
Config OK
Результат запуска .\filebeat.exe test output
:
elasticsearch: http://192.168.56.1:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 192.168.56.1
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
version: 8.2.0