Logstash基础操作-Filter
发表于:2024-09-24 作者:千家信息网编辑
千家信息网最后更新 2024年09月24日,Grok配置案例:##启动文件配置:# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasti
千家信息网最后更新 2024年09月24日Logstash基础操作-Filter
Grok配置案例:
##启动文件配置:# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{}}filter {grok {match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\%{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] }}output { stdout{ codec => "rubydebug" }}##输出文件内容172.16.213.132 [07/Feb/2018:16:24:19 +0800] "GET / HTTP/1.1" 403 5039##显示内容{ "@version" => "1", "@timestamp" => 2019-11-10T06:02:42.865Z, "host" => "localhost.localdomain", "message" => "172.16.213.132 [07/Feb/2018:16:24:19 +0800] \"GET / HTTP/1.1\" 403 5039", "timestamp" => "07/Feb/2018:16:24:19 +0800", "bytes" => "5039", "response" => "403", "clientip" => "172.16.213.132", "referrer" => "\"GET / HTTP/1.1\""}
Grok 过滤重复字段
## 配置文件# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] }}output { stdout{ codec => "rubydebug" }}
Grok搭配Date时间插件配置
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter {grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] }date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] }}output { stdout{ codec => "rubydebug" }}
Date 过滤重复得字段配置
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] }date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] }mutate { remove_field => [ "timestamp" ] }}output { stdout{ codec => "rubydebug" }}
综合练习配置参数
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate{ rename => {"response" => "response_new"} gsub => ["referrer", "\"", ""] remove_field => [ "timestamp" ] split => ["clientip", "."] }}output { stdout{ codec => "rubydebug" }}
Geoip 地理位置插件操作方式
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate{ remove_field => [ "timestamp" ] } geoip { source => "clientip" database => "/usr/local/include/GeoLite2-ASN_20191105/GeoLite2-ASN.mmdb" }}output { stdout{ codec => "rubydebug" } }
Geoip输出指定属性值
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate{ remove_field => [ "timestamp" ] }geoip {source => "clientip"#database => "/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb"database => "/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb"fields => ["city_name", "region_name", "country_name", "ip", "latitude", "longitude", "timezone"] }}output { stdout{ codec => "rubydebug" }}模拟数据:36.7.152.182 [07/Feb/2018:16:24:19 +0800] "GET / HTTP/1.1" 403 5039
综合实战
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{}}filter{grok{ match => {"message" => "%{TIMESTAMP_ISO8601:localtime}\|\~\|%{IP:clientip} \|\~\|%{GREEDYDATA:http_user_agent}\|\~\|%{GREEDYDATA:url} \|\~\|%{GREEDYDATA:mediaid}\|\~\|%{GREEDYDATA:osid}"} remove_field => [ "message" ] }date { match => ["localtime", "yyyy-MM-dd'T'HH:mm:ssZZ"] target => "@timestamp" }mutate { remove_field => ["localtime"] }geoip { source => "clientip" #database => "/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb" database => "/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb" fields => ["city_name", "region_name", "country_name", "ip", "latitude", "longitude", "timezone"] }}output { stdout { codec => "rubydebug" }}示例:2018-02-09T10:57:42+08:00|~|123.87.240.97|~|Mozilla/5.0(iPhone;CPU iPhone OS 11_2_2 like Mac OS X)AppleWebKit/604.4.7 Version/11.0 Mobile/15C202 Safari/604.1|~|http://m.sina.cn/cm/ads_ck_wap.html|~|12434785489009|~|DF45566587855P
配置
文件
内容
字段
插件
综合
输出
位置
参数
地理
地理位置
实战
属性
数据
方式
时间
案例
示例
基础
数据库的安全要保护哪些东西
数据库安全各自的含义是什么
生产安全数据库录入
数据库的安全性及管理
数据库安全策略包含哪些
海淀数据库安全审计系统
建立农村房屋安全信息数据库
易用的数据库客户端支持安全管理
连接数据库失败ssl安全错误
数据库的锁怎样保障安全
万德数据库的优势
mysql数据库和php
CNKI新版数据库中
德国 服务器
网络安全应急指挥小组领导
外网访问内网服务器包被丢弃
光纤通信网络技术
怎么关闭ept数据库连接设置
在国外做网络安全行业挣钱吗
sql数据库数据删除恢复
桌面软件开发逻辑
青芒教育网络技术
数据库每次都加modify
网络技术属于什么公司
数据库自动覆盖上一条
湖南网络安全咨询就业前景好
有哪些软件开发外包
备份服务器文件
没有服务器可以进网页吗
技术软件开发合作合同
101打印机服务器
菲律宾网络安全局
centos管理服务器
oa服务器异地访问
怎么在服务器上安装火绒
腾讯手机管家怎么更改服务器
嵌入式软件开发有哪些方向
网络技术人员职位描述
全息投影软件开发
0基础自学网络安全书籍