Logstash基础操作-Filter
发表于:2024-09-24 作者:千家信息网编辑
千家信息网最后更新 2024年09月24日,Grok配置案例:##启动文件配置:# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasti
千家信息网最后更新 2024年09月24日Logstash基础操作-Filter
Grok配置案例:
##启动文件配置:# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{}}filter {grok {match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\%{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] }}output { stdout{ codec => "rubydebug" }}##输出文件内容172.16.213.132 [07/Feb/2018:16:24:19 +0800] "GET / HTTP/1.1" 403 5039##显示内容{ "@version" => "1", "@timestamp" => 2019-11-10T06:02:42.865Z, "host" => "localhost.localdomain", "message" => "172.16.213.132 [07/Feb/2018:16:24:19 +0800] \"GET / HTTP/1.1\" 403 5039", "timestamp" => "07/Feb/2018:16:24:19 +0800", "bytes" => "5039", "response" => "403", "clientip" => "172.16.213.132", "referrer" => "\"GET / HTTP/1.1\""}
Grok 过滤重复字段
## 配置文件# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] }}output { stdout{ codec => "rubydebug" }}
Grok搭配Date时间插件配置
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter {grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] }date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] }}output { stdout{ codec => "rubydebug" }}
Date 过滤重复得字段配置
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] }date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] }mutate { remove_field => [ "timestamp" ] }}output { stdout{ codec => "rubydebug" }}
综合练习配置参数
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate{ rename => {"response" => "response_new"} gsub => ["referrer", "\"", ""] remove_field => [ "timestamp" ] split => ["clientip", "."] }}output { stdout{ codec => "rubydebug" }}
Geoip 地理位置插件操作方式
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate{ remove_field => [ "timestamp" ] } geoip { source => "clientip" database => "/usr/local/include/GeoLite2-ASN_20191105/GeoLite2-ASN.mmdb" }}output { stdout{ codec => "rubydebug" } }
Geoip输出指定属性值
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{ }}filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate{ remove_field => [ "timestamp" ] }geoip {source => "clientip"#database => "/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb"database => "/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb"fields => ["city_name", "region_name", "country_name", "ip", "latitude", "longitude", "timezone"] }}output { stdout{ codec => "rubydebug" }}模拟数据:36.7.152.182 [07/Feb/2018:16:24:19 +0800] "GET / HTTP/1.1" 403 5039
综合实战
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { stdin{}}filter{grok{ match => {"message" => "%{TIMESTAMP_ISO8601:localtime}\|\~\|%{IP:clientip} \|\~\|%{GREEDYDATA:http_user_agent}\|\~\|%{GREEDYDATA:url} \|\~\|%{GREEDYDATA:mediaid}\|\~\|%{GREEDYDATA:osid}"} remove_field => [ "message" ] }date { match => ["localtime", "yyyy-MM-dd'T'HH:mm:ssZZ"] target => "@timestamp" }mutate { remove_field => ["localtime"] }geoip { source => "clientip" #database => "/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb" database => "/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb" fields => ["city_name", "region_name", "country_name", "ip", "latitude", "longitude", "timezone"] }}output { stdout { codec => "rubydebug" }}示例:2018-02-09T10:57:42+08:00|~|123.87.240.97|~|Mozilla/5.0(iPhone;CPU iPhone OS 11_2_2 like Mac OS X)AppleWebKit/604.4.7 Version/11.0 Mobile/15C202 Safari/604.1|~|http://m.sina.cn/cm/ads_ck_wap.html|~|12434785489009|~|DF45566587855P
配置
文件
内容
字段
插件
综合
输出
位置
参数
地理
地理位置
实战
属性
数据
方式
时间
案例
示例
基础
数据库的安全要保护哪些东西
数据库安全各自的含义是什么
生产安全数据库录入
数据库的安全性及管理
数据库安全策略包含哪些
海淀数据库安全审计系统
建立农村房屋安全信息数据库
易用的数据库客户端支持安全管理
连接数据库失败ssl安全错误
数据库的锁怎样保障安全
南昌云端网络技术怎么样
网络安全面对的威胁主要有哪些
u2接口 服务器
php删除提交重复的数据库
数据库cdp备份
共建共享网络安全文明
济南市服务器公司
打印机服务器管理下载
福建数据库培训多少钱
评价一个mysql数据库案例
铁通服务器ip
北京京科网络技术有限公司
月光服务器
网络安全科技馆图片大全
sql两数据库关联查询
数据库主键是不是唯一的
官网相亲软件开发
derby数据库权限
服务器优化方案
工业软件开发龙头
数据库加密数据怎么查
大兴区信息网络技术推广怎么样
软件开发既有的特性又有的特性
三大数据库安全性
群晖共享盘备份到本地服务器
互联网科技农牧行业
上海喜氏互联网科技面试
公安网络技术基本能力总结
产品网络安全及信息安全
宜家购物软件开发