服务器在国外的网站,广告设计跟平面设计,网络外包服务公司,keywordspy网站做分析1.LogStash性能不稳定#xff08;某天关闭后#xff0c;再次启动就非常慢#xff09;#xff0c;所以后面我们用Filebeat。2.先禁用
# geoip {
# source clientip
# }3.在生产中要是用nignx服务或tomcat服务我们用EFK架构就可以排查技巧观察点
LogS…1.LogStash性能不稳定某天关闭后再次启动就非常慢所以后面我们用Filebeat。2.先禁用
# geoip {
# source clientip
# }3.在生产中要是用nignx服务或tomcat服务我们用EFK架构就可以排查技巧观察点
LogStash报错分析 java日志查找 LogStash
数据映射之带宽统计案例
见昨日内容多实例
#补充命令
ncnetcat命令是一个功能强大的网络工具用于在网络连接上进行读写数据。它可以在 TCP 或 UDP 协议下工作被广泛用于网络调试、端口扫描、数据传输等多种任务1.启动第一个实例
[11:29:55 rootelk2:/etc/logstash/conf.d]#logstash -f 01-stdin-to-stdout.conf2.启动第二个实例
[10:36:25 rootelk2:/etc/logstash/conf.d]#logstash -f 02-file-to-stdout.conf --path.data /tmp/logstash-data#注意实验事项
1.如果不指定数据路径则Logstash默认的数据路径在安装目录的data目录下。比如: /usr/share/logstash/data/
2.如果同一个节点启动了多个Logstash实例不指定数据路径则会报错Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the path.data setting.------------------------------实验开始位置----------------------------------------- 1.启动Logstash实例1
[11:31:36 rootelk2:/etc/logstash/conf.d]#vim 09-multiple-tcp-to-es.conf
input { # 监听本地的tcp端口用于接受数据tcp {port 7777}} filter {mutate {remove_field [version]}}output { stdout { codec rubydebug } elasticsearch {hosts [10.0.0.91:9200,10.0.0.92:9200,10.0.0.93:9200]index linux-elfk-multiple-tcp-%{yyyy.MM.dd}}}[11:35:07 rootelk2:/etc/logstash/conf.d]#logstash -rf 09-multiple-tcp-to-es.conf 2.启动Logstash实例2
[11:35:00 rootelk2:/etc/logstash/conf.d]#vim 10-multiple-beats-to-es.conf
input { beats {port 8888}} filter {mutate {remove_field [ version,log,tags,agent,ecs,input ]}}output { stdout { codec rubydebug } elasticsearch {hosts [10.0.0.91:9200,10.0.0.92:9200,10.0.0.93:9200]index linux-elfk-multiple-beats-%{yyyy.MM.dd}}}[11:36:43 rootelk2:/etc/logstash/conf.d]#logstash -rf 10-multiple-beats-to-es.conf --path.data /tmp/logstash-data3.启动filebeat
[11:28:53 rootelk3:/etc/filebeat]#vim 10-tcp-to-logstash.yaml
filebeat.inputs:
- type: tcphost: 0.0.0.0:9000output.logstash:hosts: [10.0.0.92:8888][11:50:12 rootelk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[11:50:15 rootelk3:/etc/filebeat]#filebeat -e -c 10-tcp-to-logstash.yaml4.发送测试数据到实例1
[11:28:44 rootelk1:~]#echo www.baidu.com | nc 10.0.0.92 77775.发送测试数据到filebeat
[11:38:17 rootelk1:~]#echo www.jingdong.com | nc 10.0.0.93 9000 多分支
1.启动Logstash实例
[12:00:26 rootelk2:/etc/logstash/conf.d]#vim 11-if-logstash.conf
input { tcp {type tcpport 7777}beats {port 8888type beats}} filter {if [type] tcp {mutate {remove_field [version]}} else {mutate {remove_field [ version,log,tags,agent,ecs,input ]}}}output { stdout { codec rubydebug } if [type] tcp {elasticsearch {hosts [10.0.0.91:9200,10.0.0.92:9200,10.0.0.93:9200]index linux-elfk-if-tcp-%{yyyy.MM.dd}}}else {elasticsearch {hosts [10.0.0.91:9200,10.0.0.92:9200,10.0.0.93:9200]index linux-elfk-if-beats-%{yyyy.MM.dd}}}}[12:04:48 rootelk2:/etc/logstash/conf.d]#logstash -rf 11-if-logstash.conf 2.启动filebeat实例
[11:50:12 rootelk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[11:50:15 rootelk3:/etc/filebeat]#filebeat -e -c 10-tcp-to-logstash.yaml3.发送测试数据到Filebeat
[11:28:44 rootelk1:~]#echo 11 | nc 10.0.0.92 7777
[11:38:17 rootelk1:~]#echo 22 | nc 10.0.0.93 9000echo 好好学习 | nc 10.0.0.92 7777
echo 天天k8s | nc 10.0.0.93 9000pipeline
1.修改Logstash的配置文件[14:59:26 rootelk2:/etc/logstash/conf.d]#vim 12-pipeline-tcp-to-es.conf
input { tcp {port 7777}} filter {mutate {remove_field [version]}}output { stdout { codec rubydebug } elasticsearch {hosts [10.0.0.91:9200,10.0.0.92:9200,10.0.0.93:9200]index linux-elfk-pipeline-tcp-%{yyyy.MM.dd}}}[14:59:48 rootelk2:/etc/logstash/conf.d]#vim 13-pipeline-beats-to-es.conf
input { beats {port 8888}} filter {mutate {remove_field [ version,log,tags,agent,ecs,input ]}}output { stdout { codec rubydebug } elasticsearch {hosts [10.0.0.91:9200,10.0.0.92:9200,10.0.0.93:9200]index linux-elfk-pipeline-beats-%{yyyy.MM.dd}}}2.修改pipeline配置文件
[14:44:49 rootelk2:/etc/logstash]#vim pipelines.yml
- pipeline.id: tcppath.config: /etc/logstash/conf.d/12-pipeline-tcp-to-es.conf- pipeline.id: beatspath.config: /etc/logstash/conf.d/13-pipeline-beats-to-es.conf3.创建pipeline的软连接
[15:01:26 rootelk2:/etc/logstash]#mkdir /usr/share/logstash/config
[15:01:33 rootelk2:/etc/logstash]#ln -svf /etc/logstash/pipelines.yml /usr/share/logstash/config/4.启动Logstash
[15:01:35 rootelk2:/etc/logstash]#logstash -r5.启动filebeat
[15:03:06 rootelk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[15:04:04 rootelk3:/etc/filebeat]#filebeat -e -c 10-tcp-to-logstash.yaml6.发送测试数据
[12:14:09 rootelk1:~]#echo 好好学习 | nc 10.0.0.92 7777
[14:59:20 rootelk1:~]#echo 天天k8s | nc 10.0.0.93 9000 自定义正则
1.准备自定义的patterns
[16:09:58 rootelk2:/etc/logstash/conf.d]#mkdir patterns[16:10:01 rootelk2:/etc/logstash/conf.d]#vim patterns/linux
YEAR \d{4}
SCHOOL [a-z]{9}
CLASS [a-z0-9]{7}2.编写配置文件
[rootelk92 conf.d]# cat 14-tcp-grok_custom_patterns-es.conf
input { tcp {port 7777}} filter {grok {# 指定自定义patterns目录会自动加载该目录下的所有文件内容相关的patterns模式patterns_dir [/etc/logstash/conf.d/patterns]match { message %{YEAR:year}北京欢迎您: https://www.%{SCHOOL:School}.com 班级: %{CLASS:cLaSs} }}}output { stdout { codec rubydebug } # elasticsearch {
# hosts [10.0.0.91:9200,10.0.0.92:9200,10.0.0.93:9200]
# index linux-elfk-grok-custom-%{yyyy.MM.dd}
# }}[16:12:10 rootelk2:/etc/logstash/conf.d]#logstash -rf 14-tcp-grok_custom_patterns-es.conf 3.发送测试数据
[rootelk93 ~]# echo 北京欢迎您: https://www.beijing.com 班级: linux | nc 10.0.0.92 77774.接收的数据效果如下
{port 48110,timestamp 2024-10-30T08:13:44.725Z,host 10.0.0.93,version 1,tags [[0] _grokparsefailure],message 北京欢迎您: https://www.beijing.com 班级: linux
}JVM优化
1.查看Logstash实例启动时占用的heap堆内存大小
[16:15:05 rootelk2:~]#ps -ef | grep logstash
root 2026 1053 99 16:15 pts/0 00:00:36 /usr/share/elasticsearch/jdk/bin/java -Xms1g -Xmx1g2.修改Logstash的堆内存大小
[16:15:02 rootelk2:~]#vim /etc/logstash/jvm.options
## JVM configuration# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space#-Xms1g
#-Xmx1g-Xms256m
-Xmx256m3.启动Logstash实例测试
[16:14:53 rootelk2:/etc/logstash/conf.d]#logstash -rf 14-tcp-grok_custom_patter4.查看JVM占用内存情况
[16:16:57 rootelk2:~]#ps -ef | grep logstash
root 2123 1053 99 16:17 pts/0 00:00:05 /usr/share/elasticsearch/jdk/bin/java -Xms128m -Xmx128m#注意出现java.lang.OutOfMemoryError: Java heap space
表示内存不足仅需要添加内存即可工作中如何不设置Logstash的堆内存大小默认为1GB。自定义Nginx的json日志格式
1.清空nginx访问日志
[16:38:12 rootelk3:~]# /var/log/nginx/access.log2.修改nginx的配置文件
[16:38:13 rootelk3:~]#vim /etc/nginx/nginx.conf ...# 注释原有的配置行# access_log /var/log/nginx/access.log;log_format nginx_json {timestamp:$time_iso8601,host:$server_addr,clientip:$remote_addr,SendBytes:$body_bytes_sent,responsetime:$request_time,upstreamtime:$upstream_response_time,upstreamhost:$upstream_addr,http_host:$host,uri:$uri,domain:$host,xff:$http_x_forwarded_for,referer:$http_referer,tcp_xff:$proxy_protocol_addr,http_user_agent:$http_user_agent,status:$status};access_log /var/log/nginx/access.log nginx_json;
...[16:39:49 rootelk3:~]#nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful[16:39:56 rootelk3:~]#systemctl reload nginx3.访问测试
[15:52:19 rootelk1:~]#curl 10.0.0.93
!DOCTYPE html
html
head
titleWelcome to nginx!/title
stylebody {width: 35em;margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif;}
/style
/head
body
h1Welcome to nginx!/h1
pIf you see this page, the nginx web server is successfully installed and
working. Further configuration is required./ppFor online documentation and support please refer to
a hrefhttp://nginx.org/nginx.org/a.br/
Commercial support is available at
a hrefhttp://nginx.com/nginx.com/a./ppemThank you for using nginx./em/p
/body
/html4.查看日志格式
[16:40:12 rootelk3:~]#tail -f /var/log/nginx/access.log
{timestamp:2024-10-30T16:40:2608:00,host:10.0.0.93,clientip:10.0.0.91,SendBytes:612,responsetime:0.000,upstreamtime:-,upstreamhost:-,http_host:10.0.0.93,uri:/index.nginx-debian.html,domain:10.0.0.93,xff:-,referer:-,tcp_xff:-,http_user_agent:curl/7.81.0,status:200}5.使用filebeat采集nginx访问日志
[16:42:49 rootelk3:/etc/filebeat]#vim 11-log-to-es.yamlfilebeat.inputs:
- type: logpaths:- /var/log/nginx/access.log# 解析message字段的JSON格式并将键值对放在顶级字段中json.keys_under_root: true#output.console:
# pretty: trueoutput.elasticsearch:hosts: [http://10.0.0.91:9200,http://10.0.0.92:9200,http://10.0.0.93:9200] index: linux-nginx-json-%{yyyy.MM.dd} setup.ilm.enabled: false
setup.template.name: linux-nginx
setup.template.pattern: linux-nginx*
setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 3index.number_of_replicas: 0[16:43:54 rootelk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[16:44:18 rootelk3:/etc/filebeat]#filebeat -e -c 11-log-to-es.yaml6.kibana查看数据
[16:40:26 rootelk1:~]#curl 10.0.0.93/beijing.html 程序名词
- SDK:英文全称为Software development kit表示软件开发工具包指的是程序员所需的开发环境工具包比如JavaGoCPythonRubyPHP等程序员依赖的开发环境。- JDK:英文全称为: Java Development Kit 表示Java开发的工具包集合针对Java开发员的软件开发工具包。JDK就是典型的Java程序员所使用的SDK。用于编写*.java文件对应的编译程序是javac工具。- JRE:英文全称为 Java Runtime Environment表示Java的运行环境相比JDK而言更轻量级软件包会更小只能保证程序能够运行缺少开发的工具包。- JVM:英文全称为 Java Virtual Machine表示Java的虚拟机一般用于执行Java程序编译后的字节码文件*.class对应的运行程序是java工具。#JVM虚拟机又细分为多个存储区域
- 对外内存
- 方法区
- ...
- 堆内存- 伊甸区表示对象产生的内存。- 幸存区经过GC时在伊甸区没有结束生命周期的对象会经过幸存区过滤到年老区。- 年老区长期存在的对象。项目篇: ElasticStack的RBAC实战
2.配置ES集群加密
2.1 生成证书文件
[17:01:21 rootelk1:~]#/usr/share/elasticsearch/bin/elasticsearch-certutil cert -out /etc/elasticsearch/elastic-certificates.p12 -pass
........
Note: Generating certificates without providing a CA certificate is deprecated.A CA certificate will become mandatory in the next major release.Certificates written to /etc/elasticsearch/elastic-certificates.p12This file should be properly secured as it contains the private key for
your instance.This file is a self contained file and can be copied and used as is
For each Elastic product that you wish to configure, you should copy
this .p12 file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.[17:33:28 rootelk1:~]#ll /etc/elasticsearch/elastic-certificates.p12
-rw------- 1 root elasticsearch 3596 Oct 30 17:33 /etc/elasticsearch/elastic-certificates.p122.2 同步证书文件到其他节点
[17:33:52 rootelk1:~]#scp /etc/elasticsearch/elastic-certificates.p12 10.0.0.92:/etc/elasticsearch/
[17:34:10 rootelk1:~]#scp /etc/elasticsearch/elastic-certificates.p12 10.0.0.93:/etc/elasticsearch/2.3 修改ES集群的配置文件
[17:34:59 rootelk1:~]#vim /etc/elasticsearch/elasticsearch.yml
........
# 在最后一行添加以下内容
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p122.4 同步ES配置文件到其他节点
[17:35:25 rootelk1:~]#scp /etc/elasticsearch/elasticsearch.yml 10.0.0.92:/etc/elasticsearch/[17:35:44 rootelk1:~]#scp /etc/elasticsearch/elasticsearch.yml 10.0.0.93:/etc/elasticsearch/2.5 ES集群所有节点修改证书文件的权限信息
chown elasticsearch:elasticsearch /etc/elasticsearch/elastic-certificates.p122.6 ES集群所有节点重启ES服务
systemctl restart elasticsearch.service 2.7 生成随机密码【请保留密码下一个步骤要用】
[17:37:53 rootelk1:~]#/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]yChanged password for user apm_system
PASSWORD apm_system 4Am23FzpVOxMeKK4NVYdChanged password for user kibana_system
PASSWORD kibana_system CJlMQpD1MxNNK6m4FVI9Changed password for user kibana
PASSWORD kibana CJlMQpD1MxNNK6m4FVI9Changed password for user logstash_system
PASSWORD logstash_system nvSvRzHYsQbhQJu2pi3JChanged password for user beats_system
PASSWORD beats_system tGiTVdjelFHGnzo8e0ZfChanged password for user remote_monitoring_user
PASSWORD remote_monitoring_user tSW5yxDEo1pnn3lLLJghChanged password for user elastic
PASSWORD elastic OSUzxjrIEq9HRcbj4bKB2.8 验证集群是否正常【注意elastic是ES集群的管理员用户密码根据自己的系统生成使用即可】
[17:43:06 rootelk2:~]#curl -u elastic:OSUzxjrIEq9HRcbj4bKB 10.0.0.91:9200/_cat/nodes
10.0.0.92 36 83 10 0.15 0.24 0.16 cdfhilmrstw * elk2
10.0.0.93 27 82 9 0.04 0.13 0.09 cdfhilmrstw - elk3
10.0.0.91 13 96 5 0.00 0.08 0.13 cdfhilmrstw - elk1#配置kibana访问ES加密集群
1.修改kibana的配置文件
[17:45:03 rootelk1:~]#vim /etc/kibana/kibana.yml
...........
# 根据你自己环境生成的kibana_system的随机密码做相应的修改即可
elasticsearch.username: kibana_system
elasticsearch.password: CJlMQpD1MxNNK6m4FVI92.重启kibana服务
[17:51:28 rootelk1:~]#systemctl restart kibana
[17:51:48 rootelk1:~]#ss -tnl|grep 5601
LISTEN 0 511 0.0.0.0:5601 0.0.0.0:* 3.访问kibana的WebUI
10.0.0.91:5601将elastic的用户名和密码进行修改 修改用户密码 总结
- 数据映射ELFK架构关于带宽统计案例 *
- Logstash的多实例 ***
- logstash的多分支语法 **
- logstash的pipeline技术 *****
- grok自定义匹配模式patterns *
- logstash的JVM配置优化 ***
- 自定义nginx的访问日志格式 **
- ES集群加密配置 *****
- kibana访问ES加密集群并修改elastic管理员密码 *****
- 故障排查技巧 *****- 1.日志界别INFO WARNFATAL- 2.找关键字ERROR- 3.对于java日志查找方式从日志的最下网上看找at上面没有at的行就是报错信息- 4.日志查找方式- 通过安装目录 /var/log/elasticsearch/ES集群名称.log - 启动脚本查看systemctl cat 启动脚本服务的名称 文章转载自： http://www.morning.hwcgg.cn.gov.cn.hwcgg.cn http://www.morning.xwlmr.cn.gov.cn.xwlmr.cn http://www.morning.qczpf.cn.gov.cn.qczpf.cn http://www.morning.nrzbq.cn.gov.cn.nrzbq.cn http://www.morning.gfrtg.com.gov.cn.gfrtg.com http://www.morning.gbfck.cn.gov.cn.gbfck.cn http://www.morning.cbnlg.cn.gov.cn.cbnlg.cn http://www.morning.kpnpd.cn.gov.cn.kpnpd.cn http://www.morning.ybhjs.cn.gov.cn.ybhjs.cn http://www.morning.nyplp.cn.gov.cn.nyplp.cn http://www.morning.ysgnb.cn.gov.cn.ysgnb.cn http://www.morning.myhpj.cn.gov.cn.myhpj.cn http://www.morning.gsdbg.cn.gov.cn.gsdbg.cn http://www.morning.jbqwb.cn.gov.cn.jbqwb.cn http://www.morning.srwny.cn.gov.cn.srwny.cn http://www.morning.ckhpg.cn.gov.cn.ckhpg.cn http://www.morning.tnmmp.cn.gov.cn.tnmmp.cn http://www.morning.fgrcd.cn.gov.cn.fgrcd.cn http://www.morning.rqzyz.cn.gov.cn.rqzyz.cn http://www.morning.gmrxh.cn.gov.cn.gmrxh.cn http://www.morning.tzmjc.cn.gov.cn.tzmjc.cn http://www.morning.rmdsd.cn.gov.cn.rmdsd.cn http://www.morning.gcxfh.cn.gov.cn.gcxfh.cn http://www.morning.qgbfx.cn.gov.cn.qgbfx.cn http://www.morning.nhzps.cn.gov.cn.nhzps.cn http://www.morning.c7510.cn.gov.cn.c7510.cn http://www.morning.dplmq.cn.gov.cn.dplmq.cn http://www.morning.sgbsr.cn.gov.cn.sgbsr.cn http://www.morning.gwjnm.cn.gov.cn.gwjnm.cn http://www.morning.zrqs.cn.gov.cn.zrqs.cn http://www.morning.kxmyj.cn.gov.cn.kxmyj.cn http://www.morning.cczzyy.com.gov.cn.cczzyy.com http://www.morning.rxcqt.cn.gov.cn.rxcqt.cn http://www.morning.txqsm.cn.gov.cn.txqsm.cn http://www.morning.mhnd.cn.gov.cn.mhnd.cn http://www.morning.jzfrl.cn.gov.cn.jzfrl.cn http://www.morning.mlnbd.cn.gov.cn.mlnbd.cn http://www.morning.rdnjc.cn.gov.cn.rdnjc.cn http://www.morning.wfbs.cn.gov.cn.wfbs.cn http://www.morning.pmwhj.cn.gov.cn.pmwhj.cn http://www.morning.nbdtdjk.cn.gov.cn.nbdtdjk.cn http://www.morning.yqkxr.cn.gov.cn.yqkxr.cn http://www.morning.rnxs.cn.gov.cn.rnxs.cn http://www.morning.nxnrt.cn.gov.cn.nxnrt.cn http://www.morning.bqpgq.cn.gov.cn.bqpgq.cn http://www.morning.flpjy.cn.gov.cn.flpjy.cn http://www.morning.krwzy.cn.gov.cn.krwzy.cn http://www.morning.fycjx.cn.gov.cn.fycjx.cn http://www.morning.lmyq.cn.gov.cn.lmyq.cn http://www.morning.lbbyx.cn.gov.cn.lbbyx.cn http://www.morning.ltdrz.cn.gov.cn.ltdrz.cn http://www.morning.thxfn.cn.gov.cn.thxfn.cn http://www.morning.jlqn.cn.gov.cn.jlqn.cn http://www.morning.rrxnz.cn.gov.cn.rrxnz.cn http://www.morning.leeong.com.gov.cn.leeong.com http://www.morning.pccqr.cn.gov.cn.pccqr.cn http://www.morning.ftmzy.cn.gov.cn.ftmzy.cn http://www.morning.lzwfg.cn.gov.cn.lzwfg.cn http://www.morning.wdpbq.cn.gov.cn.wdpbq.cn http://www.morning.cprls.cn.gov.cn.cprls.cn http://www.morning.xnwjt.cn.gov.cn.xnwjt.cn http://www.morning.duckgpt.cn.gov.cn.duckgpt.cn http://www.morning.pmjw.cn.gov.cn.pmjw.cn http://www.morning.yrpd.cn.gov.cn.yrpd.cn http://www.morning.jtsdk.cn.gov.cn.jtsdk.cn http://www.morning.tfrlj.cn.gov.cn.tfrlj.cn http://www.morning.yrbp.cn.gov.cn.yrbp.cn http://www.morning.ndmbz.cn.gov.cn.ndmbz.cn http://www.morning.xxiobql.cn.gov.cn.xxiobql.cn http://www.morning.hytqt.cn.gov.cn.hytqt.cn http://www.morning.sxjmz.cn.gov.cn.sxjmz.cn http://www.morning.mpgfk.cn.gov.cn.mpgfk.cn http://www.morning.qbrs.cn.gov.cn.qbrs.cn http://www.morning.rqkzh.cn.gov.cn.rqkzh.cn http://www.morning.zcfsq.cn.gov.cn.zcfsq.cn http://www.morning.cwkcq.cn.gov.cn.cwkcq.cn http://www.morning.bmnm.cn.gov.cn.bmnm.cn http://www.morning.gfnsh.cn.gov.cn.gfnsh.cn http://www.morning.wkgyz.cn.gov.cn.wkgyz.cn http://www.morning.kjnfs.cn.gov.cn.kjnfs.cn
