20220320作业分布式链路追踪skywalking

1.skywalking组建简介

skywalking-ui: 前端服务，端口号8080。

skywalking-ops:可观测性分析平台，11800为数据写入端口，12800为查询端口。

es7: 9200为elasticsearch的数据读写端口，skywalking支持的存储有elasticsearch,h2,mysql,tidb,influxdb,postgresql等

agent: app服务器部署skywalking agent，用于收集app中的访问请求

2.部署skywalking

下载软件包：
wget https://archive.apache.org/dist/skywalking/8.7.0/apache-skywalking-apm-es7-8.7.0.tar.gz
tar -zxf apache-skywalking-apm-es7-8.7.0.tar.gz
cd apache-skywalking-apm-es7-8.7.0
修改 config/application.yml:
	selector: ${SW_STORAGE:elasticsearch7}
	nameSpace: ${SW_NAMESPACE:"magedu"}
	clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:192.168.1.77:9200,192.168.1.78:9200,192.168.1.80:9200}
	./bin/startup.sh 

3.halo博客实现链路追踪案例

由于halo是java服务，则需要apache-skywalking-java-agent(https://archive.apache.org/dist/skywalking/java-agent/8.8.0/apache-skywalking-java-agent-8.8.0.tgz)
skywalking-agent/config/agent.config  需要修改的地方
	agent.namespace=${SW_AGENT_NAMESPACE:magedu}
	agent.service_name=${SW_AGENT_NAME:halo}
	collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:192.168.1.77:11800}
启动命令:
	java -javaagent:/apps/halo/skywalking-agent/skywalking-agent.jar -jar /apps/halo/halo-1.4.7.jar

4.tomcat运行jenkins服务

tar -zxf apache-tomcat-8.5.73.tar.gz
mv jenkins.war /apps/skywalking/tomcat
skywalking-agent/config/agent.config  需要修改的地方同3
vim apache-tomcat-8.5.73/bin/catalina.sh  
	CATALINA_OPTS="$CATALINA_OPTS -javaagent:/apps/skywalking/tomcat/skywalking-agent/skywalking-agent.jar";export CATALINA_OPTS
./apache-tomcat-8.5.73/bin/startup.sh 

5.微服务链路追踪

zk搭建过程忽略
dubbo-demo-provider:修改dubbo.properties 
	dubbo.registry.address=zookeeper://192.168.1.77:2181?backup=192.168.1.78:2181,192.168.1.80:2181
skywalking-agent/config/agent.config  需要修改的地方同3
启动命令：java -javaagent:/apps/dubbo-demo-provider-2.1.5/skywalking-agent/skywalking-agent.jar -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -server -Xmx2g -Xms2g -Xmn256m -XX:PermSize=128m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -classpath /apps/dubbo-demo-provider-2.1.5/conf:/apps/dubbo-demo-provider-2.1.5/lib/cache-api-0.4.jar:/apps/dubbo-demo-provider-2.1.5/lib/commons-codec-1.4.jar:/apps/dubbo-demo-provider-2.1.5/lib/commons-logging-1.1.1.jar:/apps/dubbo-demo-provider-2.1.5/lib/commons-pool-1.5.5.jar:/apps/dubbo-demo-provider-2.1.5/lib/dubbo-2.1.5.jar:/apps/dubbo-demo-provider-2.1.5/lib/dubbo-demo-2.1.5.jar:/apps/dubbo-demo-provider-2.1.5/lib/dubbo-demo-provider-2.1.5.jar:/apps/dubbo-demo-provider-2.1.5/lib/fastjson-1.1.8.jar:/apps/dubbo-demo-provider-2.1.5/lib/gmbal-api-only-3.0.0-b023.jar:/apps/dubbo-demo-provider-2.1.5/lib/grizzly-core-2.1.4.jar:/apps/dubbo-demo-provider-2.1.5/lib/grizzly-framework-2.1.4.jar:/apps/dubbo-demo-provider-2.1.5/lib/grizzly-portunif-2.1.4.jar:/apps/dubbo-demo-provider-2.1.5/lib/grizzly-rcm-2.1.4.jar:/apps/dubbo-demo-provider-2.1.5/lib/hessian-4.0.7.jar:/apps/dubbo-demo-provider-2.1.5/lib/hibernate-validator-4.2.0.Final.jar:/apps/dubbo-demo-provider-2.1.5/lib/httpclient-4.1.2.jar:/apps/dubbo-demo-provider-2.1.5/lib/httpcore-4.1.2.jar:/apps/dubbo-demo-provider-2.1.5/lib/javassist-3.15.0-GA.jar:/apps/dubbo-demo-provider-2.1.5/lib/jedis-2.0.0.jar:/apps/dubbo-demo-provider-2.1.5/lib/jetty-6.1.26.jar:/apps/dubbo-demo-provider-2.1.5/lib/jetty-util-6.1.26.jar:/apps/dubbo-demo-provider-2.1.5/lib/jline-0.9.94.jar:/apps/dubbo-demo-provider-2.1.5/lib/log4j-1.2.16.jar:/apps/dubbo-demo-provider-2.1.5/lib/management-api-3.0.0-b012.jar:/apps/dubbo-demo-provider-2.1.5/lib/mina-core-1.1.7.jar:/apps/dubbo-demo-provider-2.1.5/lib/netty-3.2.5.Final.jar:/apps/dubbo-demo-provider-2.1.5/lib/servlet-api-2.5-20081211.jar:/apps/dubbo-demo-provider-2.1.5/lib/slf4j-api-1.6.2.jar:/apps/dubbo-demo-provider-2.1.5/lib/spring-2.5.6.SEC03.jar:/apps/dubbo-demo-provider-2.1.5/lib/validation-api-1.0.0.GA.jar:/apps/dubbo-demo-provider-2.1.5/lib/zookeeper-3.3.3.jar: com.alibaba.dubbo.container.Main

dubbo-demo-consumer:修改dubbo.properties 
	dubbo.registry.address=zookeeper://192.168.1.77:2181?backup=192.168.1.78:2181,192.168.1.80:2181
skywalking-agent/config/agent.config  需要修改的地方同3
启动命令：java -javaagent:/apps/skywalking/dubbo-demo-consumer/skywalking-agent/skywalking-agent.jar -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -server -Xmx2g -Xms2g -Xmn256m -XX:PermSize=128m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -classpath /apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/conf:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/cache-api-0.4.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/commons-codec-1.4.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/commons-logging-1.1.1.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/commons-pool-1.5.5.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/dubbo-2.1.5.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/dubbo-demo-2.1.5.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/dubbo-demo-consumer-2.1.5.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/fastjson-1.1.8.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/gmbal-api-only-3.0.0-b023.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/grizzly-core-2.1.4.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/grizzly-framework-2.1.4.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/grizzly-portunif-2.1.4.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/grizzly-rcm-2.1.4.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/hessian-4.0.7.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/hibernate-validator-4.2.0.Final.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/httpclient-4.1.2.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/httpcore-4.1.2.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/javassist-3.15.0-GA.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/jedis-2.0.0.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/jetty-6.1.26.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/jetty-util-6.1.26.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/jline-0.9.94.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/log4j-1.2.16.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/management-api-3.0.0-b012.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/mina-core-1.1.7.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/netty-3.2.5.Final.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/servlet-api-2.5-20081211.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/slf4j-api-1.6.2.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/spring-2.5.6.SEC03.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/validation-api-1.0.0.GA.jar:/apps/skywalking/dubbo-demo-consumer/dubbo-demo-consumer-2.1.5/lib/zookeeper-3.3.3.jar: com.alibaba.dubbo.container.Main

6.python手动注册

安装
yum install -y python3-pip
pip3 install "apache-skywalking" 报错：distutils.errors.CompileError: command 'gcc' failed with exit status 1
	解决：
	pip3 install --upgrade pip
	python3 -m pip install --upgrade setuptools
	pip3 install --no-cache-dir  --force-reinstall -Iv grpcio==1.23.0
注册：
python3
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from skywalking import agent, config
>>> config.init(collector_address='192.168.1.77:11800', service_name='pytohn-test')
>>> agent.start()

7.python自动注册

安装依赖模块：
yum install -y python3-pip
[root@192-168-1-78-elk02 django-test]# cat requirements.txt 
apache-skywalking==0.7.0
asgiref==3.4.1
backports.zoneinfo==0.2.1
Django==3.2.12
grpcio==1.43.0
grpcio-tools==1.43.0
packaging==21.3
protobuf==3.19.3
PyMySQL==1.0.2
pyparsing==3.0.6
six==1.16.0
sqlparse==0.4.2
wrapt==1.13.3

pip3 install -r requirements.txt 
centos 自带sqlite3版本太低，运行Django需要升级，升级参考：https://blog.csdn.net/zhongliang415/article/details/117516050
创建Django项目mysite
django-admin startproject mysite
创建应用
cd mysite
python3 manage.py startapp myapp
初始化数据库：
python3 manage.py makemigrations
python3 manage.py migrate
创建管理员，用于登录admin
python3 manage.py createsuperuser
skywalking环境环境变量
export SW_AGENT_COLLECTOR_BACKEND_SERVICES='192.168.1.77:11800'
export SW_AGENT_NAMESPACE='magedu'
export SW_AGENT_NAME='python-django'
修改配置：
vim mysite/settings.py 
ALLOWED_HOSTS = ["*"]
启动：
sw-python -d run python3 manage.py runserver 192.168.1.78:80

8.skywalking告警系统

指标：
#服务的响应时间
service_resp_time = from(Service.latency).longAvg();
#服务的http请求成功率SLA，比如99%
service_sla = from(Service.*).percent(status == true);
#表示每分钟的吞吐量
service_cpm = from(Service.*).cpm();
#指定最近多少数据范围内的响应时间百分比，即p99,p95,p90,p75,p50在内的数据统计结果
service_percentile = from(Service.latency).percentile(10); // Multiple values including p50, p75, p90, p95, p99
#应用性能指数是0.*
service_apdex = from(Service.latency).apdex(name, status);
#端点的每分钟的吞吐量
endpoint_relation_cpm = from(EndpointRelation.*).filter(detectPoint == DetectPoint.SERVER).cpm();
#端点的响应时间
endpoint_relation_resp_time = from(EndpointRelation.rpcLatency).filter(detectPoint == DetectPoint.SERVER).longAvg();
#端点的http请求成功率SLA，比如99%等
endpoint_relation_sla = from(EndpointRelation.*).filter(detectPoint == DetectPoint.SERVER).percent(status == true);
#端点指定最近多少数据范围内的响应时间百分比，即p99,p95,p90,p75,p50在内的数据统计结果
endpoint_relation_percentile = from(EndpointRelation.rpcLatency).filter(detectPoint == DetectPoint.SERVER).percentile(10); // Multiple values including p50, p75, p90, p95, p99

告警：
rules :
   #规则唯一名称，必须以 `_rule` 结尾。
  endpoint_percent_rule :
     # Metrics 值需要是 long、double 或 int 
    metrics-name : endpoint_percent 
    threshold : 75 
    op : < 
    #评估指标的时间长度
    period : 10 
    #指标匹配条件后多少次触发告警
    count : 3 
    #检测多少次，报警触发后报警保持静音，默认与 period 相同。
    静默期：10
    #指定规则是否可以发送通知或仅作为复合规则的条件
    only-as-condition : false 
    tags :
       level : WARNING 

配置钉钉告警
webhooks : 
    - url : https://oapi.dingtalk.com/robot/send?access_token=dummy_token