AmazingCounters.com

Varnish基础配置、原理和应用

0.Varnish介绍(程序架构,原理)  

Varnish是一款高性能的开源HTTP加速器,具有反向代理,缓存的功能。

缓存类型:代理式缓存(递归方式);旁挂式缓存(迭代)

缓存机制:过期机制(Expires)、条件式缓存(通过最近文件修改时间戳Etag的扩展标签来辨别)。

过期时间:Expires
HTTP/1.0
Expires:过期
HTTP/1.1
Cache-Control: maxage=  (私有缓存,单位秒)
Cache-Control: s-maxage=  (共有缓存)

缓存层级:
私有缓存:用户代理附带的本地缓存机制;
公共缓存:反向代理服务器的缓存功能;
条件式请求:
Last-Modified/If-Modified-Since:基于文件的修改时间戳来
判别:Etag/If-None-Match:基于文件的校验码来判别;
User-Agent <--> private cache <--> public cache <--> public cache 2 <--> Original Server

Web Page Cache:squid --> varnish           varnish官方站点: http://www.varnish-cache.org/

缓存空间耗尽:LRU,最近最少使用;
web后台restful简单架构模型

应用运维三大日程工作:发布-----变更-----故障处理

程序架构:
Manager进程
Cacher进程,包含多种类型的线程:accept, worker, expiry, ...
shared memory log:统计数据:计数器(处理请求,时间,hit数,miss数);日志区域:日志记录;
varnishlog, varnishncsa, varnishstat...
配置接口:VCL       vcl complier --> c complier --> shared object

varnish的程序环境:
/etc/varnish/varnish.params: 配置varnish服务进程的工作特性,例如监听的地址和端口,缓存机制;
/etc/varnish/default.vcl:配置各Child/Cache线程的缓存策略;
主程序:/usr/sbin/varnishd
CLI interface:/usr/bin/varnishadm     如varnishadm -S  /etc/varnish/secret   -T 127.0.0.1:6082     (默认6082为varnish管理端口)
Shared Memory Log交互工具:
/usr/bin/varnishhist         /usr/bin/varnishlog         /usr/bin/varnishncsa        /usr/bin/varnishstat         /usr/bin/varnishtop
测试工具程序:
/usr/bin/varnishtest
VCL配置文件重载程序:/usr/sbin/varnish_reload_vcl
varnish服务
/usr/lib/systemd/system/varnishlog.service
/usr/lib/systemd/system/varnishncsa.service 日志持久的服务;

varnish工作逻辑

 

1.varnish配置测试(缓存、自定义非缓存、purge更新缓存、按需调度、负载均衡、健康状态检查)

vim  /etc/varnish/varnish.params

其中需要把端口改为80 (默认6081),管理地址为本机,管理端口为6082(默认),以varnish身份运行

varnish的缓存存储机制( Storage Types):
· malloc[,size]           内存存储,[,size]用于定义空间大小;重启后所有缓存项失效;一般4G较合适,内存空间有限,且内存碎片会大大影响性能;
· file[,path[,size[,granularity]]]   磁盘文件存储,黑盒;重启后所有缓存项失效;
· persistent,path,size    文件存储,黑盒;重启后所有缓存项有效;实验状态;

缓存处于稳态很难,即缓存热身时间很长,

该文件最后一行,可灵活自定义

vim  /etc/varnish/default.vcl    (cat /etc/varnish/default.vcl |egrep "^[^#]")

#必须以vcl git 开头
vcl 4.0;
#导入directors模块实现负载均衡;并定义acl,来控制purger(更新缓存)的使用 import directors; acl purgers {
"127.0.0.0"/8; }
#健康状态检查定义项
probe healthche { .url
="/index.html"; .timeout = 2s; .window = 6 ; .threshold = 5; }
#backend组即为后台web端 backend server1 { .host
= "172.18.64.7"; .port = "80"; .probe = healthche ; } backend server2 { .host="172.18.64.106"; .port="80"; .probe = healthche ; } }

#以轮询方式调度 sub vcl_init {
new srvs =directors.round_robin(); srvs.add_backend(server1); srvs.add_backend(server2); } sub vcl_recv { # Happens before we check if we have this in cache already. # # Typically you clean up the request here, removing cookies you don't need, # rewriting the request, etc. set req.backend_hint=srvs.backend();
#正则匹配 login或admin隐私信息不允许服务端缓存
if (req.url ~ "(?i)^/(login|admin)") { return(pass); }
#url重写,告诉后端服务器真实的请求者,安全避免重复添加,还可定义在记录日志中
if (req.restarts == 0) { if (req.http.X-Fowarded-For) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } }
#purge的使用:更新一个缓存,而更新一组缓存用ban
if (req.method == "PURGE"){ if (!client.ip ~ purgers) { return(synth(405,"Purging not allowed for "+client.ip)); } return(purge); } } sub vcl_backend_response { # Happens after we have read the response headers from the backend. # # Here you clean the response headers, removing silly Set-Cookie headers # and other mistakes your backend does.
#客户端的图片类信息可以除去cookies标志让服务器能够缓存,并定义缓存有效期为2H
if (bereq.url ~ "(?i)\.(jpg|jpeg|png|gif)$") { unset beresp.http.Set-cookies; set beresp.ttl =7200s; } } sub vcl_deliver { # Happens when we have all the pieces we need, and are about to send the # response to the client. # # You can do accounting or modifying the final object here.
#定义缓存响应头部
if (obj.hits>0) { set resp.http.X-Cache = "HIT via " + server.ip; } else { set resp.http.X-Cache = "MISS from " + server.ip; } }

varnish_reload_vcl 重新加载后测试效果

 

F5刷新后。。。(后台web站点172.18.64.7,而varnish端为172.18.64.107,实际生产varnish的vip为公网地址,而rip为局域网地址)

对与login或admin私有信息来说,不能缓存。

F5刷新后,依然无法缓存,且由于调度轮询则来自另一个web

 对于purge的使用,首先定义acl(清缓再热身存负载大,安全起见必须指定特定主机才有清缓存权)

覆盖后F5刷新,由于缓存还是原来图片信息

在acl定义的主机上用purge 更新单个缓存 curl  -X  PURGE  http://127.0.0.1/world.jpg

此时再刷新网页即可刷新

但purge只能刷新一个资源,且有时会经常出错一般可在vcl命令行中用ban刷新一类资源

ban req.url ~ (?i).(jpg|jpeg)$   

F5再次刷新(以将图片替换掉)

再次刷新依然是hit的

对于一般站点来说,css、html、js、txt(升级会变)和图片(一般不会变)一般分隔开,设定使用多个后端主机:

backend server1 {
.host = "172.18.64.7";
.port = "80";
}

backend server2 {
.host = "172.18.64.106";
.port = "80";
}

#在sub vcl_recv  加                          
if (req.url ~ "(?i)\.(jpg|jpeg|png)$") {
set req.backend_hint = server2;
} else {
set req.backend_hint = server1;
}

这样即可把不同资源按需调度到不同主机。 

关于负载均衡,最上面的代码段里import导入相关模块后,在sub vcl_init 中定义轮询调度,一般轮询就够,因为大多数别CDN挡在外面,剩

下的又有绝大多数别hit中,所以只有很少的请求进入后端主机,此时测试访问login/login.html那个不允许缓存的文件即可实现轮询访问效果。

当然也可基于hash的权重配置,如下基于cookie的session sticky:

sub vcl_init {
new h = directors.hash();
h.add_backend(one, 1); // backend 'one' with weight '1'
h.add_backend(two, 1); // backend 'two' with weight '1'
}

sub vcl_recv {
// pick a backend based on the cookie header of the client
set req.backend_hint = h.backend(req.http.cookie);
}

健康状态检测backend.list

最上面的代码已经实现健康状态的检测,也可实现手动健康检测

backend.set_health   server  sick | health  | auto

但一般会设置为机器检测状态

2.varnish优化与管理、监控记录命令

在线程池内部,其每一个请求由一个线程来处理; 其worker线程的最大数决定了varnish的并发响应能力;
thread_pools:Number of worker thread pools. 最好小于或等于CPU核心数量; 
thread_pool_max:The maximum number of worker threads in each pool. 每线程池的最大线程数;
thread_pool_min:The minimum number of worker threads in each pool. 额外意义为“最大空闲线程数”;
最大并发连接数 = thread_pools * thread_pool_max
thread_pool_timeout:Thread idle threshold. Threads inexcess of thread_pool_min, which have been idle for at least this long,

will be destroyed.
thread_pool_add_delay:Wait at least this long after creating a thread.
thread_pool_destroy_delay:Wait this long after destroying a thread.

设置方式:
vcl.param
如 param.set   thread_pools  4

param.show  thread_pools 可查看

永久有效的方法:
vim  /etc/varnish/varnish.params   最后一行
DEAMON_OPTS="-p PARAM1=VALUE -p PARAM2=VALUE"

varnish的几个命令行工具

varnishlog把每一个请求拆开详细显示

varnishtop (Varnish log entry ranking) 则会把varnishlog把请求method按照速率做一个简单排序 

-1   Instead of a continously updated display, print the statistics once and exit.
-i taglist,可以同时使用多个-i选项,也可以一个选项跟上多个标签;
-I <[taglist:]regex>  正则表达式匹配
-x taglist:排除列表
-X <[taglist:]regex>

varnishstat则会动态显示

varnishstat - Varnish Cache statistics
-1 -f FILED_NAME  只显示特定项一次
-l:可用于-f选项指定的字段名称列表;
MAIN.cache_hit 和MAIN.cache_miss是最常要看的参数
# varnishstat    -1    -f     MAIN.cache_hit     -f      MAIN.cache_miss显示指定参数的当前统计数据;
# varnishstat -l -f MAIN -f MEMPOOL       列出指定配置段的每个参数的意义;

varnishncsa - Display Varnish logs in Apache / NCSA combined log format 即按照ncsa格式显示日志

varnish的访问日志一般在内存中,容量大概只有80-90M,用不了多久就会被覆盖,如果不想别覆盖,可以把varnishncsa当日志服务启动

systemctl   start   varnishncsc.service

此时会在/var/log/varnish/目录下生成varnishncsa文件,此时访问即可被记录,但尽量不要开启这个日志功能!因为作用不是很大,且容易

产生干扰信息,有专门的日志监控服务完成。

3.总结 :varnish: state engine, vcl 

varnish 4.0:
vcl_init
vcl_recv
vcl_hash
vcl_hit
vcl_pass
vcl_miss
vcl_pipe
vcl_waiting
vcl_purge
vcl_deliver
vcl_synth
vcl_fini

vcl_backend_fetch
vcl_backend_response
vcl_backend_error

sub VCL_STATE_ENGINE {
...
}
backend BE_NAME {}
probe PB_NAME {}
acl ACL_NAME {}

实际生产环境实例 

 backend imgsrv1 {
                .host = "192.168.10.11";
                .port = "80";
}
        
backend imgsrv2 {
                .host = "192.168.10.12";
                .port = "80";
}       
        
backend appsrv1 {
                .host = "192.168.10.21";
                .port = "80";
}
        
backend appsrv2 {
                .host = "192.168.10.22";
                .port = "80";
}
        
sub vcl_init {
                new imgsrvs = directors.random();
                imgsrvs.add_backend(imgsrv1,10);
                imgsrvs.add_backend(imgsrv2,20);
                new staticsrvs = directors.round_robin();
                appsrvs.add_backend(appsrv1);
                appsrvs.add_backend(appsrv2);
                
                new appsrvs = directors.hash();
                appsrvs.add_backend(appsrv1,1);
                appsrvs.add_backend(appsrv2,1);         
}
        
sub vcl_recv {
                if (req.url ~ "(?i)\.(css|js)$" {
                        set req.backend_hint = staticsrvs.backend();
                }               
                if (req.url ~ "(?i)\.(jpg|jpeg|png|gif)$" {
                        set req.backend_hint = imgsrvs.backend();
                } else {                
                        set req.backend_hint = appsrvs.backend(req.http.cookie);
                }
}

 

posted @ 2017-11-02 20:12  黑夜繁星  阅读(4708)  评论(0编辑  收藏
AmazingCounters.com