分布式文件访问方案

中心机房

1. cpeh存储源站文件，利于横向扩展。
2. 7层负载均衡将分站访问分布在不同的nginx缓存服务器，分担访问。
3. nginx使用proxy_cache将ceph的访问文件缓存到服务器的各块ssd硬盘，缓解ceph压力。

分站机房

1. app根据用户来源ip或者访问内容，将请求分布在不同分站
2. 分站的负载均衡使用4层，实现高可用。
3. 分站仅缓存数据，未缓存数据向中心机房请求

nginx cache配置

# 通用配置项目
user nginx nginx;
worker_processes 11;
worker_cpu_affinity auto;
pid        logs/nginx.pid;
events {
   worker_connections 10240;
}

# Cache Path配置
sendfile on;
keepalive_timeout 65;
proxy_cache_path /data1/nginx_cache levels=1:2 keys_zone=my-cache1:10m max_size=230G inactive=1d;
proxy_cache_path /data2/nginx_cache levels=1:2 keys_zone=my-cache2:10m max_size=230G inactive=1d;
proxy_cache_path /data3/nginx_cache levels=1:2 keys_zone=my-cache3:10m max_size=230G inactive=1d;
proxy_cache_path /data4/nginx_cache levels=1:2 keys_zone=my-cache4:10m max_size=230G inactive=1d;
proxy_cache_path /data5/nginx_cache levels=1:2 keys_zone=my-cache5:10m max_size=230G inactive=1d;
proxy_cache_path /data6/nginx_cache levels=1:2 keys_zone=my-cache6:10m max_size=230G inactive=1d;
proxy_cache_path /data7/nginx_cache levels=1:2 keys_zone=my-cache7:10m max_size=230G inactive=1d;
proxy_cache_path /data8/nginx_cache levels=1:2 keys_zone=my-cache8:10m max_size=230G inactive=1d;
proxy_cache_path /data9/nginx_cache levels=1:2 keys_zone=my-cache9:10m max_size=230G inactive=1d;
proxy_cache_path /data10/nginx_cache levels=1:2 keys_zone=my-cache10:10m max_size=230G inactive=1d;
#proxy_cache_path /data11/nginx_cache levels=1:2 keys_zone=my-cache11:10m max_size=230G inactive=1d use_temp_path=off;

# Cache比率及Upstream配置
split_clients $uri $my_cache {
   10% "my-cache1";
   10% "my-cache2";
   10% "my-cache3";
   10% "my-cache4";
   10% "my-cache5";
   10% "my-cache6";
   10% "my-cache7";
   10% "my-cache8";
   10% "my-cache9";
   10% "my-cache10";
}
 
proxy_temp_path /data11/proxy_temp 1 2;
upstream dlied_back_rs {
    server xx.xx.xx.xx:8080 weight=3 max_fails=2 fail_timeout=30s;
    server xx.xx.xx.xx:8080 weight=1 max_fails=2 fail_timeout=30s ;
}

# Server的配置
server {
	  listen 8080;
	  server_name _;
	  location / {
	        expires 30d;
	        access_log logs/upstream.log cdn_src;
	        proxy_pass http://dlied_back_rs;
	        proxy_cache $my_cache;
	        proxy_cache_key $uri;
	        proxy_cache_valid 200 302 30d;
	        proxy_cache_valid 404 60m;
	        proxy_cache_use_stale error timeout invalid_header updating;
	        proxy_redirect off;
	        proxy_cache_lock on;
	        proxy_cache_revalidate on;
		proxy_cache_min_uses 3;
		add_header X-Cache-Status $upstream_cache_status;
	        proxy_cache_purge  PURGE from xx.xx.xx.xx;
    }

}

cache的扩展使用

数据按hot warm cold分级，以时间维度 <=1d 为hot数据，<=2d and <=7d 为warm数据，>7d 为cold数据
hot 放入ssd缓存，warm 放入sata缓存，cold 不放入缓存，使用访问url中的日期进行匹配

haproxy lua-load

https://neland.github.io/2017/07/27/lua-for-haproxy/
https://github.com/superstes/haproxy-geoip
https://www.qqzeng.com/article/qqzeng-ip.html