tengine-ingress 源代码分析
代码流程图
tengine-ingress包含：controller，lua脚本，nginx程序三部分，下图为controller的代码流程图：
文档
tengine文档很少，但是他兼容ingress-nginx，
所以ingress-nginx的文档，就是它的文档：
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/
所有ingress-nginx的可变配置：
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
TCP/UDP监听：
https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
运行时
通过运行起来的container，能看见程序运行情况与进程父子关系
tengine-ingress-controller的配置参数
Usage of :
--add_dir_header If true, adds the file directory to the header
--alsologtostderr log to standard error as well as files
--annotations-prefix string Prefix of the Ingress annotations specific to the NGINX controller. (default "nginx.ingress.kubernetes.io")
--apiserver-host string Address of the Kubernetes API server.
Takes the form "protocol://address:port". If not specified, it is assumed the
program runs inside a Kubernetes cluster and local discovery is attempted.
--certificate-authority string Path to a cert file for the certificate authority. This certificate is used
only when the flag --apiserver-host is specified.
--configmap string Name of the ConfigMap containing custom global configurations for the controller.
--controller-class string Ingress Class Controller value this Ingress satisfies.
The class of an Ingress object is set using the field IngressClassName in Kubernetes clusters version v1.19.0 or higher. The .spec.controller value of the IngressClass
referenced in an Ingress Object should be the same value specified here to make this object be watched. (default "k8s.io/tengine-ingress")
--default-backend-service string Service used to serve HTTP requests not matching any known server name (catch-all).
Takes the form "namespace/name". The controller configures NGINX to forward
requests to the first port of this Service.
--default-server-port int Port to use for exposing the default server (catch-all). (default 8181)
--default-ssl-certificate string Secret containing a SSL certificate to be used by the default HTTPS server (catch-all).
Takes the form "namespace/name".
--disable-catch-all Disable support for catch-all Ingresses
--election-id string Election id to use for Ingress status updates. (default "ingress-controller-leader")
--enable-metrics Enables the collection of NGINX metrics (default true)
--enable-ssl-chain-completion Autocomplete SSL certificate chains with missing intermediate CA certificates.
Certificates uploaded to Kubernetes must have the "Authority Information Access" X.509 v3
extension for this to succeed.
--enable-ssl-passthrough Enable SSL Passthrough.
--health-check-path string URL path of the health check endpoint.
Configured inside the NGINX status server. All requests received on the port
defined by the healthz-port parameter are forwarded internally to this path. (default "/healthz")
--health-check-timeout int Time limit, in seconds, for a probe to health-check-path to succeed. (default 10)
--healthz-port int Port to use for the healthz endpoint. (default 10254)
--http-port int Port to use for servicing HTTP traffic. (default 80)
--https-port int Port to use for servicing HTTPS traffic. (default 443)
--ingress-class string [IN DEPRECATION] Name of the ingress class this controller satisfies.
The class of an Ingress object is set using the annotation "kubernetes.io/ingress.class" (deprecated).
The parameter --controller-class has precedence over this. (default "tengine")
--ingress-class-by-name Define if Ingress Controller should watch for Ingress Class by Name together with Controller Class.
--kubeconfig string Path to a kubeconfig file containing authorization and API server information.
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--maxmind-license-key string Maxmind license key to download GeoLite2 Databases.
https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases
--metrics-per-host Export metrics per-host (default true)
--profiler-port int Port to use for expose the ingress controller Go profiler when it is enabled. (default 10245)
--profiling Enable profiling via web interface host:port/debug/pprof/ (default true)
--publish-service string Service fronting the Ingress controller.
Takes the form "namespace/name". When used together with update-status, the
controller mirrors the address of this service's endpoints to the load-balancer
status of all Ingress objects it satisfies.
--publish-status-address string Customized address to set as the load-balancer status of Ingress objects this controller satisfies.
Requires the update-status parameter.
--quic-port int Port to use for servicing HTTP/3+QUIC traffic. (default 443)
--report-node-internal-ip-address Set the load-balancer status of Ingress objects to internal Node addresses instead of external.
Requires the update-status parameter.
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--ssl-passthrough-proxy-port int Port to use internally for SSL Passthrough. (default 442)
--status-port int Port to use for the lua HTTP endpoint configuration. (default 10246)
--status-update-interval int Time interval in seconds in which the status should check if an update is required. Default is 60 seconds (default 60)
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--stream-port int Port to use for the lua TCP/UDP endpoint configuration. (default 10247)
--sync-period duration Period at which the controller forces the repopulation of its local object stores. Disabled by default.
--sync-rate-limit float32 Define the sync frequency upper limit (default 0.3)
--tcp-services-configmap string Name of the ConfigMap containing the definition of the TCP services to expose.
The key in the map indicates the external port to be used. The value is a
reference to a Service in the form "namespace/name:port", where "port" can
either be a port number or name. TCP ports 80 and 443 are reserved by the
controller for servicing HTTP traffic.
--udp-services-configmap string Name of the ConfigMap containing the definition of the UDP services to expose.
The key in the map indicates the external port to be used. The value is a
reference to a Service in the form "namespace/name:port", where "port" can
either be a port name or number.
--update-status Update the load-balancer status of Ingress objects this controller satisfies.
Requires setting the publish-service parameter to a valid Service reference. (default true)
--update-status-on-shutdown Update the load-balancer status of Ingress objects when the controller shuts down.
Requires the update-status parameter. (default true)
-v, --v Level number for the log level verbosity
--validating-webhook string The address to start an admission controller on to validate incoming ingresses.
Takes the form "<host>:port". If not provided, no admission controller is started.
--validating-webhook-certificate string The path of the validating webhook certificate PEM.
--validating-webhook-key string The path of the validating webhook key PEM.
--version Show release information about the NGINX Ingress controller and exit.
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
--watch-ingress-without-class Define if Ingress Controller should also watch for Ingresses without an IngressClass or the annotation specified.
--watch-namespace string Namespace the controller watches for updates to Kubernetes objects.
This includes Ingresses, Services and all configuration resources. All
namespaces are watched if this parameter is left empty.
--watch-namespace-selector string Selector selects namespaces the controller watches for updates to Kubernetes objects.
ingress代码
查看 /tengine-ingress-Tengine-Ingress-v1.1.0/build/build.sh 确定入口函数：
tengine-ingress-Tengine-Ingress-v1.1.0/cmd/nginx/main.go::func main()
tengine的编译选项：
[root@tengine-ingress-controller-65c9f55cc8-hkxrb nginx]# /usr/local/tengine/sbin/tengine -V Tengine version: Tengine/3.1.0 nginx version: nginx/1.24.0 built with OpenSSL 1.1.1h 22 Sep 2020 TLS SNI support enabled configure arguments: --prefix=/usr/local/tengine --sbin-path=sbin/tengine --conf-path=/etc/nginx/nginx.conf --modules-path=/etc/nginx/modules --http-log-path=/home/admin/tengine-ingress/logs/access.log --error-log-path=/home/admin/tengine-ingress/logs/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module=dynamic --with-http_gzip_static_module --with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-http_secure_link_module --with-http_gunzip_module --with-file-aio --with-pcre=/tmp/build/pcre-8.45 --with-pcre-opt=-fPIC --with-xquic-inc=/tmp/build/xquic-1.6.0/include --with-xquic-lib=/tmp/build/xquic-1.6.0/build --without-select_module --without-poll_module --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_uwsgi_module --without-http_scgi_module --with-openssl=/tmp/build/Tongsuo-8.3.2 --with-zlib=/tmp/build/zlib-1.2.8 --with-zlib-opt='-O3 -fPIC' --with-jemalloc=/tmp/build/jemalloc-4.0.4 --add-module=modules/ngx_debug_pool --add-module=modules/mod_common --add-module=modules/mod_strategy --add-module=modules/ngx_backtrace_module --add-module=modules/ngx_http_xquic_module --with-xudp-inc=/tmp/build/libxudp-v1.0.2/objs --with-xudp-lib=/tmp/build/libxudp-v1.0.2/objs --with-xquic_xdp-inc=/tmp/build/tengine-3.1.0/modules/mod_xudp/xquic-xdp --add-module=modules/mod_xudp --add-module=modules/ngx_http_sysguard_module --add-module=modules/ngx_http_footer_filter_module --add-module=modules/ngx_http_trim_filter_module --add-module=modules/ngx_http_reqstat_module --add-module=modules/ngx_http_proxy_connect_module --add-module=modules/ngx_http_upstream_check_module --add-module=modules/ngx_http_upstream_dyups_module --add-module=modules/ngx_http_upstream_dynamic_module --add-module=modules/ngx_http_upstream_session_sticky_module --add-module=modules/ngx_ingress_module --user=admin --group=admin --with-cc-opt='-g -Og -fstack-protector-strong -DT_HTTP_X_BODY_STREAM -Wformat -Werror=format-security -Wno-deprecated-declarations -fno-strict-aliasing -D_FORTIFY_SOURCE=2 --param=ssp-buffer-size=4 -DTCP_FASTOPEN=23 -fPIC -Wp,-U_FORTIFY_SOURCE -I modules/ngx_http_lua_module/src -m64 -mtune=native' --with-ld-opt='-fPIC -Wl,-z,relro -Wl,-z,now,-rpath=/usr/local/lib/ -L/usr/local/lib/ -lm' --add-module=modules/ngx_http_lua_module --add-module=/tmp/build/stream-lua-nginx-module-0.0.13 --add-module=/tmp/build/lua-upstream-nginx-module-0.07 --add-module=/tmp/build/ngx_devel_kit-0.3.1rc1 --add-module=/tmp/build/set-misc-nginx-module-0.32 --add-module=/tmp/build/headers-more-nginx-module-0.34 --add-module=/tmp/build/ngx_http_substitutions_filter_module-bc58cb11844bc42735bbaef7085ea86ace46d05b --add-dynamic-module=/tmp/build/nginx-http-auth-digest-1.0.0 --add-dynamic-module=/tmp/build/nginx-influxdb-module-5b09391cb7b9a889687c0aa67964c06a2d933e8b --add-dynamic-module=/tmp/build/nginx-opentracing-0.9.0/opentracing --add-dynamic-module=/tmp/build/modsecurity-nginx-v1.0.1 --add-dynamic-module=/tmp/build/ngx_http_geoip2_module-3.4 --add-dynamic-module=/tmp/build/ngx_brotli-1.0.0rc
代码架构
ingress是使用的client-go的库进行的开发，读代码之前，要先属性client-go，方能事半功倍：
./bin/pkg/mod/github.com/alibaba/tengine-ingress@v0.0.0-20231026080314-096c92c908b1/vendor/k8s.io/client-go
tengine的lua端口
上文流程图中写了端口10246与10247是controller程序与tengine程序下配置的通道之一。如下图可见所有的监听端口
使用命令：kubectl exec -it tengine-ingress-controller-65c9f55cc8-hkxrb -n tengine-ingress -- bash
配置文件分析
10246
10247
完整配置文件
# Configuration checksum: 18225032199610257382
# setup custom paths that do not require root access
pid /tmp/nginx.pid;
user root;
load_module /etc/nginx/modules/ngx_http_geoip_module.so;
daemon off;
worker_processes 2;
worker_rlimit_nofile 523264;
worker_shutdown_timeout 240s ;
events {
multi_accept on;
worker_connections 16384;
accept_mutex on;
use epoll;
}
include /etc/nginx/apps/proc.conf;
http {
lua_package_path "/etc/nginx/lua/?.lua;;";
lua_shared_dict csp_data 10m;
lua_shared_dict deny_lock 1m;
lua_shared_dict deny_data 1000m;
lua_use_default_type off;
lua_shared_dict balancer_ewma 10M;
lua_shared_dict balancer_ewma_last_touched_at 10M;
lua_shared_dict balancer_ewma_locks 1M;
lua_shared_dict certificate_data 20M;
lua_shared_dict certificate_servers 5M;
lua_shared_dict configuration_data 20M;
init_by_lua_block {
collectgarbage("collect")
-- init modules
local ok, res
ok, res = pcall(require, "lua_ingress")
if not ok then
error("require failed: " .. tostring(res))
else
lua_ingress = res
lua_ingress.set_config( {
use_forwarded_headers = false,
use_proxy_protocol = false,
is_ssl_passthrough_enabled = false,
http_redirect_code = 308,
listen_ports = {
ssl_proxy = "442", https = "443"
}
,
hsts = true,
hsts_max_age = 15724800,
hsts_include_subdomains = true,
hsts_preload = false,
}
)
end
ok, res = pcall(require, "configuration")
if not ok then
error("require failed: " .. tostring(res))
else
configuration = res
end
ok, res = pcall(require, "balancer")
if not ok then
error("require failed: " .. tostring(res))
else
balancer = res
end
ok, res = pcall(require, "monitor")
if not ok then
error("require failed: " .. tostring(res))
else
monitor = res
end
ok, res = pcall(require, "certificate")
if not ok then
error("require failed: " .. tostring(res))
else
certificate = res
end
ok, res = pcall(require, "plugins")
if not ok then
error("require failed: " .. tostring(res))
else
plugins = res
end
-- load all plugins that'll be used here
plugins.init( {
}
)
}
init_worker_by_lua_block {
lua_ingress.init_worker()
balancer.init_worker()
monitor.init_worker()
plugins.run()
}
geo $dontlog {
default 0;
127.0.0.1/32 1;
192.168.0.0/16 1;
121.0.29.226/32 1;
}
geo $nolog {
default 0;
}
geo $no_reqstatus {
default 0;
}
geo $x_connection {
default "";
}
# start hot reload config
ingress_gateway_hash_num shm_service_cfg 500000;
ingress_gateway_pool_size shm_service_cfg 256m;
ingress_gateway_update_interval shm_service_cfg 10s;
ingress_gateway_shm_config shm_service_cfg ShmServiceCfg 256m /etc/nginx/shm_service_cfg.lock;
geoip_country /etc/nginx/geoip/GeoIP.dat;
geoip_city /etc/nginx/geoip/GeoLiteCity.dat;
geoip_org /etc/nginx/geoip/GeoIPASNum.dat;
geoip_proxy_recursive on;
aio threads;
aio_write on;
tcp_nopush on;
tcp_nodelay on;
log_subrequest on;
reset_timedout_connection on;
keepalive_timeout 75s;
keepalive_requests 100;
client_body_temp_path /tmp/client-body;
fastcgi_temp_path /tmp/fastcgi-temp;
proxy_temp_path /tmp/proxy-temp;
client_header_buffer_size 1k;
client_header_timeout 60s;
large_client_header_buffers 4 8k;
client_body_buffer_size 8k;
client_body_timeout 60s;
http2_max_concurrent_streams 128;
types_hash_max_size 2048;
server_names_hash_max_size 1024;
server_names_hash_bucket_size 32;
map_hash_bucket_size 130;
proxy_headers_hash_max_size 512;
proxy_headers_hash_bucket_size 64;
variables_hash_bucket_size 256;
variables_hash_max_size 2048;
underscores_in_headers off;
ignore_invalid_headers on;
limit_req_status 503;
limit_conn_status 503;
include /etc/nginx/mime.types;
default_type text/html;
gzip on;
gzip_comp_level 5;
gzip_http_version 1.1;
gzip_min_length 256;
gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component;
gzip_proxied any;
gzip_vary on;
# Custom headers for response
server_tokens on;
# disable warnings
uninitialized_variable_warn off;
# Additional available variables:
# $namespace
# $ingress_name
# $service_name
# $service_port
log_format upstreaminfo '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id';
map $request_uri $loggable {
default 1;
}
access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
error_log /var/log/nginx/error.log notice;
resolver 10.96.0.10 valid=30s;
# See https://www.nginx.com/blog/websocket-nginx
map $http_upgrade $connection_upgrade {
default upgrade;
# See http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
'' '';
}
# Reverse proxies can detect if a client provides a X-Request-ID header, and pass it on to the backend server.
# If no such header is provided, it can provide a random value.
map $http_x_request_id $req_id {
default $http_x_request_id;
""
$request_id;
}
# Create a variable that contains the literal $ character.
# This works because the geo module will not resolve variables.
geo $literal_dollar {
default "$";
}
server_name_in_redirect off;
port_in_redirect off;
ssl_protocols TLSv1.2;
ssl_early_data off;
# turn on session caching to drastically improve performance
ssl_session_cache builtin:1000 shared:
SSL:10m;
ssl_session_timeout 10m;
# allow configuring ssl session tickets
ssl_session_tickets on;
# slightly reduce the time-to-first-byte
ssl_buffer_size 4k;
# allow configuring custom ssl ciphers
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers on;
ssl_ecdh_curve auto;
# PEM sha: 24abab98ff48f9134bf7ede71a6f651b32494be6
ssl_certificate /etc/ingress-controller/ssl/default-fake-certificate.pem;
ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem;
xquic_ssl_certificate /etc/ingress-controller/ssl/default-fake-certificate.pem;
xquic_ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem;
proxy_ssl_session_reuse on;
upstream upstream_balancer {
### Attention!!!
#
# We no longer create "upstream" section for every backend.
# Backends are handled dynamically using Lua. If you would like to debug
# and see what backends ingress-nginx has in its memory you can
# install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin.
# Once you have the plugin you can use "kubectl ingress-nginx backends" command to
# inspect current backends.
#
###
server 0.0.0.1;
# placeholder
balancer_by_lua_block {
balancer.balance()
}
keepalive 32;
keepalive_timeout 60s;
keepalive_requests 100;
}
# Cache for internal auth checks
proxy_cache_path /tmp/nginx-cache-auth levels=1:2 keys_zone=auth_cache:10m max_size=128m inactive=30m use_temp_path=off;
# Global filters
## start server _
server {
server_name _ ;
listen 80 default_server reuseport backlog=4096 ;
listen [::]:80 default_server reuseport backlog=4096 ;
listen 443 default_server reuseport backlog=4096 ssl http2 ;
listen [::]:443 default_server reuseport backlog=4096 ssl http2 ;
listen 443 default_server reuseport backlog=4096 xquic ;
listen [::]:443 default_server reuseport backlog=4096 xquic ;
ingress_gateway shm_service_cfg;
ingress_gateway_metadata "ssl-protocols" $metadata_ssl_protocols;
# set log host
set $log_host $host;
set $proxy_upstream_name "-";
ssl_certificate_by_lua_block {
certificate.call()
}
location = /status.tengine {
if ($host !~* "^\d{1,3}(\.\d{1,3}){3}|^status\.tengine\.com$") {
return 404;
break;
}
sysguard off;
access_log off;
root /etc/nginx/htdocs;
}
location / {
ingress_gateway_metadata "namespace" $metadata_namespace;
ingress_gateway_metadata "ingress_name" $metadata_ingress_name;
ingress_gateway_metadata "service_name" $metadata_service_name;
ingress_gateway_metadata "service_port" $metadata_service_port;
ingress_gateway_metadata "location_path" $metadata_location_path;
set $namespace $metadata_namespace;
set $ingress_name $metadata_ingress_name;
set $service_name $metadata_service_name;
set $service_port $metadata_service_port;
set $location_path $metadata_location_path;
set $upstream_read_time $ingress_read_timeout;
set $https_host_mode $ingress_force_https;
include /etc/nginx/apps/ssl.conf;
set $proxy_upstream_name $ingress_route_target;
add_header Ups-Target-Key $ingress_route_target always;
add_header X-protocol $server_protocol always;
add_header Alt-Svc 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000' always;
if ($ingress_route_target = "") {
return 404;
break;
}
# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
# other authentication method such as basic auth or external auth useless - all requests will be allowed.
#access_by_lua_block {
#}
header_filter_by_lua_block {
lua_ingress.header()
plugins.run()
}
body_filter_by_lua_block {
}
log_by_lua_block {
balancer.log()
monitor.call()
plugins.run()
}
sysguard off;
port_in_redirect off;
set $balancer_ewma_score -1;
set $proxy_host $proxy_upstream_name;
set $pass_access_scheme $scheme;
set $pass_server_port $server_port;
set $best_http_host $http_host;
set $pass_port $pass_server_port;
set $proxy_alternative_upstream_name "";
# CORS
ingress_gateway_metadata "enable-cors" $metadata_enable_cors;
ingress_gateway_metadata "cors-allow-origin" $metadata_cors_allow_origin;
ingress_gateway_metadata "cors-max-age" $metadata_cors_max_age;
ingress_gateway_metadata "cors-allow-credentials" $metadata_cors_allow_credentials;
ingress_gateway_metadata "cors-allow-methods" $metadata_cors_allow_methods;
ingress_gateway_metadata "cors-allow-headers" $metadata_cors_allow_headers;
set $enable_cors_options_credentials "${metadata_enable_cors}_${request_method}_${metadata_cors_allow_credentials}";
set $metadata_enable_cors_credentials "${metadata_enable_cors}_${metadata_cors_allow_credentials}";
client_max_body_size 1m;
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# tengine headers
proxy_set_header X-Real-Scheme $scheme;
proxy_set_header Web-Server-Type tengine-ingress;
proxy_set_header X-Request-From tengine-ingress;
proxy_set_header WL-Proxy-Client-IP $remote_addr;
proxy_set_header Proxy-X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-Http2 $http2;
proxy_set_header X-Connection $x_connection;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
if ($enable_cors_options_credentials = "true_OPTIONS_true") {
# Cors Preflight methods needs additional options and different Return Code
more_set_headers 'Access-Control-Allow-Origin: $metadata_cors_allow_origin';
more_set_headers 'Access-Control-Allow-Credentials: $metadata_cors_allow_credentials';
more_set_headers 'Access-Control-Allow-Methods: $metadata_cors_allow_methods';
more_set_headers 'Access-Control-Allow-Headers: $metadata_cors_allow_headers';
more_set_headers 'Access-Control-Max-Age: $metadata_cors_max_age';
more_set_headers 'Content-Type: text/plain charset=UTF-8';
more_set_headers 'Content-Length: 0';
return 204;
}
if ($enable_cors_options_credentials = "true_OPTIONS_false") {
# Cors Preflight methods needs additional options and different Return Code
more_set_headers 'Access-Control-Allow-Origin: $metadata_cors_allow_origin';
more_set_headers 'Access-Control-Allow-Methods: $metadata_cors_allow_methods';
more_set_headers 'Access-Control-Allow-Headers: $metadata_cors_allow_headers';
more_set_headers 'Access-Control-Max-Age: $metadata_cors_max_age';
more_set_headers 'Content-Type: text/plain charset=UTF-8';
more_set_headers 'Content-Length: 0';
return 204;
}
if ($metadata_enable_cors_credentials = "true_true") {
more_set_headers 'Access-Control-Allow-Origin: $metadata_cors_allow_origin';
more_set_headers 'Access-Control-Allow-Credentials: $metadata_cors_allow_credentials';
more_set_headers 'Access-Control-Allow-Methods: $metadata_cors_allow_methods';
more_set_headers 'Access-Control-Allow-Headers: $metadata_cors_allow_headers';
}
if ($metadata_enable_cors_credentials = "true_false") {
more_set_headers 'Access-Control-Allow-Origin: $metadata_cors_allow_origin';
more_set_headers 'Access-Control-Allow-Methods: $metadata_cors_allow_methods';
more_set_headers 'Access-Control-Allow-Headers: $metadata_cors_allow_headers';
}
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering off;
proxy_buffer_size 4k;
proxy_buffers 4 4k;
proxy_max_temp_file_size 1024m;
proxy_request_buffering on;
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0;
proxy_next_upstream_tries 3;
proxy_pass http://upstream_balancer;
proxy_redirect off;
}
location /robots.txt {
header_filter_by_lua_block {
lua_ingress.header()
plugins.run()
}
root /etc/nginx/htdocs;
sysguard off;
set $log_host "robots.tengine.com";
proxy_set_header Host $http_host;
proxy_set_header X-Request-From tengine-ingress;
set $upstream_read_time $ingress_read_timeout;
set $https_host_mode $ingress_force_https;
include /etc/nginx/apps/ssl.conf;
set $proxy_upstream_name $ingress_route_target;
add_header Ups-Target-Key $ingress_route_target always;
add_header X-protocol $server_protocol always;
add_header Alt-Svc 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000' always;
set $https_use_timing "*";
if ($ingress_route_target = "") {
return 404;
break;
}
ingress_gateway_metadata "disable-robots" $metadata_disable_robots;
if ($metadata_disable_robots = "false") {
proxy_pass http://upstream_balancer;
}
}
# health checks in cloud providers require the use of port 80
location /healthz {
access_log off;
return 200;
}
# this is required to avoid error if nginx is being monitored
# with an external software (like sysdig)
location /nginx_status {
allow 127.0.0.1;
allow ::1;
deny all;
sysguard off;
access_log off;
stub_status on;
}
location /traffic_status {
allow 127.0.0.1;
allow ::1;
deny all;
sysguard off;
access_log off;
req_status_show;
}
}
## end server _
# backend for when default-backend-service is not configured or it does not have endpoints
server {
listen 8181 default_server reuseport backlog=4096;
listen [::]:8181 default_server reuseport backlog=4096;
set $proxy_upstream_name "internal";
access_log off;
location / {
return 404;
}
}
# default server, used for NGINX healthcheck and access to nginx stats
server {
listen 127.0.0.1:10246;
server_name status.tengine.com;
set $proxy_upstream_name "internal";
sysguard off;
keepalive_timeout 0;
gzip off;
access_log off;
if ($host !~* "^\d{1,3}(\.\d{1,3}){3}|^status\.tengine\.com$") {
set $log_host "notfound";
return 404;
break;
}
location /healthz {
return 200;
}
location /is-dynamic-lb-initialized {
content_by_lua_block {
local configuration = require("configuration")
local backend_data = configuration.get_backends_data()
if not backend_data then
ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
return
end
ngx.say("OK")
ngx.exit(ngx.HTTP_OK)
}
}
location /nginx_status {
stub_status on;
}
location /configuration {
client_max_body_size 21m;
client_body_buffer_size 21m;
proxy_buffering off;
content_by_lua_block {
configuration.call()
}
}
location / {
content_by_lua_block {
ngx.exit(ngx.HTTP_NOT_FOUND)
}
}
# for health check on alibaba
location = /status.tengine {
root /etc/nginx/htdocs;
}
location = /traffic_status {
allow 10.0.0.0/8;
allow 11.0.0.0/8;
allow 172.16.0.0/12;
allow 127.0.0.1/32;
allow 192.168.0.0/16;
deny all;
req_status_show;
}
}
}
stream {
lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;;";
lua_shared_dict tcp_udp_configuration_data 5M;
init_by_lua_block {
collectgarbage("collect")
-- init modules
local ok, res
ok, res = pcall(require, "configuration")
if not ok then
error("require failed: " .. tostring(res))
else
configuration = res
end
ok, res = pcall(require, "tcp_udp_configuration")
if not ok then
error("require failed: " .. tostring(res))
else
tcp_udp_configuration = res
end
ok, res = pcall(require, "tcp_udp_balancer")
if not ok then
error("require failed: " .. tostring(res))
else
tcp_udp_balancer = res
end
}
init_worker_by_lua_block {
tcp_udp_balancer.init_worker()
}
lua_add_variable $proxy_upstream_name;
log_format log_stream '[$remote_addr] [$time_local] $protocol $status $bytes_sent $bytes_received $session_time';
access_log /var/log/nginx/access.log log_stream ;
error_log /var/log/nginx/error.log;
upstream upstream_balancer {
server 0.0.0.1:1234;
# placeholder
balancer_by_lua_block {
tcp_udp_balancer.balance()
}
}
server {
listen 127.0.0.1:10247;
access_log off;
content_by_lua_block {
tcp_udp_configuration.call()
}
}
# TCP services
# UDP services
}
常见问题排查：
Q：修改ingress的 configmap 新增http监听端口后不生效
kubectl edit configmap tengine-ingress-configuration -n tengine-ingress
查看pod有如下日志
AI0305 08:41:43.895187 7 event.go:298] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"tengine-ingress", Name:"tengine-ingress-configuration", UID:"6a49838b-47d5-43e4-8c40-7b94d985f02c", APIVersion:"v1", ResourceVersion:"541158", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap tengine-ingress/tengine-ingress-configuration I0305 08:41:43.924991 7 main.go:97] Get annotation ingress-rollout, err: ingress rule without annotations I0305 08:41:43.925090 7 main.go:103] Get annotation ingress-rollout-current-revision, err: ingress rule without annotations I0305 08:41:43.925099 7 main.go:109] Get annotation ingress-rollout-update-revision, err: ingress rule without annotations I0305 08:41:43.925104 7 main.go:115] Get annotation ingress-rollout-index-id, err: ingress rule without annotations I0305 08:41:43.925126 7 main.go:71] Get annotation version, err: ingress rule without annotations I0305 08:41:43.925158 7 main.go:54] Get annotation ingress-referrer, err: ingress rule without annotations I0305 08:41:43.929792 7 store.go:1509] Ingress[default/ingress-echo] with empty referrer I0305 08:41:43.932446 7 store.go:1544] Get ingress default/ingress-echo status {Gray[type:0],IngressGrayFlag[false],IngressGrayCurVer[],IngressGrayNewVer[],IngressGrayIndex[0],PodOrdinal[-1]} I0305 08:41:43.932631 7 main.go:97] Get annotation ingress-rollout, err: ingress rule without annotations I0305 08:41:43.932649 7 main.go:103] Get annotation ingress-rollout-current-revision, err: ingress rule without annotations I0305 08:41:43.932654 7 main.go:109] Get annotation ingress-rollout-update-revision, err: ingress rule without annotations I0305 08:41:43.932660 7 main.go:115] Get annotation ingress-rollout-index-id, err: ingress rule without annotations I0305 08:41:43.932681 7 main.go:71] Get annotation version, err: ingress rule without annotations I0305 08:41:43.932705 7 main.go:54] Get annotation ingress-referrer, err: ingress rule without annotations I0305 08:41:43.933862 7 ingcheck.go:40] Check Ingress ID ignored for empty IngressCheckSum I0305 08:41:43.950815 7 controller.go:161] No configuration change detected, skipping hot reload.
tengine的特性，参考文档：https://tengine.taobao.org/document/ingress_rollout_ingress.html
修改ingress的配置，后错误消息
浙公网安备 33010602011771号