cttproxy,内核补丁，haproxy全透明代理

,...

http://blog.catjia.com/linux-system/load-balance/2012_08_881/

20130514补充：

必须配置转发。。。

net.ipv4.ip_forward = 1

这个忘了写上了，感谢网友 @mcg 的提醒

～～～～～～～～

昨晚写了三分之二的文章就因为电脑突然死机而烟消云散，定是因为我废话太多惩罚我来的。。。所以这次就精简些罢，不懂的就谷歌去。。。

Haproxy透明代理三步走：

编译内核

源码安装iptables

安装配置haproxy

系统版本：

CentOS 5.5

目的：

搭建haproxy透明代理，做app服务负载均衡

步骤：

PS：当然也可以下载本人编译好的 kernel-2.6.32.27tproxy-1.i386.rpm，下载地址：
kernel-2.6.32.27tproxy-1.i386.rpm kernel-2.6.32.27tproxy-1.x86_64.rpm

1、编译内核

为何要编译内核，只因为要用到tproxy。。。

内核版本2.6.28开始不用再打tproxy的补丁，所以在这里选择了kernel-2.6.32.27来编译

 



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# yum install make rpm-build gcc gcc-c++ ncurses-devel elfutils elfutils-libs libstdc++-devel -y
 
# cd /usr/src
 
# wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.27.tar.bz2
 
# tar xf linux-2.6.32.27.tar.bz2
 
# ln -s linux-2.6.32.27 linux
 
# cd linux
 
# make clean && make mrproper
 
# cp /boot/config-`uname -r` ./.config
 
# make menuconfig

选中 Load an Alternate Configuration File, 按Enter进去，默认加载 .config文件，即上面从/boot分区拷贝过来的 .config文件

然后选中以下对应的选项以支持TPROXY

 



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

Networking
 
>Networking support->
 
>Networking options->
 
>Network packet filtering framework(Netfilter)->
 
>Core Netfilter Configuration->
 
Transparent proxying support(EXPERIMENTAL)
 
Netfilter Xtables support(required forip_tables)
 
Netfilter Connection tracking
 
>Connection tracking flow accounting
 
>Connection mark tracking support
 
"TPROXY"target support(EXPERIMENTAL)
 
"socket"match support(EXPERIMENTAL)

通常选中后会是下面两种情况：

[*]: Built-in

[M]: Module

尽可能以模块[M]选中以上对应的选项

Tips:

连按两下 <ESC> 可以返回上一层菜单

返回主菜单后选中以下选项：

 
1
2
3

General setup–>
 
　　[*]enable deprecated sysfs features tosupport old userspace tools

修改这项是因为旧版的mkinitrd及其nash在内核没有CONFIG_SYSFS_DEPRECATED_V2参数时默认使用旧版sysfs路径格式，从而在新内核下无法正确访问/sys内的硬盘信息节点。

不修改会报如下错误：

 



1
2
3
4
5
6
7
8
9
10
11

mount:could notfind filesystem‘/dev/root’
 
setuproot:moving/dev failed:No such file ordirectory
 
setuproot:error mounting/proc:No such file ordirectory
 
setuproot:error mounting/sys:No such file ordirectory
 
switchroot:mount failed:No such file ordirectory
 
Kernel panic–notsyncing:Attempted tokill init!

Tips:

注意全部操作完成保存后编辑 .config文件查看一下是否有以下内容：

 
1

CONFIG_SYSFS_DEPRECATED_V2=y

然后按enter编辑下面这个选项：

 
1
2
3

General Setup—>
 
()Local version–append tokernel release

输入一个字符串用来分辨新内核，比如：

-tproxy

然后通过连按<ESC>退出编辑菜单，最后选择 YES 保存。。。

保存后执行下面的命令以生成rpm包：

 
1

# make rpm

好了，现在可以去喝杯咖啡了－－这个过程视机子性能不同而所需时间长短也不同。。。

生成的rpm包在 /usr/src/redhat/RPMS/i386(或者是x86_64)

我这里系统是64位的，所以。。。

 
1
2
3

# rpm -ivh --nodeps /usr/src/redhat/RPMS/x86_64/kernel-2.6.32.27tproxy-1.x86_64.rpm
 
# mkinitrd /boot/initrd-2.6.32.27-tproxy.img 2.6.32.27-tproxy

编辑grub.conf文件，添加以下内容：

 
1
2
3
4
5
6
7

title CentOS-Tproxy(2.6.32.27-tproxy)
 
root(hd0,0)
 
kernel/boot/vmlinuz-2.6.32.27-tproxy ro root=LABEL=/
 
initrd/boot/initrd-2.6.32.27-tproxy.img

保存后重启。。。

记住了，重启后须是从新内核进入系统。。。

Tips:

上面kernel那一行后面不要加其他东西，像我之前加了acpi=off noapic这两个参数启动时内核就崩溃了。。

从新内核启动进入系统，可以运行下面这条命令查看一下内核版本：

 
1

# uname -a

2、源码安装iptables

内核编译完了，现在需要从源码安装iptables，因为centos5上的iptables版本太低，要支持tproxy的话还得打补丁，所以这里选择了较新版本 iptables1.4.8

 



1
2
3
4
5
6
7
8
9
10
11

# wget http://www.netfilter.org/projects/iptables/files/iptables-1.4.8.tar.bz2
 
# yum remove iptables -y
 
# tar xf iptables-1.4.8.tar.bz2
 
# cd iptables-1.4.8
 
# ./configure --prefix=/usr/local/iptables
 
# make && make install

3、安装配置haproxy

 



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.21.tar.gz
 
# tar xf haproxy-1.4.21.tar.gz
 
# cd haproxy-1.4.21
 
# yum -y install pcre-devel
 
# make TARGET=linux26 CPU=x86_64 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
 
# make install target=linux26
 
# mkdir -p /usr/local/haproxy/{conf,run,sbin}
 
# ln -s /usr/local/sbin/haproxy /usr/local/haproxy/sbin

配置haproxy:

 
1

# vi /usr/local/haproxy/conf/haproxy.cfg

添加以下内容：

 



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

global
 
log127.0.0.1local1notice
 
maxconn32000
 
daemon
 
pidfile/usr/local/haproxy/run/haproxy.pid
 
defaults applications TCP
 
log global
 
mode tcp
 
maxconn32000
 
retries3
 
option redispatch
 
balance leastconn
 
listen app_tcp172.168.16.10:3999
 
mode tcp
 
source0.0.0.0usesrc clientip
 
option srvtcpka
 
option clitcpka
 
option tcpka
 
option tcplog
 
clitimeout3d
 
srvtimeout3d
 
contimeout3d
 
server app1172.168.16.54:3999check
 
listen stats_auth172.168.16.10:10800
 
mode http
 
option httplog
 
stats enable
 
stats refresh20s
 
stats uri/status
 
stats auth admin:admin
 
stats hide-version

设置开机自启动：

 
1

# vi /etc/rc.local

添加以下内容：

 
1
2

/usr/local/haproxy/sbin/haproxy-D-f
/usr/local/haproxy/conf/haproxy.cfg

配置平台日志支持：

 
1

# vi /etc/syslog.conf

添加：

 
1
2
3

local3.*/var/log/haproxy.log
 
local0.*/var/log/haproxy.log

 
1

# vi /etc/sysconfig/syslog

修改：

 
1

SYSLOGD_OPTIONS="-r -m 0"

重启一下：

 
1

# /etc/init.d/syslog restart

配置路由转发：

 
1

# vi /etc/sysctl.conf

 
1

net.ipv4.conf.default.rp_filter=1

添加以下内容到/etc/rc.local

 



1
2
3
4
5
6
7
8
9
10
11

iptables-tmangle-NDIVERT
 
iptables-tmangle-APREROUTING-ptcp-msocket-jDIVERT
 
iptables-tmangle-ADIVERT-jMARK--set-mark111
 
iptables-tmangle-ADIVERT-jACCEPT
 
ip rule add fwmark111lookup100
 
ip route add local0.0.0.0/0dev lo table100

Tips:

需要注意的一点，后端服务器的网关必须指向ha。。。

切记切记。。。

==================================================================

http://blog.loadbalancer.org/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy/

Standard Kernel builds don’t support TPROXY ( 2.6.28 does now!). For example if you use HaProxy as the load balancer then all of the backend servers see the traffic coming from the IP address of the load balancer. TPROXY allows you to make sure the backend servers see the true client IP address in the logs.

Ps. An easier alternative is inserting the clients ip in the x-forwarded-for header (option forwardfor).

For TPROXY to work you need three things:

1) TPROXY compiled into the linux kernel 2) TPROXY / Socket compiled into netfilter / iptables (due in v1.4.3?) 3) HaProxy compiled with the USE_LINUX_TPROXY option

The TPROXY patch for Linux Kernel 2.6.25.11 is here: http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2

The following is a guide how to install on Centos 5.1:

Heavily borrowed from: http://howtoforge.com/kernel_compilation_centos

Download The Kernel Sources

First we download our desired kernel to /usr/src. Go to www.kernel.org and select the kernel you want to install, e.g. linux-2.6.25.11.tar.bz2 (you can find all 2.6 kernels here: http://www.kernel.org/pub/linux/kernel/v2.6/). Then you can download it to /usr/src like this:

cd /usr/src wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.25.11.tar.bz2 Then we unpack the kernel sources and create a symlink linux to the kernel sources directory:

tar xjf linux-2.6.25.11.tar.bz2 ln -s linux-2.6.25.11 linux cd /usr/src/

wget http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2

tar -xjf tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2

cd linux

cat ../tproxy-kernel-2.6.25-20080519-165031-1211208631/00*.patch | patch -p1 --dry-run
cat ../tproxy-kernel-2.6.25-20080519-165031-1211208631/00*.patch | patch -p1

Configure The Kernel

It’s a good idea to use the configuration of your current working kernel as a basis for your new kernel. Therefore we copy the existing configuration to /usr/src/linux:

make clean && make mrproper cp /boot/config-`uname -r` ./.config

I needed to do a: yum install ncurses-devel gcc gcc-c++ make rpm-build

Then we run

make menuconfig

which brings up the kernel configuration menu. Go to Load an Alternate Configuration File and choose .config (which contains the configuration of your current working kernel) as the configuration file:

Then browse through the kernel configuration menu and make your choices.

Make sure you enable tproxy support, `socket' and `TPROXY' modules (with optional conntrack support if you need SNAT)

Make sure you specify a kernel version identification string under General Setup —> () Local version – append to kernel release. I use CS3 so our kernel rpm package will be named kernel-2.6.25.11CS3.x86_64.rpm. You can leave the string empty or specify a different one which helps you identify the kernel (e.g. -custom or whatever you like).

Please note: After you have installed kernel-2.6.25.11CS3.x86_64.rpm and decide to compile another 2.6.25 kernel rpm package, it is important to use a different version string, e.g. -default1, -default2, etc., because otherwise you can’t install your new kernel because rpm complains that kernel-2.6.25.11CS3.x86_64.rpm is already installed!

Once you are happy with the kernel configuration, save & exit menuconfig then simply:

make rpm

This may take quite a long time…. Once it has finished:

Source RPM is here: ls -l /usr/src/redhat/SRPMS/ Binary RPM is here: ls -l /usr/src/redhat/RPMS/x86_64/

now install the new kernel: cd /usr/src/redhat/RPMS/x86_64/ rpm -ivh --nodeps kernel-2.6.25CS-1.x86_64.rpm

Now you can either run the following command: /sbin/new-kernel-pkg –package kernel –mkinitrd –depmod –install 2.6.25CS

Or you can do the usual manual steps i.e.

Make sure you create a new initrd file: mkinitrd /boot/initrd-2.6.25.11CS3.img 2.6.25.11CS3

Now configure the boot loader: vi /boot/grub/menu.lst

default=0 timeout=5 splashimage=(hd0,0)/boot/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.25.11CS3) root (hd0,0) kernel /boot/vmlinuz-2.6.25.11CS3 ro root=LABEL=/ initrd /boot/initrd-2.6.25.11CS3.img

That’s it, so reboot and do a: uname -a

To check that we are using the new kernel: Linux lbmaster 2.6.25.11CS3 #6 SMP Mon Jul 28 13:10:43 GMT 2008 x86_64 x86_64 x86_64 GNU/Linux

Compiling iptables with TPROXY support:

First download the current iptables source code: cd /usr/src/ wget http://www.netfilter.org/projects/iptables/files/iptables-1.4.0.tar.bz2 tar -xjf iptables-1.4.0.tar.bz2 Then download the tproxy patch: wget http://www.balabit.com/downloads/files/tproxy/tproxy-iptables-1.4.0-20080521-113954-1211362794.patch

cd /usr/src/iptables-1.4.0/
cat ../tproxy-iptables*.patch | patch -p1
make
make install

Compiling HAProxy with TPROXY support:

Download the latest version of the HAProxy source code:

wget http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.15.7.tar.gz tar -xvf haproxy-1.3.15.7.tar.gz cd haproxy-1.3.15.7/

Then compile making sure to enable TPROXY make TARGET=linux26 CPU=x86_64 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1 make install target=linux26 If you have got this far then great, thats the hard part done!

Now before Haproxy can utilise TPROXY we need to set up some firewall marks: You can put this script in a start up file such as rc.local etc. #!/bin/bash iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 111 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 111 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 We also need to ensure that we have the correct architecture for the TPROXY trick to work. Using the normal HAProxy you can have real servers anywhere on the internet because the source address always points back at the HAProxy units IP address. However if the clients source IP address is going to be used then the HAProxy server MUST BE IN THE PATH of the return traffic. The easiest way to do this is to put the backend servers in a different subnet to the front end clients and make sure that the default gateway points back at the HAProxy load balancer.

NB. With clever routing this should be possible on the same subnet but I haven’t tried that yet!

So here is an example configuration that I used for HAProxy:

# HAProxy configuration file
global
#	uid 99
#	gid 99
	daemon
	stats socket /var/run/haproxy.stat mode 600
	log 127.0.0.1 local4
	maxconn 40000
	ulimit-n 80013
	pidfile /var/run/haproxy.pid
defaults
	log global
	mode	http
	contimeout	4000
	clitimeout	42000
	srvtimeout	43000
	balance	roundrobin
listen	VIP_Name 192.168.2.87:80
	mode	http
	option	forwardfor
	source 0.0.0.0 usesrc clientip
	cookie	SERVERID insert nocache indirect
	server server1 10.0.0.60:80 weight 1 cookie server1 check
	server server2 10.0.0.61:80 weight 1 cookie server2 check
	server	backup 127.0.0.1:80 backup
	option redispatch

The most important line is this one:

	source 0.0.0.0 usesrc clientip

If your test setup doesn’t work then remove this line to check if a standard configuration does work.

Check your backend server logs to ensure that the client source IP address is correctly showing.

NB. One gotcha (of the many) is that you can no long use any local (i.e. 127.0.0.1) backup servers due to routing issues. To resolve this change the backup server definition as follows: server backup 127.0.0.1:80 backup source 0.0.0.0

Ps. Many thanks to John Lauro for his help with the firewall marks stuff

Oh and forgot to say change your sysctrls to allow redirects.. i.e. echo 1 > /proc/sys/net/ipv4/conf/all/forwarding echo 1 > /proc/sys/net/ipv4/conf/all/send_redirects echo 1 > /proc/sys/net/ipv4/conf/eth0/send_redirects

==================================================================

http://bbs.chinaunix.net/thread-1950059-1-1.html

http://blog.chinaunix.net/uid-20546486-id-1927805.html

Standard Kernel builds don’t support TPROXY ( 2.6.28 does now!).
For example if you use HaProxy as the load balancer then all of the backend servers see the traffic coming from the IP address of the load balancer. TPROXY allows you to make sure the backend servers see the true client IP address in the logs.

Ps. An easier alternative is inserting the clients ip in the x-forwarded-for header (option forwardfor).

For TPROXY to work you need three things:

1) TPROXY compiled into the linux kernel
2) TPROXY / Socket compiled into netfilter / iptables (due in v1.4.3?)
3) HaProxy compiled with the USE_LINUX_TPROXY option

The TPROXY patch for Linux Kernel 2.6.25.11 is here:
http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2

The following is a guide how to install on Centos 5.1:

Heavily borrowed from: http://howtoforge.com/kernel_compilation_centos

Download The Kernel Sources

cd /usr/src wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.25.11.tar.bz2
Then we unpack the kernel sources and create a symlink linux to the kernel sources directory:

tar xjf linux-2.6.25.11.tar.bz2 ln -s linux-2.6.25.11 linux cd /usr/src/

wget http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2

tar -xjf tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2

cd linux

cat ../tproxy-kernel-2.6.25-20080519-165031-1211208631/00*.patch | patch -p1 --dry-run
cat ../tproxy-kernel-2.6.25-20080519-165031-1211208631/00*.patch | patch -p1

Configure The Kernel

It’s a good idea to use the configuration of your current working kernel as a basis for your new kernel. Therefore we copy the existing configuration to /usr/src/linux:

make clean && make mrproper cp /boot/config-`uname -r` ./.config

I needed to do a:
yum install ncurses-devel gcc gcc-c++ make rpm-build

Then we run

make menuconfig

Then browse through the kernel configuration menu and make your choices.

Make sure you enable tproxy support, `socket' and `TPROXY' modules (with optional conntrack support if you need SNAT)

Make sure you specify a kernel version identification string under General Setup —> () Local version - append to kernel release. I use CS3 so our kernel rpm package will be named kernel-2.6.25.11CS3.x86_64.rpm. You can leave the string empty or specify a different one which helps you identify the kernel (e.g. -custom or whatever you like).

Once you are happy with the kernel configuration, save & exit menuconfig then simply:

make rpm

This may take quite a long time….
Once it has finished:

Source RPM is here:
ls -l /usr/src/redhat/SRPMS/
Binary RPM is here:
ls -l /usr/src/redhat/RPMS/x86_64/

now install the new kernel:
cd /usr/src/redhat/RPMS/x86_64/ rpm -ivh --nodeps kernel-2.6.25CS-1.x86_64.rpm

Now you can either run the following command:
/sbin/new-kernel-pkg –package kernel –mkinitrd –depmod –install 2.6.25CS

Or you can do the usual manual steps i.e.

Make sure you create a new initrd file:
mkinitrd /boot/initrd-2.6.25.11CS3.img 2.6.25.11CS3

Now configure the boot loader:
vi /boot/grub/menu.lst

That’s it, so reboot and do a:
uname -a

To check that we are using the new kernel:
Linux lbmaster 2.6.25.11CS3 #6 SMP Mon Jul 28 13:10:43 GMT 2008 x86_64 x86_64 x86_64 GNU/Linux

Compiling iptables with TPROXY support:

First download the current iptables source code:
cd /usr/src/ wget http://www.netfilter.org/projects/iptables/files/iptables-1.4.0.tar.bz2 tar -xjf iptables-1.4.0.tar.bz2
Then download the tproxy patch:
wget http://www.balabit.com/downloads/files/tproxy/tproxy-iptables-1.4.0-20080521-113954-1211362794.patch

cd /usr/src/iptables-1.4.0/
cat ../tproxy-iptables*.patch | patch -p1
make
make install

Compiling HAProxy with TPROXY support:

Download the latest version of the HAProxy source code:

wget http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.15.7.tar.gz
tar -xvf haproxy-1.3.15.7.tar.gz
cd haproxy-1.3.15.7/

Then compile making sure to enable TPROXY
make TARGET=linux26 CPU=x86_64 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1 make install target=linux26
If you have got this far then great, thats the hard part done!

Now before Haproxy can utilise TPROXY we need to set up some firewall marks:
You can put this script in a start up file such as rc.local etc.
#!/bin/bash iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 111 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 111 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100
We also need to ensure that we have the correct architecture for the TPROXY trick to work. Using the normal HAProxy you can have real servers anywhere on the internet because the source address always points back at the HAProxy units IP address. However if the clients source IP address is going to be used then the HAProxy server MUST BE IN THE PATH of the return traffic.
The easiest way to do this is to put the backend servers in a different subnet to the front end clients and make sure that the default gateway points back at the HAProxy load balancer.

NB. With clever routing this should be possible on the same subnet but I haven’t tried that yet!

So here is an example configuration that I used for HAProxy:

# HAProxy configuration file
global
#	uid 99
#	gid 99
	daemon
	stats socket /var/run/haproxy.stat mode 600
	log 127.0.0.1 local4
	maxconn 40000
	ulimit-n 80013
	pidfile /var/run/haproxy.pid
defaults
	log global
	mode	http
	contimeout	4000
	clitimeout	42000
	srvtimeout	43000
	balance	roundrobin
listen	VIP_Name 192.168.2.87:80
	mode	http
	option	forwardfor
	source 0.0.0.0 usesrc clientip
	cookie	SERVERID insert nocache indirect
	server server1 10.0.0.60:80 weight 1 cookie server1 check
	server server2 10.0.0.61:80 weight 1 cookie server2 check
	server	backup 127.0.0.1:80 backup
	option redispatch

The most important line is this one:

	source 0.0.0.0 usesrc clientip

If your test setup doesn’t work then remove this line to check if a standard configuration does work.

Check your backend server logs to ensure that the client source IP address is correctly showing.

NB. One gotcha (of the many) is that you can no long use any local (i.e. 127.0.0.1) backup servers due to routing issues.
To resolve this change the backup server definition as follows:
server backup 127.0.0.1:80 backup source 0.0.0.0

Ps. Many thanks to John Lauro for his help with the firewall marks stuff

Oh and forgot to say change your sysctrls to allow redirects.. i.e.
echo 1 > /proc/sys/net/ipv4/conf/all/forwarding echo 1 > /proc/sys/net/ipv4/conf/all/send_redirects echo 1 > /proc/sys/net/ipv4/conf/eth0/send_redirects

This entry was posted on Wednesday, February 11th, 2009 at 2:12 pm and is filed under Linux, Load Balancing, haproxy. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

18 Responses to “Configure HAProxy with TPROXY kernel for full transparent proxy”

Carlo Says:
May 6th, 2009 at 10:40 am
Hello, when I try to compile iptables, after “make”, I run “make install”: I have an error when it build dependencies:

Unable to resolve dependency on asm/swab.h. Try ‘make clean’

Have you got ideas?
Malcolm Turnbull Says:
May 6th, 2009 at 11:41 am
Strange, are you sure you left your kernel headers in place before compiling iptables?
make mrproper will destroy them, make clean should be safe enough.
Carlo Says:
May 7th, 2009 at 4:51 pm
I’ve solved my problem by compiling iptables 1.4.3 that have tproxy patch inside (problem was introduced by balabit patch).

But now I’ve another problem: line “source 0.0.0.0 usesrc clientip” doesn’t work: I obtain “503 Service Unavailable”.
Please note that haproxy bind on a public IP and servers (two) are in a private network (haproxy’s server have two eth with public and private ip).

Whitout “source 0.0.0.0 usesrc clientip” it work.

Have you got ideas?
Malcolm Turnbull Says:
May 7th, 2009 at 6:12 pm
Carlo,

Yes, patches only work against the exact version….
Are you sure you have set the default gateway on the backend servers to point at the internal interface on the haproxy instance. Traffic must pass through the proxy both ways.
Does your setup work without that line?
Carlo Says:
May 7th, 2009 at 7:08 pm
Firstly I’ve followeb your howto but iptables patch fails. So I have compiled iptables 1.4.3

Haproxy configuration is the same that you have posted here.
Can you provide me an example please?

Without “source” line my setup work.

Thanks.
Carlo Says:
May 13th, 2009 at 2:09 pm
Solved.
There was a problem in haproxy.cfg iptables:

for haproxy use “source HAPROXY_IP usesrc clientip” (HAPROXY_IP can be public o private ip)

for iptables:

/usr/local/sbin/iptables -t mangle -N DIVERT
/usr/local/sbin/iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
/usr/local/sbin/iptables -t mangle -A DIVERT -j MARK –set-mark 1
/usr/local/sbin/iptables -t mangle -A DIVERT -j ACCEPT

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

Thanks!
Joseph Says:
May 14th, 2009 at 7:33 am
Hello Malcolm,

Can you please provide details on what specific module to enable for the kernel? I am having a problem with my iptables, please see below;

[root@SE-Caching ~]# service iptables status
Table: mangle
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 DIVERT tcp — 0.0.0.0/0 0.0.0.0/0 UNKNOWN match `socket’
2 TPROXY tcp — 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 [16 bytes of unknown target data]

What causes this one (UNKNOWN match `socket’ )?
Malcolm Turnbull Says:
May 14th, 2009 at 7:05 pm
Hi Joseph,

The modules load on demand so…

Yes you’ve probably missed one of the required options while doing make menuconfig….
Its really annoying that make menuconfig only shows the nice names of modules and not the precise name as in the .config file. As I forgot to take an exact copy of the text from each option at the time I can’t help much on the nice description side….

Looking at my .config file I think the important ones are:
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_TPROXY=m
CONFIG_NETFILTER_XTABLES=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CT_ACCT=y
CONFIG_NF_CONNTRACK_MARK=y

IF in doubt select them ALL!

Contact support@loadbalancer.org by email if you want the entire .config which works and may well help.
I can even give you a full Centos5 kernel rpm if you want.
Jim Says:
May 16th, 2009 at 6:29 am
Hi I have the same issue as Joseph:

Table: mangle
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 DIVERT tcp — 0.0.0.0/0 0.0.0.0/0 UNKNOWN match `socket’

I recompiled the kernel a couple of times making sure I had at least all the modules you mention.

Any pointers as to why I’m getting this error would be greatly appreciated.
Thanks.
Malcolm Turnbull Says:
June 9th, 2009 at 6:09 pm
Jim,
I think it means iptables can’t find or load the socket module, did your patch and re-compile of iptables go OK? Does the lsmod command show the relevant modules?
Jim Says:
June 11th, 2009 at 11:55 pm
Thanks for you response Jim, it looks like the relevant modules are there, below is the lsmod output.
The weird thing is it’s working fine but that UNKNOWN match `socket’ still shows when I do /etc/init.d/iptables status.

Module Size Used by
xt_MARK 1892 1
xt_socket 2212 1
nf_conntrack 59436 1 xt_socket
nf_defrag_ipv4 1764 1 xt_socket
nf_tproxy_core 2404 1 xt_socket,[permanent]
iptable_mangle 2372 1
ip_tables 10452 1 iptable_mangle
x_tables 13864 3 xt_MARK,xt_socket,ip_tables
ipv6 214388 12
autofs4 20648 2
hidp 12196 2
rfcomm 28692 0
l2cap 18020 10 hidp,rfcomm
bluetooth 48900 5 hidp,rfcomm,l2cap
sunrpc 160704 1
dm_multipath 12880 0
sbs 10956 0
sbshc 5124 1 sbs
battery 10088 0
lp 8804 0
sg 25208 0
floppy 46980 0
ide_cd_mod 25996 0
cdrom 30176 1 ide_cd_mod
serio_raw 4712 0
parport_pc 23044 1
ac 4072 0
button 6004 0
parport 30796 2 lp,parport_pc
pcnet32 29188 0
mii 4740 1 pcnet32
i2c_piix4 8564 0
i2c_core 20824 1 i2c_piix4
pcspkr 2276 0
dm_snapshot 16040 0
dm_zero 1348 0
dm_mirror 11912 0
dm_region_hash 9828 1 dm_mirror
dm_log 8360 2 dm_mirror,dm_region_hash
dm_mod 47720 11 dm_multipath,dm_snapshot,dm_zero,dm_mirror,dm_l og
ata_piix 21384 0
libata 147852 1 ata_piix
mptspi 14988 2
mptscsih 28996 1 mptspi
mptbase 71364 2 mptspi,mptscsih
scsi_transport_spi 18980 1 mptspi
sd_mod 23456 3
scsi_mod 134356 6 sg,libata,mptspi,mptscsih,scsi_transport_spi,sd_mod
ext3 107308 2
jbd 40280 1 ext3
uhci_hcd 18516 0
ohci_hcd 19444 0
ehci_hcd 29264 0
Michael Dodwell Says:
July 13th, 2009 at 6:35 am
Joseph,

If you’ve installed iptables from source make sure your calling it, and not the package installed version.

ie

/usr/local/sbin/iptables and not /sbin/iptables
Loadbalancer.org Blog » Blog Archive » Transparent proxy of SSL traffic using Pound to HAProxy backend patch and howto Says:
July 20th, 2009 at 4:14 pm
[...] so I’ve previously blogged about how to get TPROXY and HAProxy working nicely together. But what if you want to terminate SSL traffic on the load balancer in order to use HaProxy to [...]
Vadim Says:
August 27th, 2009 at 9:22 pm
Hi friends! I used haproxy in my project. But I have one problem. What I can switch between two backends servers (me need used rule url_sub) if I used haproxy as frontend

global
uid 1004
gid 1002
daemon
#stats socket /var/run/haproxy.stat mode 600
log 127.0.0.1 local4
maxconn 40000
ulimit-n 80013
pidfile /var/run/haproxy.pid
defaults
log global
mode http
contimeout 4000
clitimeout 42000
srvtimeout 43000
balance roundrobin
listen XXX.COM x.x.x.x:80
mode http
option forwardfor
stats enable
stats auth bvv2001:la-la-la
cookie XXX.COM insert
source 0.0.0.0 usesrc clientip
server server1 x.x.x.x:10000 cookie XXX.COM check
server server2 x.x.x.x:10001 cookie XXX.COM check
dst_port 10000 if url_sub !sms
dst_port 10001 if url_sub sms

this config NOT WORKING
Help me please!
Twitter Trackbacks for Loadbalancer.org Blog » Blog Archive » Configure HAProxy with TPROXY kernel for full transparent proxy [loadbalancer.org] on Topsy.comSays:
August 31st, 2009 at 9:44 am
[...] Loadbalancer.org Blog » Blog Archive » Configure HAProxy with TPROXY kernel for full transparent p… blog.loadbalancer.org/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy – view page – cached loadbalancer logo phone number 0870 4438779 space image email sales@loadbalancer.org — From the page [...]
Gael Says:
September 17th, 2009 at 7:43 pm
Hi Malcolm and thanks for your brilliant article!

I have wasted a couple of days trying to debug the kernel which was throwing SOFT LOCKUP bugs. After investigation it was down to having 2 vCPU in the VM (from VMWare) I was using. Moving back to only one vCPU did the trick. Hope this can help someone else!

In your post, you have commented the following line:
# uid 99
# gid 99

Therefore, haproxy is running as root.

I know it produces the following error if haproxy is run with another user then root:
“Starting haproxy: [ALERT] 259/184020 (3497) : [/usr/sbin/haproxy.main()] Some configuration options require full privileges, so global.uid cannot be changed.”

So it seems compulsory to run haproxy as root.
I am now wondering if this can cause any problem related to using root to run haproxy.

Thanks in advance for your answer!

Gael
Malcolm Turnbull Says:
September 25th, 2009 at 11:52 am
Willy suggests starting HAProxy as root because it can then jail itself in a chroot and drop all of its privileges before starting the instances. This is not possible if it is not started as root because only root can execute chroot()
Malcolm Turnbull Says:
September 25th, 2009 at 12:05 pm
You need something more like:

frontend blah acl notsms url_sub !sms use_backend backend1 acl sms url_sub sms use_backend backend2

backend backend1 source 0.0.0.0 usesrc clientip server server1 x.x.x.x:10000 cookie XXX.COM check

NB. I haven’t tested this just off the top of my head…..
Check out the HAProxy manual and mailing list for more specific help on ACLs…

posted @ 2013-06-18 09:26 陳聽溪阅读(1728) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

陳聽溪

cttproxy,内核补丁，haproxy全透明代理

18 Responses to “Configure HAProxy with TPROXY kernel for full transparent proxy”

公告