1、LVS-fwm
fwm：FireWall Mark
在netfilter上給報文打標記；mangle表；
ipvsadm -A|E -t|u|f service-address [-s scheduler]
-t, -u: service-address
ip:port
-f: service-address
firewall mark
iptables的功能：
filter, nat, mangle, raw
mangle：
target: MARK
–set-mark value[/mask]
基于fwm定義集群服務的步驟：
(1) 打標
# iptables -t mangle -A PREROUTING -d $vip -p $protocol –dport $serviceport -j MARK –set-mark #
(2) 定義集群服務
# ipvsadm -A -f # [-s scheduler]
實驗拓撲：
DS地址：192.168.150.137 DIP、192.168.150.131 VIP
RS地址：192.168.150.138 RS1、192.168.150.139 RS1
CLIENT:192.168.150.133
LVS-fwm測試
DS配置（RS配置同LVS-dr配置）
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp –dport 80 -j MARK –s et-mark 1 iptables上添加MARK規則，規則在PREROUTING上設置，設定在VIP上面
[root@localhost ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 12 packets, 792 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK tcp — * * 0.0.0.0/0 192.168.150.131 tcp dpt:80 M
ARK set 0x1
Chain INPUT (policy ACCEPT 12 packets, 792 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 7 packets, 1284 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 7 packets, 1284 bytes)
pkts bytes target prot opt in out source destination
[root@localhost ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 20 packets, 1284 bytes)
pkts bytes target prot opt in out source destination
3 152 MARK tcp — * * 0.0.0.0/0 192.168.150.131 tcp dpt:80 M 當有客戶端請求時，iptables開始打mark
ARK set 0x1
Chain INPUT (policy ACCEPT 20 packets, 1284 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 15 packets, 2656 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 15 packets, 2656 bytes)
pkts bytes target prot opt in out source destination
[root@localhost ~]# ipvsadm -A -f 1 -s rr 添加LVS集群服務 -f即為fwm -s調度方法為rr輪詢
[root@localhost ~]# ipvsadm -a -f 1 -r 192.168.150.138 -g -w 1 管理集群服務上的RS -g為gateway默認模式
[root@localhost ~]# ipvsadm -a -f 1 -r 192.168.150.139 -g -w 1
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 rr
-> 192.168.150.138:0 Route 1 0 0
-> 192.168.150.139:0 Route 1 0 0
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 rr
-> 192.168.150.138:0 Route 1 0 0
-> 192.168.150.139:0 Route 1 0 0
此時的fwm已經配置完成并生效
客戶端訪問結果：
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS1</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS1</h1>
2、LVS persistent持久連接
功能：無論ipvs使用何種scheduler，其都能夠實現在指定時間范圍內始終將來自同一個ip地址的請求發往同一個RS；此功能是通過lvs持久連接模板實現，其與調度方法無關；
ipvs持久連接的模式：
每端口持久（PPC）:對某個端口進行持久
每客戶端持久（PCC）：定義tcp或udp協議的0號端口為集群服務端口；
每FWM持久（PFWMC）：對防火墻編輯下的端口進行持久
http, https
定義持久連接服務的方法：
ipvsadm -A|E -t|u|f service-address [-s scheduler]
[-p [timeout]]
實驗拓撲：
DS地址：192.168.150.137 DIP、192.168.150.131 VIP
RS地址：192.168.150.138 RS1、192.168.150.139 RS1
CLIENT:192.168.150.133
1、每客戶端持久（PCC）：定義tcp或udp協議的0號端口為集群服務端口；
[root@localhost ~]# ipvsadm -E -f 1 -s rr -p 300 此時將此前的LVS集群服務加上-p進行持久
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 rr persistent 300
-> 192.168.150.138:0 Route 1 0 0 添加時ip地址不加端口號指定默認為0，代表沒客戶端持久連接，客戶端第一次連入此RS后，后面連接均在此RS上
-> 192.168.150.139:0 Route 1 0 0
客戶端測試結果，調度算法無效果：
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
DS端也可以對ip進行:0指定進行設置
[root@localhost ~]# ipvsadm -A -t 192.168.150.131:0 -s rr -p
[root@localhost ~]# ipvsadm -a -t 192.168.150.131:0 -r 192.168.150.138 -g -w 1
[root@localhost ~]# ipvsadm -a -t 192.168.150.131:0 -r 192.168.150.139 -g -w 1
2、每端口持久（PPC）:對某個端口進行持久
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp –dport 80 -j MARK –s et-mark 1
[root@localhost ~]# ipvsadm -A -t 192.168.150.131:23 -s rr -p 300 加入-p選項時即表明開啟持久連接，此時持久連接為300s，地址后面加端口號表示僅對某個端口進行持久
[root@localhost ~]# ipvsadm -a -t 192.168.150.131:23 -r 192.168.150.138 -g -w 1
[root@localhost ~]# ipvsadm -a -t 192.168.150.131:23 -r 192.168.150.139 -g -w 1
RS端開啟telnet服務1和2均開啟
[root@localhost ~]# systemctl start telnet.socket
[root@localhost ~]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::80 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 :::23 :::*
LISTEN 0 100 ::1:25 :::*
客戶端測試結果：
[root@localhost ~]# ssh 192.168.150.131 ssh訪問會持久在192.168.150.139這臺RS上
The authenticity of host '192.168.150.131 (192.168.150.131)' can't be established.
RSA key fingerprint is 22:fc:db:5b:e5:26:8a:35:96:9f:2d:c4:4f:07:d1:e8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.150.131' (RSA) to the list of known hosts.
root@192.168.150.131's password:
Permission denied, please try again.
root@192.168.150.131's password:
Last failed login: Thu Dec 1 01:23:09 CST 2016 from 192.168.150.133 on ssh:notty
There were 2 failed login attempts since the last successful login.
Last login: Thu Dec 1 00:34:06 2016 from 192.168.150.1
[root@localhost ~]# ifconfig
eno33554976: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.150.139 netmask 255.255.255.0 broadcast 192.168.150.255
inet6 fe80::20c:29ff:fe7c:2ca9 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:7c:2c:a9 txqueuelen 1000 (Ethernet)
RX packets 8532 bytes 8978251 (8.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3594 bytes 320788 (313.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 20 bytes 1760 (1.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20 bytes 1760 (1.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.150.131 netmask 255.255.255.255
loop txqueuelen 0 (Local Loopback)
3、對于多端口的持久可以通過使用FWM中的防火墻下的端口進行持久
測試：進行http和https的端口持久
DS主機配置：
情況規則
root@localhost ~]# ipvsadm -C
創建私有CA
[root@localhost ~]# cd /etc/pki/CA/
[root@localhost CA]# (umask 077;openssl genrsa -out private/cakey.pem 2048)
Generating RSA private key, 2048 bit long modulus
…………………………………………+++
……..+++
e is 65537 (0x10001)
[root@localhost CA]# ls -l private/
總用量 4
-rw——- 1 root root 1679 12月 1 01:28 cakey.pem
[root@localhost CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 365
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
—–
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:SZ
Locality Name (eg, city) [Default City]:SZ
Organization Name (eg, company) [Default Company Ltd]:CPTW
Organizational Unit Name (eg, section) []:OPS
Common Name (eg, your name or your server's hostname) []:CPTW.COM.CN
Email Address []:MAIL.CPTW.COM.CN
[root@localhost CA]# touch index.txt
[root@localhost CA]# echo 01 > serial
RS1配置
包安裝，RS2上同步安裝
[root@localhost ~]# yum -y install mod_ssl
已加載插件：fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* extras: mirrors.cn99.com
* updates: mirrors.cn99.com
正在解決依賴關系
–> 正在檢查事務
—> 軟件包 mod_ssl.x86_64.1.2.4.6-40.el7.centos.4 將被 安裝
–> 解決依賴關系完成
依賴關系解決
=====================================================================================================
Package 架構 版本 源 大小
=====================================================================================================
正在安裝:
mod_ssl x86_64 1:2.4.6-40.el7.centos.4 updates 104 k
事務概要
=====================================================================================================
安裝 1 軟件包
總下載量：104 k
安裝大小：224 k
Downloading packages:
mod_ssl-2.4.6-40.el7.centos.4.x86_64.rpm | 104 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安裝 : 1:mod_ssl-2.4.6-40.el7.centos.4.x86_64 1/1
驗證中 : 1:mod_ssl-2.4.6-40.el7.centos.4.x86_64 1/1
已安裝:
mod_ssl.x86_64 1:2.4.6-40.el7.centos.4
完畢！
秘鑰加密
[root@localhost ~]# cd /etc/httpd/
[root@localhost httpd]# ls
conf conf.d conf.modules.d logs modules run
[root@localhost httpd]# mkdir ssl
[root@localhost httpd]# cd ssl
[root@localhost ssl]# ls
[root@localhost ssl]# (umask 077;openssl genrsa -out httpd.key 1024)
Generating RSA private key, 1024 bit long modulus
……..++++++
……..++++++
e is 65537 (0x10001)
[root@localhost ssl]# ls
httpd.key
證書請求
[root@localhost ssl]# openssl req -new -key httpd.key -out httpd.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
—–
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:SZ
Locality Name (eg, city) [Default City]:SZ
Organization Name (eg, company) [Default Company Ltd]:CPTW
Organizational Unit Name (eg, section) []:OPS
Common Name (eg, your name or your server's hostname) []:WWW.CPTW.COM.CN
Email Address []:MAIL.CPTW.COM.CN
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@localhost ssl]# ls
httpd.csr httpd.key root@192.168.150.137
將證書請求發送至頒發證書機構進行簽證
[root@localhost ssl]# scp httpd.csr root@192.168.150.137:/tmp
root@192.168.150.137's password:
httpd.csr
DS上進行證書簽證并回拷至RS1
[root@localhost tmp]# ls
httpd.crt httpd.csr ipvsadm-config.v1
[root@localhost tmp]# openssl ca -in /tmp/httpd.csr -out /tmp/httpd.crt -days 365
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Nov 30 18:08:27 2016 GMT
Not After : Nov 30 18:08:27 2017 GMT
Subject:
countryName = CN
stateOrProvinceName = SZ
organizationName = CPTW
organizationalUnitName = OPS
commonName = WWW.CPTW.COM.CN
emailAddress = MAIL.CPTW.COM.CN
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
C2:C2:F3:E3:91:CC:82:96:B2:38:CB:23:84:F4:4F:93:FF:32:FC:BD
X509v3 Authority Key Identifier:
keyid:F0:26:D0:A8:94:A9:81:E2:C5:C0:5A:95:5B:D3:1B:BB:BB:28:59:87
Certificate is to be certified until Nov 30 18:08:27 2017 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
[root@localhost tmp]# ls
httpd.crt httpd.csr ipvsadm-config.v1
[root@localhost tmp]# scp httpd.crt root@192.168.150.138:/etc/httpd/ssl
root@192.168.150.138's password:
httpd.crt
RS1修改httpd相關配置，并將證書和配置拷貝至RS2
[root@localhost httpd]# vim conf.d/ssl.conf
SSLCertificateFile /etc/httpd/ssl/httpd.crt
SSLCertificateKeyFile /etc/httpd/ssl/httpd.key
[root@localhost ssl]# scp -rp /etc/httpd/ssl/ root@192.168.150.139:/etc/httpd/
root@192.168.150.139's password:
httpd.key 100% 887 0.9KB/s 00:00
root@192.168.150.137 100% 676 0.7KB/s 00:00
httpd.csr 100% 684 0.7KB/s 00:00
httpd.crt 100% 3808 3.7KB/s 00:00
[root@localhost ssl]# scp /etc/httpd/conf.d/ssl.conf root@192.168.150.139:/etc/httpd/conf.d/ssl.conf
root@192.168.150.139's password:
ssl.conf
兩臺RS均進行httpd的重啟并查看443是否已正常開啟
[root@localhost httpd]# systemctl start httpd.service
[root@localhost httpd]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::80 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 :::23 :::*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 128 :::443 :::*
DS主機配置
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp -m multiport –dports 80,443 -j MARK –set-mark 10 添加了80和443兩個目標端口并mark
[root@localhost ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 135 packets, 11432 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK tcp — * * 0.0.0.0/0 192.168.150.131 multiport dp
orts 80,443 MARK set 0xa
Chain INPUT (policy ACCEPT 135 packets, 11432 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 117 packets, 11516 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 117 packets, 11516 bytes)
pkts bytes target prot opt in out source destination
客戶端進行請求
[root@localhost ~]# curl http://192.168.150.131
curl: (7) couldn't connect to host
[root@localhost ~]# curl https://192.168.150.131
curl: (7) couldn't connect to host
[root@localhost ~]# curl https://192.168.150.138
curl: (60) Peer certificate cannot be authenticated with known CA certificates
此時DS上狀態，已經有包請求并以mark
[root@localhost ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 142 packets, 11892 bytes)
pkts bytes target prot opt in out source destination
2 120 MARK tcp — * * 0.0.0.0/0 192.168.150.131 multiport dp
orts 80,443 MARK set 0xa
Chain INPUT (policy ACCEPT 142 packets, 11892 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 124 packets, 12864 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 124 packets, 12864 bytes)
pkts bytes target prot opt in out source destination
DS上添加持久
[root@localhost ~]# ipvsadm -A -f 10 -s rr -p
[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.150.138 -g -w 1
[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.150.139 -g -w 2
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 10 rr persistent 360
-> 192.168.150.138:0 Route 1 0 0
-> 192.168.150.139:0 Route 2 0 0
客戶端執行結果：此時客戶端訪問的443和80端口均會綁定持久在同一臺RS上
[root@localhost ~]# curl http://192.168.150.131
<h1>RS1</h1>
[root@localhost ~]# curl -k https://192.168.150.131
<h1>RS1</h1>
原創文章，作者：N23-蘇州-void，如若轉載，請注明出處：http://www.www58058.com/61846
總結的比較詳細，能給出物理拓撲和實驗驗證的例子會更好~~繼續加油~