LVS-fwm&persistence

1、LVS-fwm

fwm：FireWall Mark

       在netfilter上給報文打標記；mangle表；

       ipvsadm -A|E -t|u|f service-address [-s scheduler]

           -t, -u: service-address

               ip:port

           -f: service-address

               firewall mark

       iptables的功能：

           filter, nat, mangle, raw

           mangle：

               target: MARK

                   –set-mark value[/mask]

       基于fwm定義集群服務的步驟：

           (1) 打標

               # iptables -t mangle -A PREROUTING -d $vip -p $protocol –dport $serviceport -j MARK –set-mark #

           (2) 定義集群服務

               # ipvsadm -A -f # [-s scheduler]

實驗拓撲：

         DS地址：192.168.150.137 DIP、192.168.150.131 VIP

         RS地址：192.168.150.138 RS1、192.168.150.139 RS1

         CLIENT:192.168.150.133

LVS-fwm測試

DS配置（RS配置同LVS-dr配置）

[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp –dport 80 -j MARK –s et-mark 1         iptables上添加MARK規則，規則在PREROUTING上設置，設定在VIP上面

[root@localhost ~]# iptables -t mangle -vnL

Chain PREROUTING (policy ACCEPT 12 packets, 792 bytes)

pkts bytes target     prot opt in     out     source               destination

0     0 MARK       tcp  —  *      *       0.0.0.0/0            192.168.150.131      tcp dpt:80 M

ARK set 0x1

Chain INPUT (policy ACCEPT 12 packets, 792 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 7 packets, 1284 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 7 packets, 1284 bytes)

pkts bytes target     prot opt in     out     source               destination

[root@localhost ~]# iptables -t mangle -vnL

Chain PREROUTING (policy ACCEPT 20 packets, 1284 bytes)

pkts bytes target     prot opt in     out     source               destination

   3   152 MARK       tcp  —  *      *       0.0.0.0/0            192.168.150.131      tcp dpt:80 M          當有客戶端請求時，iptables開始打mark

ARK set 0x1

Chain INPUT (policy ACCEPT 20 packets, 1284 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 15 packets, 2656 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 15 packets, 2656 bytes)

pkts bytes target     prot opt in     out     source               destination

[root@localhost ~]# ipvsadm -A -f 1 -s rr          添加LVS集群服務 -f即為fwm -s調度方法為rr輪詢

[root@localhost ~]# ipvsadm -a -f 1 -r 192.168.150.138 -g -w 1      管理集群服務上的RS -g為gateway默認模式

[root@localhost ~]# ipvsadm -a -f 1 -r 192.168.150.139 -g -w 1

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port           Forward Weight ActiveConn InActConn

FWM  1 rr

-> 192.168.150.138:0            Route   1      0          0

-> 192.168.150.139:0            Route   1      0          0

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port           Forward Weight ActiveConn InActConn

FWM  1 rr

-> 192.168.150.138:0            Route   1      0          0

-> 192.168.150.139:0            Route   1      0          0

此時的fwm已經配置完成并生效

客戶端訪問結果：

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS1</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS1</h1>

2、LVS persistent持久連接

       功能：無論ipvs使用何種scheduler，其都能夠實現在指定時間范圍內始終將來自同一個ip地址的請求發往同一個RS；此功能是通過lvs持久連接模板實現，其與調度方法無關；

       ipvs持久連接的模式：

           每端口持久（PPC）:對某個端口進行持久

           每客戶端持久（PCC）：定義tcp或udp協議的0號端口為集群服務端口；

           每FWM持久（PFWMC）：對防火墻編輯下的端口進行持久

               http, https

       定義持久連接服務的方法：

           ipvsadm -A|E -t|u|f service-address [-s scheduler]

           [-p [timeout]]

實驗拓撲：

         DS地址：192.168.150.137 DIP、192.168.150.131 VIP

         RS地址：192.168.150.138 RS1、192.168.150.139 RS1

         CLIENT:192.168.150.133

1、每客戶端持久（PCC）：定義tcp或udp協議的0號端口為集群服務端口；

[root@localhost ~]# ipvsadm -E -f 1 -s rr -p 300     此時將此前的LVS集群服務加上-p進行持久

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port           Forward Weight ActiveConn InActConn

FWM  1 rr persistent 300

-> 192.168.150.138:0            Route   1      0          0          添加時ip地址不加端口號指定默認為0，代表沒客戶端持久連接，客戶端第一次連入此RS后，后面連接均在此RS上

-> 192.168.150.139:0            Route   1      0          0

客戶端測試結果，調度算法無效果：

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

DS端也可以對ip進行:0指定進行設置

[root@localhost ~]# ipvsadm -A -t 192.168.150.131:0 -s rr -p

[root@localhost ~]# ipvsadm -a -t 192.168.150.131:0 -r 192.168.150.138 -g -w 1

[root@localhost ~]# ipvsadm -a -t 192.168.150.131:0 -r 192.168.150.139 -g -w 1

2、每端口持久（PPC）:對某個端口進行持久

[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp –dport 80 -j MARK –s et-mark 1

[root@localhost ~]# ipvsadm -A -t 192.168.150.131:23 -s rr -p 300          加入-p選項時即表明開啟持久連接，此時持久連接為300s，地址后面加端口號表示僅對某個端口進行持久

[root@localhost ~]# ipvsadm -a -t 192.168.150.131:23 -r 192.168.150.138 -g -w 1

[root@localhost ~]# ipvsadm -a -t 192.168.150.131:23 -r 192.168.150.139 -g -w 1

RS端開啟telnet服務1和2均開啟

[root@localhost ~]# systemctl start telnet.socket

[root@localhost ~]# ss -tnl

State       Recv-Q Send-Q     Local Address:Port                    Peer Address:Port

LISTEN      0      128                    *:22                                 *:*

LISTEN      0      100            127.0.0.1:25                                 *:*

LISTEN      0      128                   :::80                                :::*

LISTEN      0      128                   :::22                                :::*

LISTEN      0      128                   :::23                                :::*

LISTEN      0      100                  ::1:25                                :::*

客戶端測試結果：

[root@localhost ~]# ssh 192.168.150.131     ssh訪問會持久在192.168.150.139這臺RS上

The authenticity of host '192.168.150.131 (192.168.150.131)' can't be established.

RSA key fingerprint is 22:fc:db:5b:e5:26:8a:35:96:9f:2d:c4:4f:07:d1:e8.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '192.168.150.131' (RSA) to the list of known hosts.

root@192.168.150.131's password:

Permission denied, please try again.

root@192.168.150.131's password:

Last failed login: Thu Dec  1 01:23:09 CST 2016 from 192.168.150.133 on ssh:notty

There were 2 failed login attempts since the last successful login.

Last login: Thu Dec  1 00:34:06 2016 from 192.168.150.1

[root@localhost ~]# ifconfig

eno33554976: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

inet 192.168.150.139  netmask 255.255.255.0  broadcast 192.168.150.255

inet6 fe80::20c:29ff:fe7c:2ca9  prefixlen 64  scopeid 0x20<link>

ether 00:0c:29:7c:2c:a9  txqueuelen 1000  (Ethernet)

RX packets 8532  bytes 8978251 (8.5 MiB)

RX errors 0  dropped 0  overruns 0  frame 0

TX packets 3594  bytes 320788 (313.2 KiB)

TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

inet 127.0.0.1  netmask 255.0.0.0

inet6 ::1  prefixlen 128  scopeid 0x10<host>

loop  txqueuelen 0  (Local Loopback)

RX packets 20  bytes 1760 (1.7 KiB)

RX errors 0  dropped 0  overruns 0  frame 0

TX packets 20  bytes 1760 (1.7 KiB)

TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo:0: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

inet 192.168.150.131  netmask 255.255.255.255

loop  txqueuelen 0  (Local Loopback)

3、對于多端口的持久可以通過使用FWM中的防火墻下的端口進行持久

測試：進行http和https的端口持久

DS主機配置：

情況規則

root@localhost ~]# ipvsadm -C

創建私有CA

[root@localhost ~]# cd /etc/pki/CA/

[root@localhost CA]# (umask 077;openssl genrsa -out private/cakey.pem 2048)

Generating RSA private key, 2048 bit long modulus

…………………………………………+++

……..+++

e is 65537 (0x10001)

[root@localhost CA]# ls -l private/

總用量 4

-rw——- 1 root root 1679 12月  1 01:28 cakey.pem

[root@localhost CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 365

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

—–

Country Name (2 letter code) [XX]:CN

State or Province Name (full name) []:SZ

Locality Name (eg, city) [Default City]:SZ

Organization Name (eg, company) [Default Company Ltd]:CPTW

Organizational Unit Name (eg, section) []:OPS

Common Name (eg, your name or your server's hostname) []:CPTW.COM.CN

Email Address []:MAIL.CPTW.COM.CN

[root@localhost CA]# touch index.txt

[root@localhost CA]# echo 01 > serial

RS1配置

包安裝，RS2上同步安裝

[root@localhost ~]# yum -y install mod_ssl

已加載插件：fastestmirror

Loading mirror speeds from cached hostfile

* base: mirrors.cn99.com

* extras: mirrors.cn99.com

* updates: mirrors.cn99.com

正在解決依賴關系

–> 正在檢查事務

—> 軟件包 mod_ssl.x86_64.1.2.4.6-40.el7.centos.4 將被 安裝

–> 解決依賴關系完成

依賴關系解決

=====================================================================================================

Package            架構              版本                                  源                  大小

=====================================================================================================

正在安裝:

mod_ssl            x86_64            1:2.4.6-40.el7.centos.4               updates            104 k

事務概要

=====================================================================================================

安裝  1 軟件包

總下載量：104 k

安裝大?。?24 k

Downloading packages:

mod_ssl-2.4.6-40.el7.centos.4.x86_64.rpm                                      | 104 kB  00:00:00

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

正在安裝    : 1:mod_ssl-2.4.6-40.el7.centos.4.x86_64                                           1/1

驗證中      : 1:mod_ssl-2.4.6-40.el7.centos.4.x86_64                                           1/1

已安裝:

mod_ssl.x86_64 1:2.4.6-40.el7.centos.4

完畢！

秘鑰加密

[root@localhost ~]# cd /etc/httpd/

[root@localhost httpd]# ls

conf  conf.d  conf.modules.d  logs  modules  run

[root@localhost httpd]# mkdir ssl

[root@localhost httpd]# cd ssl

[root@localhost ssl]# ls

[root@localhost ssl]# (umask 077;openssl genrsa -out httpd.key 1024)

Generating RSA private key, 1024 bit long modulus

……..++++++

……..++++++

e is 65537 (0x10001)

[root@localhost ssl]# ls

httpd.key

證書請求

[root@localhost ssl]# openssl req -new -key httpd.key -out httpd.csr

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

—–

Country Name (2 letter code) [XX]:CN

State or Province Name (full name) []:SZ

Locality Name (eg, city) [Default City]:SZ

Organization Name (eg, company) [Default Company Ltd]:CPTW

Organizational Unit Name (eg, section) []:OPS

Common Name (eg, your name or your server's hostname) []:WWW.CPTW.COM.CN

Email Address []:MAIL.CPTW.COM.CN

Please enter the following 'extra' attributes

to be sent with your certificate request

A challenge password []:

An optional company name []:

[root@localhost ssl]# ls

httpd.csr  httpd.key  root@192.168.150.137

將證書請求發送至頒發證書機構進行簽證

[root@localhost ssl]# scp httpd.csr root@192.168.150.137:/tmp

root@192.168.150.137's password:

httpd.csr

DS上進行證書簽證并回拷至RS1

[root@localhost tmp]# ls

httpd.crt  httpd.csr  ipvsadm-config.v1

[root@localhost tmp]# openssl ca -in /tmp/httpd.csr -out /tmp/httpd.crt -days 365

Using configuration from /etc/pki/tls/openssl.cnf

Check that the request matches the signature

Signature ok

Certificate Details:

Serial Number: 1 (0x1)

Validity

Not Before: Nov 30 18:08:27 2016 GMT

Not After : Nov 30 18:08:27 2017 GMT

Subject:

countryName               = CN

stateOrProvinceName       = SZ

organizationName          = CPTW

organizationalUnitName    = OPS

commonName                = WWW.CPTW.COM.CN

emailAddress              = MAIL.CPTW.COM.CN

X509v3 extensions:

X509v3 Basic Constraints:

CA:FALSE

Netscape Comment:

OpenSSL Generated Certificate

X509v3 Subject Key Identifier:

C2:C2:F3:E3:91:CC:82:96:B2:38:CB:23:84:F4:4F:93:FF:32:FC:BD

X509v3 Authority Key Identifier:

keyid:F0:26:D0:A8:94:A9:81:E2:C5:C0:5A:95:5B:D3:1B:BB:BB:28:59:87

Certificate is to be certified until Nov 30 18:08:27 2017 GMT (365 days)

Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y

Write out database with 1 new entries

Data Base Updated

[root@localhost tmp]# ls

httpd.crt  httpd.csr  ipvsadm-config.v1

[root@localhost tmp]# scp httpd.crt root@192.168.150.138:/etc/httpd/ssl

root@192.168.150.138's password:

httpd.crt

RS1修改httpd相關配置，并將證書和配置拷貝至RS2

[root@localhost httpd]# vim conf.d/ssl.conf

SSLCertificateFile /etc/httpd/ssl/httpd.crt

SSLCertificateKeyFile /etc/httpd/ssl/httpd.key

[root@localhost ssl]# scp -rp /etc/httpd/ssl/ root@192.168.150.139:/etc/httpd/

root@192.168.150.139's password:

httpd.key                                                                                                                                                    100%  887     0.9KB/s   00:00

root@192.168.150.137                                                                                                                                         100%  676     0.7KB/s   00:00

httpd.csr                                                                                                                                                    100%  684     0.7KB/s   00:00

httpd.crt                                                                                                                                                    100% 3808     3.7KB/s   00:00

[root@localhost ssl]# scp /etc/httpd/conf.d/ssl.conf root@192.168.150.139:/etc/httpd/conf.d/ssl.conf

root@192.168.150.139's password:

ssl.conf

兩臺RS均進行httpd的重啟并查看443是否已正常開啟

[root@localhost httpd]# systemctl start httpd.service

[root@localhost httpd]# ss -tnl

State       Recv-Q Send-Q                                                  Local Address:Port                                                                 Peer Address:Port

LISTEN      0      128                                                                 *:22                                                                              *:*

LISTEN      0      100                                                         127.0.0.1:25                                                                              *:*

LISTEN      0      128                                                                :::80                                                                             :::*

LISTEN      0      128                                                                :::22                                                                             :::*

LISTEN      0      128                                                                :::23                                                                             :::*

LISTEN      0      100                                                               ::1:25                                                                             :::*

LISTEN      0      128                                                                :::443                                                                            :::*

DS主機配置

[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp -m multiport –dports 80,443 -j MARK –set-mark 10     添加了80和443兩個目標端口并mark

[root@localhost ~]# iptables -t mangle -vnL

Chain PREROUTING (policy ACCEPT 135 packets, 11432 bytes)

pkts bytes target     prot opt in     out     source               destination

0     0 MARK       tcp  —  *      *       0.0.0.0/0            192.168.150.131      multiport dp

orts 80,443 MARK set 0xa

Chain INPUT (policy ACCEPT 135 packets, 11432 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 117 packets, 11516 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 117 packets, 11516 bytes)

pkts bytes target     prot opt in     out     source               destination

客戶端進行請求

[root@localhost ~]# curl http://192.168.150.131

curl: (7) couldn't connect to host

[root@localhost ~]# curl https://192.168.150.131

curl: (7) couldn't connect to host

[root@localhost ~]# curl https://192.168.150.138

curl: (60) Peer certificate cannot be authenticated with known CA certificates

此時DS上狀態，已經有包請求并以mark

[root@localhost ~]# iptables -t mangle -vnL

Chain PREROUTING (policy ACCEPT 142 packets, 11892 bytes)

pkts bytes target     prot opt in     out     source               destination

2   120 MARK       tcp  —  *      *       0.0.0.0/0            192.168.150.131      multiport dp

orts 80,443 MARK set 0xa

Chain INPUT (policy ACCEPT 142 packets, 11892 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 124 packets, 12864 bytes)

pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 124 packets, 12864 bytes)

pkts bytes target     prot opt in     out     source               destination

DS上添加持久

[root@localhost ~]# ipvsadm -A -f 10 -s rr -p

[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.150.138 -g -w 1

[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.150.139 -g -w 2

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port           Forward Weight ActiveConn InActConn

FWM  10 rr persistent 360

-> 192.168.150.138:0            Route   1      0          0

-> 192.168.150.139:0            Route   2      0          0

客戶端執行結果：此時客戶端訪問的443和80端口均會綁定持久在同一臺RS上

[root@localhost ~]# curl http://192.168.150.131

<h1>RS1</h1>

[root@localhost ~]# curl -k  https://192.168.150.131

<h1>RS1</h1>