1. 机器列表

主机名 IP docker0 IP docker容器IP

node101 192.168.80.101 10.1.1.1/24 10.1.1.2/24

node102 192.168.80.102 20.1.1.1/24 20.1.1.2/24

 

2. 网络示意图

3. rpm制作

mkdir -p ~/rpmbuild/SOURCES

cp openvswitch-2.5.0.tar.gz ~/rpmbuild/SOURCES

使用源码包中的SPEC文件制作rpm安装包

tar -xf openvswitch-2.5.0.tar.gz

rpmbuild -bb openvswitch-2.5.0/rhel/openvswitch.spec

4. node101

==================

4.1 安装制作的rpm包

yum localinstall ~/rpmbuild/RPMS/x86_64/openvswitch-2.5.0-1.x86_64.rpm

 

4.2 启动openswitch

/etc/init.d/openvswitch start

/etc/init.d/openvswitch status

检查日志输出

tail -100 /var/log/messages

4.3 ovs配置

创建网桥br0

ovs-vsctl add-br br0

把网络设备gre1添加到网桥br0

ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre option:remote_ip=192.168.80.102

添加br0到本地docker0，使得容器流量通过OVS流经tunnel

brctl addif docker0 br0

修改网络设备状态为up

ip link set dev br0 up

ip link set docker0 up

查看网桥和ovs接口

brctl show

ovs-vsctl list-br

ovs-vsctl list-ifaces br0

ovs-vsctl list-ports br0

4.4 防火墙放行icmp

iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited

iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited

4.5 添加到node102主机的docker路由

route add -net 20.1.1.0/24 gw 192.168.80.102

4.6 启动容器

docker run -it mysql bash

查看该容器ip地址

ip addr

 

5. node102

==================

5.1 安装制作的rpm包

yum localinstall ~/rpmbuild/RPMS/x86_64/openvswitch-2.5.0-1.x86_64.rpm

5.2 启动openswitch

/etc/init.d/openvswitch start

/etc/init.d/openvswitch status

检查日志输出

tail -100 /var/log/messages

5.3 OVS配置

创建网桥br0

ovs-vsctl add-br br0

把网络设备gre1添加到网桥br0

ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre option:remote_ip=192.168.80.101

添加br0到本地docker0，使得容器流量通过OVS流经tunnel

brctl addif docker0 br0

修改网络设备状态为up

ip link set dev br0 up

ip link set docker0 up

查看网桥和ovs接口

brctl show

ovs-vsctl list-br

ovs-vsctl list-ifaces br0

ovs-vsctl list-ports br0

5.4 防火墙放行icmp

iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited

iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited

5.5 添加到node102主机的docker路由

route add -net 10.1.1.0/24 gw 192.168.80.101

5.6 启动容器

docker run -it mysql bash

查看该容器ip地址

ip addr

 

6. 抓包分析

在node101中的docker实例中pingnode102的docker实例IP，抓包分析OVS数据流向

node101

==================

[veth2a3e623] 04:03:59.861136 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.1.1.2 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[veth2a3e623] 04:03:59.861986 IP (tos 0x0, ttl 62, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 10.1.1.2: ICMP echo reply, id 24, seq 0, length 64

[docker0] 04:03:59.861136 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.1.1.2 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[docker0] 04:03:59.861979 IP (tos 0x0, ttl 62, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 10.1.1.2: ICMP echo reply, id 24, seq 0, length 64

[eno16777728] 04:03:59.861185 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[eno16777728] 04:03:59.861946 IP (tos 0x0, ttl 63, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

OUT方向按时间排序

[veth2a3e623] 04:03:59.861136 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.1.1.2 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[docker0] 04:03:59.861136 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.1.1.2 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[eno16777728] 04:03:59.861185 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

IN方向按时间排序

[eno16777728] 04:03:59.861946 IP (tos 0x0, ttl 63, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

[docker0] 04:03:59.861979 IP (tos 0x0, ttl 62, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 10.1.1.2: ICMP echo reply, id 24, seq 0, length 64

[veth2a3e623] 04:03:59.861986 IP (tos 0x0, ttl 62, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 10.1.1.2: ICMP echo reply, id 24, seq 0, length 64

node102

==================

[veth8198030] 04:03:59.043575 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[veth8198030] 04:03:59.043621 IP (tos 0x0, ttl 64, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

[docker0] 04:03:59.043565 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[docker0] 04:03:59.043621 IP (tos 0x0, ttl 64, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

[eno16777728] 04:03:59.043509 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[eno16777728] 04:03:59.043634 IP (tos 0x0, ttl 63, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

IN方向按时间排序

[eno16777728] 04:03:59.043509 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[docker0] 04:03:59.043565 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

[veth8198030] 04:03:59.043575 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

OUT方向按时间排序

[veth8198030] 04:03:59.043621 IP (tos 0x0, ttl 64, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

[docker0] 04:03:59.043621 IP (tos 0x0, ttl 64, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

[eno16777728] 04:03:59.043634 IP (tos 0x0, ttl 63, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

 

br0抓包无数据，gre通道是虚拟的，实际还是从物理网卡传输