Kubernetes网络分析中如何实现Container间通信

这篇文章主要介绍Kubernetes网络分析中如何实现Container间通信，文中介绍的非常详细，具有一定的参考价值，感兴趣的小伙伴们一定要看完！

1. 引言##

上图显示了Kubernetes的基本结构图。

• Master管理多个Slave节点

• Slave节点上面可以运行多个Pod

• Pod可以部署多个副本，多个副本可以运行在不同的Node上

• 一个Pod可以包含多个Container，一个Pod内的Container共享同样的网络地址空间

最重要的是最后一句话：一个Pod内的Container共享同样的网络地址空间。这是通过Mapped Container做到的。

2. Mapped Container##

基本描述为下：

• 容器 A 的网络模式为正常docker的网络模式

• 容器 B 的网络模式为应用容器A的网络模式

###2.1 共享网络模式###

下面是一个例子来验证，我这里创建了一个busybox的Pod。

[root@centos7-node-221 ~]$ kubectl get poNAME      READY     STATUS    RESTARTS   AGEbusybox   1/1       Running   224        9d[root@centos7-node-221 ~]$ kubectl describe po busyboxName:                           busyboxNamespace:                      defaultImage(s):                       busyboxNode:                           centos7-node-226/192.168.1.226Labels:                         Status:                         RunningReason:                         Message:                        IP:                             172.16.58.6Replication Controllers:   Containers:  busybox:    Image:                  busybox    State:                  Running      Started:                    Thu, 08 Oct 2015 08:20:30 -0400    Last Termination State:       Terminated      Exit Code:             0      Started:                    Thu, 08 Oct 2015 07:20:26 -0400      Finished:                   Thu, 08 Oct 2015 08:20:26 -0400    Ready:                  True    Restart Count:             224    Variables:Conditions:  Type                Status  Ready  True Volumes:  default-token-lv94w:    Type:   Secret (a secret that should populate this volume)    SecretName:     default-token-lv94wEvents:  FirstSeen   LastSeen        Count   From                            SubobjectPath                   Reason  Message  9d          37m             225     {kubelet centos7-node-226} spec.containers{busybox}        pulled  Container image "busybox" already present on machine  37m         37m             1       {kubelet centos7-node-226} spec.containers{busybox}        Created Created with docker id fc8580292210  37m         37m             1       {kubelet centos7-node-226} spec.containers{busybox}        Started Started with docker id fc8580292210

我们去192.168.1.226看下这个Pod和其Container.

[root@centos7-node-226 ~]$ docker ps | grep busyboxfc8580292210        busybox                                               "sleep 3600"        37 minutes ago      Up 37 minutes                           k8s_busybox.62fa0587_busybox_default_86e98e8c-665f-11e5-af98-525400d7abb6_7f734c4d                                                02d259dc8ab5        gcr.io/google_containers/pause:0.8.0                  "/pause"            9 days ago          Up 9 days                               k8s_POD.7be6d81d_busybox_default_86e98e8c-665f-11e5-af98-525400d7abb6_ff9224f5

发现有两个容器，一个是pause容器，一个是busybox容器。其中pause容器为主网络容器，其他容器都共享pause容器的网络模式。我们分别看下其网络模式。下面是两个容器的网络模式。

[root@centos7-node-226 ~]$ docker inspect 02d259dc8ab5  | grep NetworkMode        "NetworkMode": "bridge",[root@centos7-node-226 ~]$ docker inspect fc8580292210 | grep NetworkMode        "NetworkMode": "container:02d259dc8ab59c1746d54d2df24d8733b2b9379a9fdfbfdc2066429b4a934a04", # 这个container的id号码就是上一个container的id的long形式

所以可以看到fc8580292210(busybox)使用的是pause容器的网络空间。

让我们进一步验证。

2.2 IP地址和hostname、网络IO###

下面我在 192.168.1.224 搭建了一个dns的pod，里面有4个容器，共享一个网络空间，我们采用查看其ip地址、hostname和网络IO的方式来鉴定。 下面是容器的id号

[root@centos7-node-224 ~]$ docker ps | grep dnsb00a08d078d6        dockerimages.yinnut.com:15043/skydns:2015-03-11-001   "/skydns -machines=h   8 hours ago         Up 8 hours                              k8s_skydns.c878079e_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_46f95e60                                   4e843585b938        dockerimages.yinnut.com:15043/exechealthz:1.0         "/exechealthz '-cmd=   11 days ago         Up 11 days                              k8s_healthz.8ab20f84_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_f7c469e5                                  296ff779abb2        dockerimages.yinnut.com:15043/kube2sky:1.11           "/kube2sky -domain=c   11 days ago         Up 11 days                              k8s_kube2sky.2a46d768_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_349c7246                                 f0118fac6952        dockerimages.yinnut.com:15043/etcd:2.0.9              "/usr/local/bin/etcd   11 days ago         Up 11 days                              k8s_etcd.64e02c2f_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_9235054b                                     f281dbf1ec41        gcr.io/google_containers/pause:0.8.0                  "/pause"               11 days ago         Up 11 days                              k8s_POD.6e934112_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_a8ea96d0

我们查看前三个 b00a08d078d6 4e843585b938 296ff779abb2 的上述属性。

2.2.1 dns设置和hostname####

[root@centos7-node-224 ~]$ for id in b00a08d078d6 4e843585b938 296ff779abb2 ; do echo $id; docker exec $id cat /etc/hosts ; docker exec $id cat /etc/resolv.conf ; echo  "" ; doneb00a08d078d6172.16.60.4     kube-dns-v9-y05vd127.0.0.1       localhost::1     localhost ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allroutersnameserver 192.168.1.208search 8.8.8.8options ndots:54e843585b938172.16.60.4     kube-dns-v9-y05vd127.0.0.1       localhost::1     localhost ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allroutersnameserver 192.168.1.208search 8.8.8.8options ndots:5296ff779abb2172.16.60.4     kube-dns-v9-y05vd127.0.0.1       localhost::1     localhost ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allroutersnameserver 192.168.1.208search 8.8.8.8options ndots:5

2.2.2 IP地址####

[root@centos7-node-224 ~]$ for id in b00a08d078d6 4e843585b938 296ff779abb2 ; do echo $id; docker exec $id ip a  ; echo  "" ; doneb00a08d078d61: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever4e843585b9381: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever296ff779abb21: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever

2.2.3 网络连接IO####

[root@centos7-node-224 ~]$ for id in b00a08d078d6 4e843585b938 296ff779abb2 ; do echo $id; docker exec $id netstat -lan  ; echo  "" ; doneb00a08d078d6Active Internet connections (servers and established)Proto Recv-Q Send-Q Local Address           Foreign Address         State       tcp        0      0 127.0.0.1:4001          0.0.0.0:*               LISTEN      tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      tcp        0      0 127.0.0.1:7001          0.0.0.0:*               LISTEN      tcp        0      0 172.16.60.4:48582       10.254.0.1:443          ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:35394         ESTABLISHED tcp        0      0 172.16.60.4:48584       10.254.0.1:443          ESTABLISHED tcp        0      0 127.0.0.1:60161         127.0.0.1:2379          ESTABLISHED tcp        0      0 127.0.0.1:51445         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:51445         ESTABLISHED tcp        0      0 127.0.0.1:35550         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:35394         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:2379          127.0.0.1:60161         ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:51433         ESTABLISHED tcp        0      0 127.0.0.1:51433         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:35550         ESTABLISHED Active UNIX domain sockets (servers and established)Proto RefCnt Flags       Type       State         I-Node Path4e843585b938Active Internet connections (servers and established)Proto Recv-Q Send-Q Local Address           Foreign Address         State       tcp        0      0 127.0.0.1:4001          0.0.0.0:*               LISTEN      tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      tcp        0      0 127.0.0.1:7001          0.0.0.0:*               LISTEN      tcp        0      0 172.16.60.4:48582       10.254.0.1:443          ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:35394         ESTABLISHED tcp        0      0 172.16.60.4:48584       10.254.0.1:443          ESTABLISHED tcp        0      0 127.0.0.1:60161         127.0.0.1:2379          ESTABLISHED tcp        0      0 127.0.0.1:51445         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:51445         ESTABLISHED tcp        0      0 127.0.0.1:35550         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:35394         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:2379          127.0.0.1:60161         ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:51433         ESTABLISHED tcp        0      0 127.0.0.1:51433         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:35550         ESTABLISHED Active UNIX domain sockets (servers and established)Proto RefCnt Flags       Type       State         I-Node Path296ff779abb2Active Internet connections (servers and established)Proto Recv-Q Send-Q Local Address           Foreign Address         State       tcp        0      0 127.0.0.1:4001          0.0.0.0:*               LISTEN      tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      tcp        0      0 127.0.0.1:7001          0.0.0.0:*               LISTEN      tcp        0      0 172.16.60.4:48582       10.254.0.1:443          ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:35394         ESTABLISHED tcp        0      0 172.16.60.4:48584       10.254.0.1:443          ESTABLISHED tcp        0      0 127.0.0.1:60161         127.0.0.1:2379          ESTABLISHED tcp        0      0 127.0.0.1:51445         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:51445         ESTABLISHED tcp        0      0 127.0.0.1:35550         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:35394         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:2379          127.0.0.1:60161         ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:51433         ESTABLISHED tcp        0      0 127.0.0.1:51433         127.0.0.1:4001          ESTABLISHED tcp        0      0 127.0.0.1:4001          127.0.0.1:35550         ESTABLISHED Active UNIX domain sockets (servers and established)Proto RefCnt Flags       Type       State         I-Node Path

3. Container 之通信##

下面分析下最复杂的Container之间的通信。

3.1 Pod 之内 Container 通信

先说最简单的, Pod内的Container通信，由于共享网络地址空间，直接访问127.0.0.1即可。

3.2 跨机器之间Contianer通信###

3.2.1 背景引导####

例子： 192.168.1.224的fluentd-elasticsearch容器要连接192.168.1.223的elasticsearch-logging容器。

• 192.168.1.223的elasticsearch-logging容器及其IP地址:

[root@centos7-node-223 ~]$ docker ps |grep elasticsearch-logging667cfd84c979        dockerimages.yinnut.com:15043/elasticsearch:1.7       "/run.sh"              12 days ago          Up 12 days                              k8s_elasticsearch-logging.89fda9f_elasticsearch-logging-v1-i8x6q_kube-system_8b558d2c-62a3-11e5-9d7b-525400d7abb6_2a02a2c8        5201c8cbdebd        gcr.io/google_containers/pause:0.8.0                  "/pause"               12 days ago          Up 12 days                              k8s_POD.8ecd2043_elasticsearch-logging-v1-i8x6q_kube-system_8b558d2c-62a3-11e5-9d7b-525400d7abb6_4022db35                         [root@centos7-node-223 ~]$ docker inspect 5201c8cbdebd |grep IPAddress        "IPAddress": "172.16.77.4", #IP地址        "SecondaryIPAddresses": null,

可以看到elasticsearch-logging的容器的Pod的IP地址为172.16.77.4

• 192.168.1.223的fluentd-elasticsearch容器及其IP地址:

[root@centos7-node-224 ~]$ docker ps |grep fluentd-elasticsearchd326d81468b5        gcr.io/google_containers/fluentd-elasticsearch:1.11   "td-agent -q"          12 days ago         Up 12 days                              k8s_fluentd-elasticsearch.27a08aa3_fluentd-elasticsearch-centos7-node-224_kube-system_7dcc6ce562f3742190a876fda85e2359_58c54ef3   f9b76639d241        gcr.io/google_containers/pause:0.8.0                  "/pause"               12 days ago         Up 12 days                              k8s_POD.7be6d81d_fluentd-elasticsearch-centos7-node-224_kube-system_7dcc6ce562f3742190a876fda85e2359_333e52c0                     [root@centos7-node-224 ~]$ docker inspect f9b76639d241 | grep IPAddress        "IPAddress": "172.16.60.2",        "SecondaryIPAddresses": null,

可以看到fluentd-elasticsearch的容器的Pod的IP地址为172.16.60.2

• 我们看下 fluentd-elasticsearch 的网络连接情况

[root@centos7-node-224 ~]$ docker exec d326d81468b5 netstat -nla | grep 172.16.77.4tcp        0      0 172.16.60.2:56354       172.16.77.4:9200        TIME_WAIT  tcp        0      0 172.16.60.2:56350       172.16.77.4:9200        TIME_WAIT  tcp        0      0 172.16.60.2:56347       172.16.77.4:9200        TIME_WAIT  tcp        0      0 172.16.60.2:56357       172.16.77.4:9200        TIME_WAIT  tcp        0      0 172.16.60.2:56344       172.16.77.4:9200        TIME_WAIT  tcp        0      0 172.16.60.2:56352       172.16.77.4:9200        TIME_WAIT

可以看到其的确是连接了 172.16.77.4 的9200端口。 而对方 elasticsearch-logging 容器的确开启了9200端口

[root@centos7-node-223 ~]$ docker exec  667cfd84c979  ss -l|grep LISTENtcp    LISTEN     0      50                  :::9200                 :::*       tcp    LISTEN     0      50                  :::9300                 :::*

那么这个过程是如何完成的呢？？？

3.2.2 Container间通信流程####

192.168.1.224/fluentd-elasticsearch -> 192.168.1.223/elasticsearch-logging

192.168.1.224/fluentd-elasticsearch 需要连接到elasticsearch-logging容器.

• 域名到IP对应。 elasticsearch-logging -> 解析为10.254.24.205

root@fluentd-elasticsearch-centos7-node-224:/$ dig elasticsearch-logging; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> elasticsearch-logging;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 39181;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0;; QUESTION SECTION:;elasticsearch-logging.         IN      A;; Query time: 1 msec;; SERVER: 10.254.0.10#53(10.254.0.10);; WHEN: Fri Oct 09 06:04:35 UTC 2015;; MSG SIZE  rcvd: 39

• 访问该IP地址10.254.24.205:9200端口。根据路由，请求将会到达网关172.16.60.1,也就是这个docker的宿主机的docker0网卡地址。

# 容器内root@fluentd-elasticsearch-centos7-node-224:/$ ip routedefault via 172.16.60.1 dev eth0 172.16.60.0/24 dev eth0  proto kernel  scope link  src 172.16.60.2 # 物理机[root@centos7-node-224 ~]$ ifconfig docker0docker0: flags=4163  mtu 1450        inet 172.16.60.1  netmask 255.255.255.0  broadcast 0.0.0.0        inet6 fe80::5484:7aff:fefe:9799  prefixlen 64  scopeid 0x20        ether 56:84:7a:fe:97:99  txqueuelen 0  (Ethernet)        RX packets 10182154  bytes 1777103288 (1.6 GiB)        RX errors 0  dropped 0  overruns 0  frame 0        TX packets 11195534  bytes 2271907616 (2.1 GiB)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

• iptables负责转发请求到192.168.1.224:36967，而这个端口上kube-proxy进程在监听。

[root@centos7-node-224 ~]$ iptables-save  | grep 10.254.24.205 |grep 9200-A KUBE-PORTALS-CONTAINER -d 10.254.24.205/32 -p tcp -m comment --comment "kube-system/elasticsearch-logging:" -m tcp --dport 9200 -j REDIRECT --to-ports 36967-A KUBE-PORTALS-HOST -d 10.254.24.205/32 -p tcp -m comment --comment "kube-system/elasticsearch-logging:" -m tcp --dport 9200 -j DNAT --to-destination 192.168.1.224:36967[root@centos7-node-224 ~]$ netstat -nlp|grep 36967tcp6       0      0 :::36967                :::*                    LISTEN      930/kube-proxy

• 谁负责响应10.254.24.205:9200的请求？由上述分析，看起来是kube-proxy，那么kube-proxy进程看起来是个proxy，那么被转发给谁处理？当然给Pod啦。可以到这个服务的Selector是k8s-app=elasticsearch-logging

[root@centos7-node-224 ~]$ kubectl get svc --all-namespaces  | grep 10.254.24.205kube-system   elasticsearch-logging   10.254.24.205            9200/TCP            k8s-app=elasticsearch-logging   14d

• 找到其对应的Pod为elasticsearch-logging-v1-gph5i和elasticsearch-logging-v1-i8x6q

[root@centos7-node-224 ~]$ kubectl get po -l k8s-app=elasticsearch-logging --all-namespacesNAMESPACE     NAME                             READY     STATUS    RESTARTS   AGEkube-system   elasticsearch-logging-v1-gph5i   1/1       Running   6          14dkube-system   elasticsearch-logging-v1-i8x6q   1/1       Running   5          14d

• 我们查看其中的elasticsearch-logging-v1-i8x6q容器的IP地址，发现为172.16.77.4

[root@centos7-node-224 ~]$ kubectl describe po elasticsearch-logging-v1-i8x6q --namespace=kube-system | grep IPIP:                             172.16.77.4

• 故而很明显kube-proxy 会把部分请求转发给 其中的一个Pod来处理，而这个Pod的IP地址是172.16.77.4 . 而 172.16.77.4 这个Pod 在 192.168.1.223 机器上.

[root@centos7-node-223 ~]$ docker inspect 5201c8cbdebd | grep IPAddress        "IPAddress": "172.16.77.4",        "SecondaryIPAddresses": null,

• 那如何与172.16.77.4进行通信呢？跨机器之间通信则采用flannel等诸如此类的overlay网络或者ovs等L2网络。
  

以上是"Kubernetes网络分析中如何实现Container间通信"这篇文章的所有内容，感谢各位的阅读！希望分享的内容对大家有帮助，更多相关知识，欢迎关注行业资讯频道！