抱歉,您的浏览器无法访问本站
本页面需要浏览器支持(启用)JavaScript
了解详情 >

[toc]

kubernetes 操作记录五

配置网络插件flannel

在Kubernetes集群中要解决四种通信的问题;
Kubernetes网络通信:
(1) 容器间通信: 同一个Pod内的多个容器间的通信,lo
(2) Pod通信: Pod IP <--> Pod IP
(3) Pod与 Service通信: PodIP <--> ClusterIP,不在同一网段,通过iptables规则实现通信
(4) Service 与集群外部客户端的通信;
CNI: (Container Network Interface)
flannel,calico,canel,kube-router …

    # kubectl get configmap kube-proxy -n kube-system -o yaml
    mode: "" # 改为ipvs 就可以了

解决方案:
虚拟网桥: 纯软件的方式,实现一个虚拟网卡接到网桥上去;
多路复用: MacVlAN 配置多个Mac物理地址,使得一个物理网卡,可以承载多个容器去使用;
硬件交换: SR-IOV 单根IO虚拟化;
kubelet, /etc/cin/net.d/

# cat /etc/cni/net.d/10-flannel.conflist
{
  "name": "cbr0",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

flannel:
支持多种后端
VxLAN
host-gw: Host Gateway

# kubectl  get daemonset -n kube-system
NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
kube-flannel-ds           3         3         3       3            3           beta.kubernetes.io/arch=amd64     31d
kube-flannel-ds-amd64     4         4         4       4            4           beta.kubernetes.io/arch=amd64     31d
kube-flannel-ds-arm       0         0         0       0            0           beta.kubernetes.io/arch=arm       31d
kube-flannel-ds-arm64     0         0         0       0            0           beta.kubernetes.io/arch=arm64     31d
kube-flannel-ds-ppc64le   0         0         0       0            0           beta.kubernetes.io/arch=ppc64le   31d
kube-flannel-ds-s390x     0         0         0       0            0           beta.kubernetes.io/arch=s390x     31d
kube-proxy                4         4         4       4            4           <none>                            31d
# kubectl  get pods -n kube-system -o wide | grep "flannel"
kube-flannel-ds-amd64-45rhc                  1/1     Running   2          31d    10.1.87.80     bj-zb-vm-ops-test5     <none>           <none>
kube-flannel-ds-amd64-4cs6r                  1/1     Running   0          31d    10.1.87.83     node03   <none>           <none>
kube-flannel-ds-amd64-bst7g                  1/1     Running   0          31d    10.1.87.81     node01   <none>           <none>
kube-flannel-ds-amd64-gqvz2                  1/1     Running   0          31d    10.1.87.82     node02   <none>           <none>
kube-flannel-ds-brzp7                        2/2     Running   0          31d    10.1.87.83     node03   <none>           <none>
kube-flannel-ds-khpr5                        2/2     Running   2          31d    10.1.87.82     node02   <none>           <none>
kube-flannel-ds-q6z65                        2/2     Running   0          31d    10.1.87.81     node01

kube-flannel-cfg 用来配置以上flannel Pod是如何运行的;

flannel的配置参数:
Network: flannel使用的CIDR格式的网络地址,用于为Pod的配置网络功能;
10.244.0.0/16 ->
master: 10.244.0.0/24
node01:10.244.1.0/24

node255: 10.244.255.0./24
SubnetLen: 把Network切分子网供各节点使用时,使用多长的掩码进行切分,默认为24位;
SubnetMin: 10.244.10.0/24 指定子网使用起始,从哪里开始;
SubnetMax: 10.244.10.0/24 指定子网使用最大限制;
Backend: 各Pod通信时使用什么方式进行通信: vxlan, host-gw , udp


测试

# kubectl get pods -o wide
# kubectl  get pods -o wide | grep "myapp-deploy"
myapp-deploy-675558bfc5-947g6    1/1     Running   0          4d1h    10.244.1.173   node01   <none>           <none>
myapp-deploy-675558bfc5-9xfdm    1/1     Running   0          4d6h    10.244.3.176   node03   <none>           <none>
myapp-deploy-675558bfc5-qhl4w    1/1     Running   0          5h41m   10.244.2.195   node02   <none>           <none>

接入一个容器ping另一容器的ip

# kubectl exec -it myapp-deploy-675558bfc5-947g6 -- /bin/sh
/ # ping 10.244.2.195
PING 10.244.2.195 (10.244.2.195): 56 data bytes
64 bytes from 10.244.2.195: seq=0 ttl=62 time=0.871 ms
64 bytes from 10.244.2.195: seq=1 ttl=62 time=0.925 ms
64 bytes from 10.244.2.195: seq=2 ttl=62 time=0.886 ms
64 bytes from 10.244.2.195: seq=3 ttl=62 time=1.171 ms

在容器所在的物理服务器上抓包,

# tcpdump -i cni0 -nn icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cni0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:52:45.457148 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 0, length 64
16:52:45.457743 IP 10.244.2.195 > 10.244.1.173: ICMP echo reply, id 3328, seq 0, length 64
16:52:46.457445 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 1, length 64
16:52:46.457948 IP 10.244.2.195 > 10.244.1.173: ICMP echo reply, id 3328, seq 1, length 64
16:52:47.457722 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 2, length 64
16:52:47.458313 IP 10.244.2.195 > 10.244.1.173: ICMP echo reply, id 3328, seq 2, length 64
16:52:48.458036 IP 10.244.1.173 > 10.244.2.195: ICMP echo request, id 3328, seq 3, length 64
# tcpdump -i flannel.1 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on flannel.1, link-type EN10MB (Ethernet), capture size 262144 bytes
17:01:13.757750 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 37, length 64
17:01:13.758396 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 2, seq 37, length 64
17:01:14.758058 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 38, length 64
17:01:14.758791 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 2, seq 38, length 64
17:01:15.758376 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 39, length 64
17:01:15.759072 IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 2, seq 39, length 64
17:01:16.758661 IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 2, seq 40, length 64
# tcpdump -i eth0 -nn host 10.1.87.82
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:03:44.145010 IP 10.1.87.81.44247 > 10.1.87.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 3, seq 0, length 64
17:03:44.145551 IP 10.1.87.82.8942 > 10.1.87.81.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.2.195 > 10.244.1.0: ICMP echo reply, id 3, seq 0, length 64
17:03:45.145300 IP 10.1.87.81.44247 > 10.1.87.82.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.1.0 > 10.244.2.195: ICMP echo request, id 3, seq 1, length 64
17:03:45.145854 IP 10.1.87.82.8942 > 10.1.87.81.8472: OTV, flags [I] (0x08), overlay 0, instance 1

Flannel VxLAN的Direct routing模式配置

# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# vim  kube-flannel.yml   
net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan",
        "Directrouting": true  # 添加字段
      }
    }
# kubectl delete -f kube-flannel.yml
# kubectl apply -f kube-flannel.yml
# ip route show
default via 10.1.87.1 dev eth0
10.1.87.0/24 dev eth0 proto kernel scope link src 10.1.87.80
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.1.0/24 via 10.1.87.81 dev eth0
10.244.2.0/24 via 10.1.87.82 dev eth0
10.244.3.0/24 via 10.1.87.83 dev eth0

注: 在生产环境中不可以这么做,会影响到Pod的网络环境。生产环境一般在使用前会考虑并初始化好网络环境,也就是说先修改 flannel 才开始使用创建Pod;


calico 安装使用

Canal/flannel Hosted Install 官方文档

# kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
# kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/canal.yaml

创建名称空间
podSelector: {} 为空时,代表所有Pod

# kubectl create namespace dev
namespace/dev created
# kubectl create namespace prod
namespace/prod created
# vim ingress-def.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress
# kubectl apply -f ingress-def.yaml  -n dev
# kubectl  get netpol -n dev
NAME               POD-SELECTOR   AGE
deny-all-ingress   <none>         93s
# vim pod-a.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
# kubectl apply -f pod-a.yaml -n dev
# kubectl get pods -n dev -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE                   NOMINATED NODE   READINESS GATES
pod1   1/1     Running   0          54s   10.244.3.2   node03   <none>           <none>

这是请求10.244.3.2 发现是请求不通的

# curl  10.244.3.2
curl: (7) Failed connect to 10.244.3.2:80; Connection timed out

而把 pod-a.yaml 创建在prod 名称空间,此时prod名称空间是没有定义的

# kubectl  apply -f pod-a.yaml -n prod
pod/pod1 created
# kubectl get pods -n prod -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE                   NOMINATED NODE   READINESS GATES
pod1   1/1     Running   0          17s   10.244.3.3   node03   <none>           <none>node

请求10.244.3.3名称空间的可以请求通

# curl 10.244.3.3
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

当允许dev名称空间的Pod 都可以访问时,只需加 ingress: - {} 就可以了

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector: {}
  ingress:
  - {}
  policyTypes:
  - Ingress
# kubectl apply -f ingress-def.yaml  -n dev
# curl  10.244.3.2
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

定义允许访问一组Pod,使用标签来实现

# kubectl label pods pod1 app=myapp -n dev
pod/pod1 labeled

还原ingress-def 为默认,拒绝所有dev 名称空间所有Pod的访问

# vim ingress-def.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress
# kubectl apply -f ingress-def.yaml  -n dev
# curl  10.244.3.2
curl: (7) Failed connect to 10.244.3.2:80; Connection timed out

配置测略,允许某网段允许访问指定策略

# vim allow-netpol-demo.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-myapp-ingress
spec:
  podSelector:
    matchLabels:
      app: myapp
  ingress:
  - from:
    - ipBlock:
        cidr: 10.244.0.0/16
        except:
        - 10.244.1.2/32
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443
# kubectl apply -f allow-netpol-demo.yaml -n dev
# kubectl get netpol -n dev
NAME                  POD-SELECTOR   AGE
allow-myapp-ingress   app=myapp      2m
deny-all-ingress      <none>         34m
# curl  10.244.3.2:80
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

管控出站流量的方式

# vim egress-def.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
# kubectl  apply -f egress-def.yaml -n prod
networkpolicy.networking.k8s.io/deny-all-egress created
# kubectl apply -f pod-a.yaml -n prod # 将之前的pod-a.yaml 创建到prod名称空间
pod/pod1 unchanged
# kubectl get pods -n prod -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE                   NOMINATED NODE   READINESS GATES
pod1   1/1     Running   0          34m   10.244.3.3   node03   <none>           <none>node
# kubectl exec pod1 -it -n prod -- /bin/sh
/ # ping 10.1.87.81
PING 10.1.87.81 (10.1.87.81): 56 data bytes
^C
--- 10.1.87.81 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
/ #
# vim egress-def.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-egress
spec:
  podSelector: {}
  egress:
  - {}
  policyTypes:
  - Egress
# kubectl  apply -f egress-def.yaml -n prod
# kubectl exec pod1 -it -n prod -- /bin/sh
/ # ping 10.1.87.81
PING 10.1.87.81 (10.1.87.81): 56 data bytes
64 bytes from 10.1.87.81: seq=0 ttl=63 time=0.660 ms
64 bytes from 10.1.87.81: seq=1 ttl=63 time=0.564 ms
^C
--- 10.1.87.81 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.564/0.612/0.660 ms

可以看到放行所有Pod出站规则,网络是通的;

网络策略:
名称空间:
拒绝所有出站,入站;
放行所有出站目标本名称空间内的所有Pod;

评论