1

Развернут кластер kubernetes на vServer'ах.

Каждый vServer имеет серый адрес 172.31.1.100. У хостера подход 1:1 nat - ссылка на hetzner

etcd, apiserver слушают externalIP

podNetwork на нодах не видят друг друга и нет резолва

Пингую сам себя:

kube@kube1:~$ ping 10.244.0.0
PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
64 bytes from 10.244.0.0: icmp_seq=1 ttl=64 time=0.065 ms
64 bytes from 10.244.0.0: icmp_seq=2 ttl=64 time=0.039 ms
^C
--- 10.244.0.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1027ms
rtt min/avg/max/mdev = 0.039/0.052/0.065/0.013 ms

Пингую podNetwork, расположенную на другой ноде:

kube@kube1:~$ ping 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
^C
--- 10.244.1.0 ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time 12274ms

Пинг с pod'a в мир:

kube@kube1:~$ kubectl exec dnsutils -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=3.71 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=6.49 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=3.73 ms
^C

Резолв с pod'a:

kube@kube1:~$ kubectl exec dnsutils -- nslookup google.com
;; connection timed out; no servers could be reached

command terminated with exit code 1

Состояние:

kube@kube1:~$ kubectl get no
NAME         STATUS    ROLES     AGE       VERSION
kube1.htzn   Ready     master    2d        v1.8.4
kube3.htzn   Ready     <none>    2d        v1.8.4
kube4.htzn   Ready     <none>    2d        v1.8.4

kube@kubectl get services -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   2d

kube@kubectl get po -n kube-system
kube-apiserver-kube1.htzn            1/1       Running   0          2d        externalIP   kube1.htzn
kube-controller-manager-kube1.htzn   1/1       Running   0          2d        externalIP   kube1.htzn
kube-dns-545bc4bfd4-4c84v            3/3       Running   0          2d        10.244.2.6       kube4.htzn
kube-dns-545bc4bfd4-4xbbr            3/3       Running   0          2d        10.244.1.10      kube3.htzn
kube-dns-545bc4bfd4-j25wx            3/3       Running   0          2d        10.244.1.9       kube3.htzn
kube-dns-545bc4bfd4-l4q9d            3/3       Running   0          2d        10.244.0.10      kube1.htzn
kube-dns-545bc4bfd4-lh8rc            3/3       Running   0          2d        10.244.2.4       kube4.htzn
kube-flannel-ds-8wdcg                1/1       Running   0          2d        externalIP   kube4.htzn
kube-flannel-ds-9zj22                1/1       Running   0          2d        externalIP   kube1.htzn
kube-flannel-ds-m48lq                1/1       Running   0          2d        externalIP   kube3.htzn
kube-proxy-22lpv                     1/1       Running   0          2d        externalIP   kube4.htzn
kube-proxy-g5gcq                     1/1       Running   0          2d        externalIP   kube1.htzn
kube-proxy-ll96w                     1/1       Running   0          2d        externalIP   kube3.htzn
kube-scheduler-kube1.htzn            1/1       Running   0          2d        externalIP   kube1.htzn

ifconfig на мастере:

br-37317408aabe: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:00:96:fc:f1  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 

cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.0.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::68b3:f8ff:fec7:f2a5  prefixlen 64  scopeid 0x20<link>
        ether 0a:58:0a:f4:00:01  txqueuelen 1000  (Ethernet)
        RX packets 730314  bytes 171524562 (163.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 778142  bytes 168028178 (160.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::42:8aff:fe57:275a  prefixlen 64  scopeid 0x20<link>
        ether 02:42:8a:57:27:5a  txqueuelen 0  (Ethernet)
        RX packets 179202  bytes 23207264 (22.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 144645  bytes 65400826 (62.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.31.1.100  netmask 255.255.255.0  broadcast 172.31.1.255
        inet6 2a01:4f8:c0c:3c00::2  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::5054:a2ff:fe02:264a  prefixlen 64  scopeid 0x20<link>
        ether 52:54:a2:02:26:4a  txqueuelen 1000  (Ethernet)
        RX packets 22655506  bytes 6814574498 (6.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22836130  bytes 7824566265 (7.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::a85d:92ff:fed1:ec2  prefixlen 64  scopeid 0x20<link>
        ether aa:5d:92:d1:0e:c2  txqueuelen 0  (Ethernet)
        RX packets 20  bytes 1960 (1.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 20  bytes 1960 (1.9 KiB)
        TX errors 0  dropped 4 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 3184471  bytes 424875074 (405.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3184471  bytes 424875074 (405.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth3d3e856: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::9450:8aff:fe39:db80  prefixlen 64  scopeid 0x20<link>
        ether 96:50:8a:39:db:80  txqueuelen 0  (Ethernet)
        RX packets 179202  bytes 25716092 (24.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 144649  bytes 65401186 (62.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth520581d9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::5019:8aff:fe71:5e02  prefixlen 64  scopeid 0x20<link>
        ether 52:19:8a:71:5e:02  txqueuelen 0  (Ethernet)
        RX packets 725841  bytes 180660769 (172.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 773371  bytes 166986212 (159.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip r:

default via 172.31.1.1 dev ens3
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 dev br-37317408aabe proto kernel scope link src 172.18.0.1 linkdown
172.31.1.0/24 dev ens3 proto kernel scope link src 172.31.1.100

iptables -t nat -L -n:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0
MASQUERADE  all  --  172.18.0.0/16        0.0.0.0/0
RETURN     all  --  10.244.0.0/16        10.244.0.0/16
MASQUERADE  all  --  10.244.0.0/16       !224.0.0.0/4
RETURN     all  -- !10.244.0.0/16        10.244.0.0/24
MASQUERADE  all  -- !10.244.0.0/16        10.244.0.0/16

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-MARK-DROP (0 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-MARK-MASQ (14 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-2STLUSOBI3YCA5AH (2 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  94.130.110.182       0.0.0.0/0            /* default/kubernetes:https */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: SET name: KUBE-SEP-2STLUSOBI3YCA5AH side: source mask: 255.255.255.255 tcp to:94.130.110.182:6443

Chain KUBE-SEP-BGFZBX5GRCLSKMIC (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.2.4           0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.2.4:53

Chain KUBE-SEP-DFSYEE766Q4KZDEE (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.2.6           0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.2.6:53

Chain KUBE-SEP-E62IRM24GBWVMUE5 (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.1.10          0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.1.10:53

Chain KUBE-SEP-HPQF756YQTNK43WA (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.1.9           0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.1.9:53

Chain KUBE-SEP-NWYX6ZRA4HKJWFJ6 (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.1.9           0.0.0.0/0            /* kube-system/kube-dns:dns */
DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:10.244.1.9:53

Chain KUBE-SEP-OP4AXEAS4OXHBEQX (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.0.10          0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.0.10:53

Chain KUBE-SEP-QZ2RMBQRVUNY2HKV (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.1.10          0.0.0.0/0            /* kube-system/kube-dns:dns */
DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:10.244.1.10:53

Chain KUBE-SEP-R7EMXN5TTQQVP4UW (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.0.10          0.0.0.0/0            /* kube-system/kube-dns:dns */
DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:10.244.0.10:53

Chain KUBE-SEP-RJHMR3QLYGJVBWVL (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.2.4           0.0.0.0/0            /* kube-system/kube-dns:dns */
DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:10.244.2.4:53

Chain KUBE-SEP-XABH6FUURDMVT5Y2 (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.2.6           0.0.0.0/0            /* kube-system/kube-dns:dns */
DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:10.244.2.6:53

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0            10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-MARK-MASQ  udp  -- !10.244.0.0/16        10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
target     prot opt source               destination
KUBE-SEP-OP4AXEAS4OXHBEQX  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.20000000019
KUBE-SEP-E62IRM24GBWVMUE5  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.25000000000
KUBE-SEP-HPQF756YQTNK43WA  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.33332999982
KUBE-SEP-BGFZBX5GRCLSKMIC  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.50000000000
KUBE-SEP-DFSYEE766Q4KZDEE  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
target     prot opt source               destination
KUBE-SEP-2STLUSOBI3YCA5AH  all  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-2STLUSOBI3YCA5AH side: source mask: 255.255.255.255
KUBE-SEP-2STLUSOBI3YCA5AH  all  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
target     prot opt source               destination
KUBE-SEP-R7EMXN5TTQQVP4UW  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ statistic mode random probability 0.20000000019
KUBE-SEP-QZ2RMBQRVUNY2HKV  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ statistic mode random probability 0.25000000000
KUBE-SEP-NWYX6ZRA4HKJWFJ6  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ statistic mode random probability 0.33332999982
KUBE-SEP-RJHMR3QLYGJVBWVL  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ statistic mode random probability 0.50000000000
KUBE-SEP-XABH6FUURDMVT5Y2  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */

1 ответ 1

0

Была подобная фигня на хестнере когда виртуалка была в в датацентре финляндии а другая в другом датацентре. При чем это только у финских виртуалок такое с сетью бывает и не у каждой.

Удали виртуалку и добавь ноду по новой в кластер

и личше используй сетку calico вместо flannel

Ваш ответ

By clicking “Отправить ответ”, you agree to our terms of service and acknowledge you have read our privacy policy.

Всё ещё ищете ответ? Посмотрите другие вопросы с метками или задайте свой вопрос.