In Kubernetes, each pod is assigned a unique IP. This article compares how 3 Kubernetes network plugins interact in a Cumulus Linux Router to a Host L3 CLOS.

Topology

L3 Topology

Feature Comparison between Kubernetes Network plugins

Plugin Name Pod IP assignment POD Reachability
Flannel Each Pod is assigned a /32 from Kubernetes assigned range of IPs. vxLAN used to communicate. VxLAN discovery managed using etcd database. hypervisor IP to pod mapping is stored in the Etcd database.
Calico Each Pod is assigned a /32 from Kubernetes assigned range of IPs Each pod IP added dynamically to kernel hypervisor routing table as a /32. Use Cumulus Linux Quagga OSPF or BGP protocol provide Pod reachability
Weave ?? ??

Flannel

How Flannel works

[ Pic describing how Flannel works]

Advantages

  • Easy to manage. No extra routing protocol complexity. Cumulus Linux Quagga only has to manage single IP address on hypervisor. No need to manage dynamically created Pod IPs in the BGP or OSPF routing table.

Disadvantages

  • Requires Kernel VXLAN acceleration for apps that have high bandwidth requirements. VXLAN software encapsulation/decapsulation produces can product about 70% bandwidth utilization depending on the processors used. At least one CPU core will be used exclusively for VXLAN encapsulation/decapsulation

  • Hard to trace exact data path of Pod from network infrastructure perspective. This is because in the underlay, data is encapsulated in a VxLAN header.

Calico

How Calico works

Advantages

  • Easy to trace exact datapath of a pod based on its IP address using BGP AS Path. This assumes that one is using eBGP peering and not OSPF. example
This is an example of AS path to identify the location of a pod.
  • Apps that require high bandwidth can work well because there is no encapsulation/decapsulation of data.

Disadvantages

  • Switches will have to store a high number of /32 routes. Not possible to summarize routes in the underlay. This assumes you have only 1 Kubernetes cluster in your underlay. [ PIC ]

With multiple Kubernetes clusters one can apply some summarization between Kubernetes cluster underlays.

[PIC]

Weave

How Weave Works