Kubernetes networking - Flannel and Calico

 

1. Kubernetes networking and CNI

Kubernetes uses the Container Network Interface, usually called CNI, to plug different networking implementations into the cluster.

A CNI plugin is responsible for things like:

• Giving pods IP addresses

• Setting up routes and rules so that pods can reach each other and the outside world

• Optionally enforcing network policy for security

Popular CNI plugins include Flannel, Calico, Cilium and Weave Net.

Flannel and Calico are two of the most common choices, but they focus on slightly different goals.

---

2. Flannel

2.1 Main purpose

Flannel is a simple, lightweight layer 3 network fabric for Kubernetes. It gives each pod a unique IP and provides connectivity between nodes in the cluster.

You can think of it as “just make the pods talk to each other” without many extra features.

2.2 How it works (high level)

• Flannel runs an agent called flanneld on every node

• It allocates a subnet to each node from a larger cluster network

• Packets between nodes are usually carried using an overlay format such as VXLAN, or via a simpler host gateway mode

• Flannel often acts as a “meta plugin” that wraps basic CNI reference plugins like the bridge plugin

Encapsulation with VXLAN means packets are wrapped inside another UDP packet while moving between nodes, which simplifies routing but adds some overhead.

2.3 Features and strengths

• Very easy to deploy and operate

• Good enough performance for many small and medium clusters

• Supports several back ends such as VXLAN and host gateway

• Widely supported by distributions and installers

2.4 Limitations

• Flannel focuses on basic connectivity and does not implement network policy by itself, so you need another component if you want policy enforcement.

• Lacks more advanced security and observability features

• Overlay networks introduce extra headers which can reduce raw throughput compared to a pure underlay design

---

3. Calico

3.1 Main purpose

Calico is both a networking provider and a powerful network policy engine for Kubernetes and other platforms.

It aims at high performance, strong security and good scalability for large clusters.

3.2 How it works (high level)

By default Calico builds a pure IP layer 3 network using standard routing protocols, usually BGP, to advertise pod networks between nodes. This underlay style avoids an extra overlay in many setups, which can improve performance and simplify troubleshooting.

Calico can also use encapsulation modes such as IP in IP or VXLAN when direct routing is not possible, so it is quite flexible.

3.3 Network policy

Calico implements and enforces Kubernetes NetworkPolicy resources and also provides its own extended policy model with richer features:

• Policy ordering and priority

• Explicit deny rules

• Policies for pods, host interfaces and virtual machines

• More flexible matching based on labels and other selectors

Kubernetes itself defines the NetworkPolicy API but depends on a plugin such as Calico to actually enforce those rules.

3.4 Features and strengths

• High performance networking using routed layer 3 design

• Rich and mature network policy implementation

• Good fit for large or security sensitive clusters

• Works across Kubernetes, virtual machines and host interfaces, not just pods

3.5 Limitations

• More components and concepts to learn, especially BGP and its configuration

• Setup can be more complex than Flannel, particularly in very simple lab environments

• Some cloud managed Kubernetes offerings already include their own policy engine, which may overlap with Calico features

---

4. Similarities between Flannel and Calico

Flannel and Calico have quite a lot in common:

1. Both are CNI plugins that integrate natively with Kubernetes

2. Both create a flat pod network where each pod gets its own routable IP

3. Both have agents running on every node that program routes and rules

4. Both support different back end modes to adapt to various environments

5. Both are open source and widely used in the community

In other words, both solve “pods on different nodes should always be able to reach each other using IP”.

---

5. Key differences

Here is a conceptual comparison, without trying to be vendor marketing.

5.1 Networking model

• Flannel

  • Primarily focuses on an overlay model such as VXLAN or a simple host gateway mode

  • Routing logic is relatively simple, which keeps operations easy

• Calico

  • Prefers a routed underlay model using BGP so that pods are directly reachable in the network

  • Can also use encapsulation (IP in IP or VXLAN) when needed

  • Often gives better raw performance, especially at scale

5.2 Network policy support

• Flannel

  • Does not implement network policy itself

  • If you need Kubernetes NetworkPolicy you usually combine Flannel with another policy engine or choose a different CNI

• Calico

  • Full Kubernetes NetworkPolicy support and an extended policy language

  • Frequently used primarily because of its policy features, even when basic connectivity is covered elsewhere

5.3 Complexity and operational model

• Flannel is easier to understand and maintain, which makes it attractive for very small or simple clusters such as test labs or single purpose clusters.

• Calico introduces more components (for example BGP speakers and policy controllers) and more configuration options, but in exchange it handles complex topologies and scaling better.

5.4 Typical use cases

• Flannel

  • Small to medium clusters

  • Environments where you only need basic connectivity and do not plan to use network policy

  • Educational or lab clusters that should stay simple

• Calico

  • Production clusters with stricter security requirements

  • Large deployments with many nodes and namespaces

  • Environments that require detailed control over traffic flows, including east west traffic inside the cluster

---

6. Other CNI alternatives

If you are choosing networking for Kubernetes, Flannel and Calico are not the only options.

Some notable alternatives:

1. Cilium

   • Uses eBPF for data plane processing

   • Very strong focus on security and observability and on higher layer policies

   • Frequently ranked highly in recent performance comparisons

2. Weave Net

   • Simple to deploy, creates an encrypted mesh overlay between nodes

   • Supports features like automatic encryption and network isolation

3. Canal

   • A combination of Flannel for networking and Calico for network policy

   • Often used as a convenient middle ground when you want simple deployment plus policy features

4. Cloud provider specific CNI plugins

   • Amazon VPC CNI, Azure CNI and others integrate pods directly into the cloud provider network

   • These are often used alongside or instead of Calico or other policy engines, depending on the platform

5. CNI Genie and similar meta plugins

   • Allow a cluster to choose or combine multiple CNIs, for example Calico, Flannel or Weave, inside one environment

---

7. How to think about the choice

A quick mental model:

• Choose Flannel when you want the simplest possible solution that just gives pod to pod connectivity, typically for smaller or low risk clusters.

• Choose Calico when you care about performance, need strong network policy and plan to scale to many nodes or tenants.

• Consider Cilium, Weave Net or Canal when you want extra capabilities such as eBPF based security, built in encryption or an integrated blend of Flannel style networking with Calico style policy.

Kubernetes - operator

Summary

A CRD extends the Kubernetes API by defining a new resource type (its schema, versions, validation, etc.). Once a CRD is installed, the API server can accept, validate, and store objects of that new type.

A CR is an instance of that new type—an actual object created by users or tools that carries the desired state for your operator to realize. The API server treats it like any other object once the CRD exists. 

The API server validates your Custom Resource (CR) against its Custom Resource Definition (CRD) and, if valid, stores it in the etcd database. The controller is a separate process that reacts to this event.

Relationship between Controller and Reconciler

They are fundamentally linked. The reconciler is the core piece of logic inside the controller.

* Controller: The controller is the entire running program (the operator). Its primary job is to "watch" the Kubernetes API server for changes to specific resources (like the creation of your CR). It handles the boilerplate work of connecting to the cluster, managing caches of objects, and putting events into a work queue.

* Reconciler: The reconciler is the specific function or method that the controller calls to process an event from the work queue. It contains the actual business logic of your operator. Its one and only goal is to make the current state of the world match the desired state defined in a resource.

Think of the controller as the engine and the reconciler as the specific set of instructions the engine executes for a given task.

The Flow: From CR Creation to Reconciliation

Here is a detailed, step-by-step explanation of the flow:

   1. CR Creation: You (or another program) send a manifest for a new Custom Resource to the Kubernetes API server (e.g., via kubectl apply).

   2. API Server & etcd: The API server authenticates and validates the request. If it's valid, the API server persists the new CR object as a record in the etcd database. At this point, the object exists in the cluster as the "desired state."

   3. The Watcher (Informer): Your running controller has established a "watch" on the API server for the type of CR you just created. The API server notifies the controller's watcher that a new object has been created.

   4. The Work Queue: The controller's internal machinery (often called an Informer) receives this notification and places a key for the object (typically in the format namespace/name) into a work queue. This queue ensures that every change is processed, even if the operator is busy, and handles retries on failure.

   5. The Reconcile Loop: The controller has a loop that constantly pulls keys from the work queue. When it pulls the key for your new CR, it directly calls the Reconciler function, passing it that key.

   6. Reconciliation Logic: Inside the Reconciler function, the following happens:

       * It fetches the full CR object from the cluster using the provided key.

       * It observes the actual state of the world (e.g., checks if a Pod, a ConfigMap, or an external resource it's supposed to manage already exists).

       * It compares the desired state (from the CR's spec) with the actual state.

       * It takes action to converge the actual state toward the desired state. For a new CR, this usually means creating new Kubernetes objects (like Deployments, Services, etc.) or interacting with an external API.

       * Finally, it often updates the status field of your CR to report on the actual state of the resources it now manages.

CRDs can be categorized into two main types

1. Active CRDs (The kind you are thinking of)

This is the most common pattern. The CR represents a desired, active state in the cluster. You create the CR because you want the operator to do something and create other resources.

    * Example: agent
    * You want: A running agent pod, a Kubernetes Service, a ConfigMap, etc.
    * How it works: You kubectl apply an agent manifest. The agent controller sees it, reads its spec, and creates a Deployment, Service, etc., to make your wish a reality. The controller's job is to reconcile the agent object.

2. Passive / Configuration CRDs (The exception)

This is a less common but powerful pattern. The CR does not represent an active resource, but rather a reusable piece of configuration, a template, or a policy. It doesn't do anything on its own. It just sits in the API server, holding data for other controllers to read and use.

    * Example: ServingRuntime
    * You want: A reusable blueprint for how agent pods should be configured (e.g., what container image to use, what environment variables are needed).
    * How it works: An administrator kubectl applies a ServingRuntime manifest. Nothing happens in the cluster. No pods are created. The CR is just stored as data.

Example CRD, CR

click to open

Active CRD - CRD - Agent

Defines the schema your controller watches and reconciles.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: agents.example.mycompany.io
spec:
  group: example.mycompany.io
  scope: Namespaced
  names:
    plural: agents
    singular: agent
    kind: Agent
    shortNames: [ag]
    categories: [all, agents]
  versions:
    - name: v1alpha1
      served: true
      storage: true
      additionalPrinterColumns:
        - name: Image
          type: string
          jsonPath: .spec.image
        - name: Replicas
          type: integer
          jsonPath: .spec.replicas
        - name: Phase
          type: string
          jsonPath: .status.phase
      subresources:
        status: {}
      schema:
        openAPIV3Schema:
          type: object
          description: Agent represents a running agent workload managed by the operator.
          properties:
            spec:
              type: object
              required: ["image"]
              properties:
                image:
                  type: string
                  description: Container image for the agent.
                replicas:
                  type: integer
                  minimum: 0
                  default: 1
                env:
                  type: array
                  description: Environment variables injected into the agent pod(s).
                  items:
                    type: object
                    required: ["name"]
                    properties:
                      name:
                        type: string
                      value:
                        type: string
                config:
                  type: object
                  description: Free-form configuration passed via a ConfigMap.
                  properties:
                    configMapRef:
                      type: object
                      properties:
                        name:
                          type: string
                service:
                  type: object
                  description: Exposes the agent via a Service.
                  properties:
                    type:
                      type: string
                      enum: ["ClusterIP","NodePort","LoadBalancer"]
                      default: "ClusterIP"
                    port:
                      type: integer
                      minimum: 1
                      maximum: 65535
                      default: 8080
                resources:
                  type: object
                  description: Pod resource requests/limits.
                  properties:
                    requests:
                      type: object
                      additionalProperties:
                        type: string
                    limits:
                      type: object
                      additionalProperties:
                        type: string
                servingRuntimeRef:
                  type: object
                  description: Optional reference to a passive ServingRuntime config.
                  properties:
                    name:
                      type: string
                    namespace:
                      type: string
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum: ["Pending","Reconciling","Ready","Error"]
                readyReplicas:
                  type: integer
                  minimum: 0
                endpoints:
                  type: array
                  items:
                    type: string
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type: { type: string }
                      status: { type: string }
                      reason: { type: string }
                      message: { type: string }
                      lastTransitionTime: { type: string, format: date-time }

CRs - Agent

Minimal:

apiVersion: example.mycompany.io/v1alpha1
kind: Agent
metadata:
  name: agent-minimal
spec:
  image: ghcr.io/myco/agent:1.0.0

Full (references a ConfigMap and exposes a Service):

apiVersion: example.mycompany.io/v1alpha1
kind: Agent
metadata:
  name: agent-full
spec:
  image: ghcr.io/myco/agent:1.2.3
  replicas: 3
  env:
    - name: LOG_LEVEL
      value: info
  config:
    configMapRef:
      name: agent-full-config
  service:
    type: ClusterIP
    port: 8080
  resources:
    requests:
      cpu: "200m"
      memory: "256Mi"
    limits:
      cpu: "1"
      memory: "512Mi"
  servingRuntimeRef:
    name: default-serving-runtime
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-full-config
data:
  agent.yaml: |
    featureFlags:
      coolThing: true
    endpoints:
      metrics: /metrics

---

Passive / Configuration CRD - CRD - ServingRuntime

A reusable config object that doesn’t create pods by itself; your controller reads it when reconciling Agent.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: servingruntimes.example.mycompany.io
spec:
  group: example.mycompany.io
  scope: Namespaced
  names:
    plural: servingruntimes
    singular: servingruntime
    kind: ServingRuntime
    shortNames: [sr]
    categories: [all, config]
  versions:
    - name: v1alpha1
      served: true
      storage: true
      additionalPrinterColumns:
        - name: Image
          type: string
          jsonPath: .spec.image
      schema:
        openAPIV3Schema:
          type: object
          description: Passive config describing how agents should be configured.
          properties:
            spec:
              type: object
              required: ["image"]
              properties:
                image:
                  type: string
                  description: Default container image to use for agents.
                envDefaults:
                  type: array
                  items:
                    type: object
                    required: ["name","value"]
                    properties:
                      name: { type: string }
                      value: { type: string }
                ports:
                  type: array
                  items:
                    type: object
                    properties:
                      name: { type: string }
                      containerPort: { type: integer }
                podAnnotations:
                  type: object
                  additionalProperties:
                    type: string

CRs - ServingRuntime

apiVersion: example.mycompany.io/v1alpha1
kind: ServingRuntime
metadata:
  name: default-serving-runtime
spec:
  image: ghcr.io/myco/agent:stable
  envDefaults:
    - name: TZ
      value: UTC
    - name: LOG_FORMAT
      value: json
  ports:
    - name: http
      containerPort: 8080
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"

REF

https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

https://book.kubebuilder.io/reference/watching-resources
https://aws-controllers-k8s.github.io/community/docs/community/overview/