Securing Cilium policies with Kyverno

Adding one more layer to the onion

Nov 03, 2023

Hey there, Kubernetes aficionados!

We've nearly hit the 200-reader mark for this weekly newsletter, and for that, I'm incredibly grateful! Knowing that you're out there, dedicating time to read through these insights makes the writing process much more rewarding.

I've received a lot of valuable feedback via direct messages, and I can't express enough how much I appreciate it!

Cilium and Kyverno

This week, we're diving into an issue that's been hot on the minds of many in our community—governing and controlling Cilium Network Policies. Whether you've been working with Kubernetes for years or just getting your feet wet, this topic has never been more critical, especially as Kubernetes deployments get more complex.

Although Cilium itself doesn't offer built-in governance capabilities, fear not! Adding these controls is straightforward, thanks partly to Cilium's native Kubernetes integration. Cilium's resources are actually Custom Resource Definitions (CRDs), making them compatible with a myriad of existing Kubernetes-native policy tools.

Why does this matter? Well, with these tools at your disposal, you can enforce granular permissions, allowing or blocking the creation of network policies based on criteria that you define. It's all about giving you more control over your cluster's network security posture.

To show you just how this can be done, I've crafted a compelling example using none other than the acclaimed Kyverno policy management solution, renowned for its Kubernetes-native design and real-time policy enforcement capabilities.

For this example, I've utilized Azure Kubernetes Service (AKS) with Bring Your Own CNI (BYOCNI), specifically running Cilium version 1.14.

Before diving in, it's crucial to clarify that this isn't intended as a production-ready guide. Instead, consider this an educational exploration, demonstrating a proof of concept that showcases what's possible with the right tools in hand.

Example 1: Enforce CiliumNetworkPolicy Annotations

To kick things off, let's get Kyverno installed. Run the following command to install version v1.10.0:

kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.10.0/install.yaml

Next, let's formulate our inaugural policy aimed at enforcing Kubernetes annotations. Annotations are metadata attached to Kubernetes resources, often utilized for resource management, labeling, and other organizational tasks.

This practice is a cornerstone for effective Kubernetes management. It offers tangible benefits, particularly during audit processes and debugging sessions where identifying and isolating resources quickly is paramount.

Without further ado, let's delve into a Kyverno rule that makes this all possible:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-ciliumnetworkpolicy-annotations
spec:
  validationFailureAction: Enforce
  background: false
  rules:
  - name: check-annotations
    match:
      resources:
        kinds:
        - CiliumNetworkPolicy
    validate:
      message: "Missing required annotation"
      pattern:
        metadata:
          annotations:
            purpose: "?*"
            owner: "?*"

apiVersion and kind

The policy starts by specifying the apiVersion as kyverno.io/v1, indicating that this is a Kyverno policy and we're using version 1 of the API. The kind is set to ClusterPolicy, meaning this policy will be applicable at the cluster level, across all namespaces.

metadata

The metadata section contains the name field set to enforce-ciliumnetworkpolicy-annotations. This provides a unique identifier for the policy within the cluster.

spec

The main body of the policy is defined under spec, which includes several key fields:

validationFailureAction: Set to enforce, this field specifies that if the policy rule is violated, the operation will be denied.

background: This is set to false, meaning the policy will not scan existing resources but will only act on new or modified resources.

rules

The rules array contains one rule named check-annotations.

match

resources: Specifies that the rule should apply to CiliumNetworkPolicy objects.

validate

message: If validation fails, this message will be shown to the user. It is set to "Missing required annotation".

pattern: Defines what the annotations on CiliumNetworkPolicy resources should look like.

metadata: Specifies that the annotations to be checked are under the metadata field of the resource.

annotations: A sub-field specifying which annotations need to be present.

purpose: The question mark followed by an asterisk (?*) indicates that the purpose annotation is optional but if present, it can have any value.

owner: Similar to purpose, the owner annotation is optional but can have any value if present.

The next step is to apply this policy to our cluster:

kubectl apply -f enforce-annotation.yaml

You can now verify that your resources have been deployed:

kubectl get clusterpolicies.kyverno.io

To continue, we want to deploy a Cilium Network Policy(CNP). For this example, I have created a namespace called Endor with a couple of deployments. In this namespace, I will try to deploy a CNP to allow all intercommunication in the namespace:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  namespace: endor
  name: allow-all
spec:
  endpointSelector:
    {}
  ingress:
    - fromEntities:
      - all
  egress:
    - toEntities:
      - all

kubectl apply -f allow-all.yaml -n endor

The admission webhook denies the policy creation:

Error from server: error when creating "../samples/policies/allow-all.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource CiliumNetworkPolicy/endor/allow-all was blocked due to the following policies

enforce-ciliumnetworkpolicy-annotations:
  check-annotations: 'validation error: Missing required annotation. rule check-annotations
    failed at path /metadata/annotations/owner/'

Let's update our policy to add the required annotations:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  namespace: endor
  name: allow-all
  annotations:
    purpose: "allowing_intercommunication_in_namespace"
    owner: "kubestory"
spec:
  endpointSelector:
    {}
  ingress:
    - fromEntities:
      - all
  egress:
    - toEntities:
      - all

kubectl apply -f allow-all.yaml -n endor

This time, it should be a great success!

You can verify your Kyverno report in the namespace where you apply your CNP:

k get policyreports.wgpolicyk8s.io -n endor

And finally, describe the report to know what CNP matches your Kyverno policy:

k describe policyreports.wgpolicyk8s.io -n endor cpol-enforce-ciliumnetworkpolicy-annotations

This example offers a comprehensive understanding of how Cilium and Kyverno interact, serving as a foundational guide to deploy and implement additional use cases. Over the past week, I've dedicated time to conceptualizing use cases that are both meaningful and based on real-world observations and customer interactions. I'm eager to share these valuable insights with you!

Example 2: Limit CiliumNetworkPolicy Scope

If you want to enforce that CiliumNetworkPolicies should only apply to specific namespaces, you can create a Kyverno rule to restrict their scope. This can ensure that network policies don't accidentally affect unintended namespaces.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: limit-ciliumnetworkpolicy-scope
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: limit-scope
    match:
      resources:
        kinds:
        - CiliumNetworkPolicy
    validate:
      message: "CiliumNetworkPolicy can only be applied to specific namespaces"
      anyPattern:
      - spec:
          endpointSelector:
            matchLabels:
              namespace: "allowed-namespace-1"
      - spec:
          endpointSelector:
            matchLabels:
              namespace: "allowed-namespace-2"

Example 3: Enforce Endpoint Security Best Practices

Let's say you want to ensure that CiliumNetworkPolicies always specify certain best practices, such as denying ingress from external IPs by default. You can enforce this through a Kyverno rule.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-endpoint-security
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: deny-external-ingress
    match:
      resources:
        kinds:
        - CiliumNetworkPolicy
    validate:
      message: "CiliumNetworkPolicy must deny ingress from external IPs"
      pattern:
        spec:
          ingress:
          - fromCIDR:
            - "!192.168.0.0/16"

Example 4: Restrict Usage of Certain Egress Ports

This policy restricts Cilium Network Policies from allowing egress to specific sensitive ports, such as port 22 (SSH).

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-egress-ports
spec:
  validationFailureAction: enforce
  rules:
  - name: check-egress-ports
    match:
      resources:
        kinds:
        - CiliumNetworkPolicy
    preconditions:
      - key: "{{ request.object.spec.egress }}"
        operator: NotEquals
        value: null
    validate:
      message: "Egress to port 22 is not allowed."
      deny:
        conditions:
        - key: "{{ request.object.spec.egress[0].toPorts[0].ports[0].port }}"
          operator: In
          value: ["22"]

In this policy, the preconditions section checks if the egress field is not null in the Cilium Network Policy object. Then, the validate section denies the policy if egress to port 22 is allowed.

These examples not only help you enforce essential metadata attributes but also add a layer of security by controlling egress ports. The policies are designed to provide enhanced governance and fine-grained control over Cilium Network Policies in your cluster.

Do you want more examples?

I found myself engrossed in brainstorming examples, quickly realizing it would make this article too lengthy to include them all. So, I've curated these examples and made them available in a GitHub repository for your convenience. Be sure to give it a star if you find it useful!

Music of the week

My top musical recommendation for this week is none other than Jacob Collier. He recently released a live show from Lisbon, which I consider to be one of his most outstanding performances of 2022. If you're not yet familiar with his work, you're in for a treat as you discover his extraordinary musical talent

KubeStory

Discussion about this post