At KubeCon EU 2016 in London, I gave a first talk about using BPF and
Kubernetes together. I was
presenting a proof of concept to introduce various degraded network scenarios
in specific pods for testing the reliability of apps. There was not a lot of
BPF + Kubernetes talks back then. In the meantime, Kinvolk has worked on
various projects mixing Kubernetes and BPF together. The latest such project is
our own Inspektor Gadget, a
collection of “gadgets” for debugging and inspecting Kubernetes applications.
Today I would like to introduce Inspektor Gadget’s newest gadget that helps to
write proper Kubernetes network policies.
Writing Kubernetes network policies easily
Securing your Kubernetes clusters is a task that involves many aspects:
controlling what goes into your container images, writing RBAC rules for
different users and services, etc. Here I focus on one important aspect:
At Kinvolk we regularly do security assessments of Kubernetes in the form of
penetration testing for customers. Sometimes, the application is Kubernetes
native and the network policies are developed at the same time as the
application. This is ideal because the development team has a clear idea of
which pod is supposed to talk to which pod. But sometimes, a pre-Kubernetes
application is ported to Kubernetes and the developer tasked with writing the
network policies may not have a clear idea of the architecture. Architecture
documents might be missing or incomplete. Adding pod security as an
afterthought might not be ideal, but thankfully the Network Policy Advisor in
Inspektor Gadget can help us here.
The Network Policy Advisor workflow
A workflow we suggest that can improve things is to deploy the application in a
development cluster and let Inspektor Gadget monitor and analyse the network
traffic so it can suggest network policies. The developer can then review the
output and add them in the project.
We will use GoogleCloudPlatform’s microservices-demo application as an example.
contains various deployments and services but no network policies.
After preparing a “demo” namespace, let’s ask Inspektor Gadget to monitor the
network traffic from this namespace:
$ kubectl gadget network-policy monitor --namespaces demo --output ./networktrace.log
While it’s running in the background, deploy the application in the demo
namespace from another terminal:
$ wget -O network-policy-demo.yaml https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/ccff406cdcd3e043b432fe99b4038d1b4699c702/release/kubernetes-manifests.yaml $ kubectl apply -f network-policy-demo.yaml -n demo
Once the demo is deployed and running correctly, we can see all the pods in the
$ kubectl get pod -n demo NAME READY STATUS RESTARTS AGE adservice-58c85c77d8-k5667 1/1 Running 0 44s cartservice-579bdd6865-2wcbk 0/1 Running 1 45s checkoutservice-66d68cbdd-smp6w 1/1 Running 0 46s currencyservice-65dd85f486-62vld 1/1 Running 0 45s emailservice-84c98657cb-lqwfz 0/1 Running 2 46s frontend-788f7bdc86-q56rw 0/1 Running 1 46s loadgenerator-7699dc7d4b-j6vq6 1/1 Running 1 45s paymentservice-5c54c9887b-prz7n 1/1 Running 0 45s productcatalogservice-7df777f796-29lmz 1/1 Running 0 45s recommendationservice-89547cff8-xf4mv 0/1 Running 1 46s redis-cart-5f59546cdd-6rq8f 0/1 Running 2 44s shippingservice-778db496dd-mhdk5 1/1 Running 0 45s
At this point, the different pods will have communicated with each other. The
networktrace.log file contains one line per TCP connection with enough details
to be able to infer network policies later on.
Let’s stop the network monitoring by Inspektor Gadget using Ctrl-C, and
generate the Kubernetes network policies:
$ kubectl gadget network-policy report --input ./networktrace.log > network-policy.yaml
Note: Here we are running Inspektor Gadget as a kubectl subcommand. You could
also run it as a stand-alone binary using
One of the network policies it creates is for the cartservice: Inspektor Gadget
noticed that it received connections from the frontend and initiated
connections to redis-cart. It displays the following suggestion accordingly:
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: creationTimestamp: null name: cartservice-network namespace: demo spec: egress: - ports: - port: 6379 protocol: TCP to: - podSelector: matchLabels: app: redis-cart ingress: - from: - podSelector: matchLabels: app: frontend ports: - port: 7070 protocol: TCP podSelector: matchLabels: app: cartservice policyTypes: - Ingress - Egress
As you can see, it converted the set of connection tuples into a set of network
policies using usual Kubernetes label selectors instead of IP addresses.
Of course, those automatically-produced network policies should not be used
blindly: a developer should verify that the connections observed are
legitimate. The Network Policy Advisor gadget has some limitations too (see
#39), but it’s a lot
easier to review them and possibly make some small changes, rather than writing
them from scratch with a frustrating trial and error development cycle. This
saves precious development time and likely costs too.
Inspektor Gadget has useful features for developers of Kubernetes applications.
As an Open Source project, contributions are welcome. Join the discussions on
the #inspektor-gadget channel in the Kubernetes Slack or, if you want to know
about our services related to pentesting and security, reach us at