Cilium Documentation Release 1.0.0-Rc9

Cilium Documentation Release 1.0.0-rc9

Cilium Authors

Apr 18, 2018 Getting Started

1 Introduction to Cilium 2 1.1 What is Cilium?...... 2 1.2 Why Cilium?...... 2 1.3 Functionality Overview...... 3

2 Getting Started Guides 5 2.1 Getting Started Using Minikube...... 5 2.2 Getting Started Using Istio...... 18 2.3 Getting Started Securing Kafka...... 33 2.4 Getting Started Securing gRPC...... 42 2.5 Getting Started Using Mesos/Marathon...... 49 2.6 Getting Started Using Docker Compose...... 56

3 Concepts 64 3.1 Component Overview...... 64 3.2 Terminology...... 67 3.3 Address Management...... 70 3.4 Multi Host Networking...... 71 3.5 Security...... 73 3.6 Datapath...... 76

4 Getting Help 77

5 Kubernetes 78 5.1 Quick Start...... 78 5.2 Introduction...... 79 5.3 Installation Guide...... 80 5.4 Network Policy...... 87 5.5 Troubleshooting...... 88

6 Istio 91

7 Docker 92

8 Mesos 93

9 System Requirements 94 9.1 Summary...... 94

i 9.2 Linux Distribution Compatibility Matrix...... 95 9.3 Linux Kernel...... 95 9.4 Key-Value store...... 95 9.5 clang+LLVM...... 96 9.6 iproute2...... 96

10 Installation Guides 97 10.1 Kubernetes Installation Guide (Generic)...... 97 10.2 CoreOS Installation Guide...... 97 10.3 Installation From Source...... 106 10.4 Advanced Options...... 106

11 Upgrade Guide 108 11.1 Kubernetes Cilium Upgrade...... 108

12 Network Policy 110 12.1 Policy Enforcement Modes...... 110 12.2 Rule Basics...... 111 12.3 Layer 3 Examples...... 112 12.4 Layer 4 Examples...... 118 12.5 Layer 7 Examples...... 119 12.6 Troubleshooting...... 123

13 Monitoring & Metrics 126 13.1 Exported Metrics...... 126 13.2 Cilium as a Kubernetes pod...... 127 13.3 Cilium as a host-agent on a node...... 128

14 Troubleshooting 129 14.1 Overall Health...... 129 14.2 Connectivity Issues...... 130 14.3 Cluster Diagnosis Tool...... 131 14.4 Cluster connectivity check...... 131 14.5 Monitoring Packet Drops...... 132 14.6 Policy Tracing...... 132 14.7 Debugging the datapath...... 132 14.8 Debugging information...... 133 14.9 Single Node Bugtool...... 133

15 Developer / Contributor Guide 136 15.1 Setting up a development environment...... 136 15.2 Testing...... 139 15.3 Development Cycle (Bash Script Framework)...... 140 15.4 Development Cycle (Ginkgo Framework)...... 141 15.5 Building Documentation...... 148 15.6 Update cilium-builder and cilium-runtime images...... 149 15.7 Submitting a pull request...... 155 15.8 Triggering Pull-Request Builds With Jenkins...... 156 15.9 Jenkins Job Descriptions...... 157 15.10 CI / Testing environment...... 158 15.11 Release Process...... 158 15.12 Developer’s Certiﬁcate of Origin...... 160

16 BPF and XDP Reference Guide 168 16.1 BPF Architecture...... 168 ii 16.2 Toolchain...... 178 16.3 Program Types...... 211 16.4 Further Reading...... 221

17 API Reference 229 17.1 Introduction...... 229 17.2 How to access the API...... 229 17.3 Compatibility Guarantees...... 230 17.4 API Reference...... 230

18 Command Cheatsheet 240 18.1 Command utilities:...... 241 18.2 Command examples:...... 241 18.3 Kubernetes examples:...... 244

19 Command Reference 246

20 Further Reading 247 20.1 Related Material...... 247 20.2 Presentations...... 247 20.3 Podcasts...... 247 20.4 Community blog posts...... 248

21 Glossary 249

HTTP Routing Table 251

iii Cilium Documentation, Release 1.0.0-rc9

The documentation is divided into the following sections: • Getting Started Guides: Provides a simple tutorial for running a small Cilium setup on your laptop. Intended as an easy way to get your hands dirty applying Cilium security policies between containers. • Concepts: Describes the components of the Cilium architecture, and the different models for deploying Cilium. Provides the high-level understanding required to run a full Cilium deployment and understand its behavior. • Installation Guides : Details instructions for installing, configuring, and troubleshooting Cilium in different deployment modes. • Policy Enforcement Modes : Detailed walkthrough of the policy language structure and the supported formats. • Monitoring & Metrics : Instructions for configuring metrics collection from Cilium. • Troubleshooting : Describes how to troubleshoot Cilium in different deployment modes. • BPF and XDP Reference Guide : Provides a technical deep dive of BPF and XDP technology, primarily focused at developers. • API Reference : Details the Cilium agent API for interacting with a local Cilium instance. • Developer / Contributor Guide : Gives background to those looking to develop and contribute modifications to the Cilium code or documentation.

Getting Started 1 CHAPTER 1

Introduction to Cilium

1.1 What is Cilium?

Cilium is open source software for transparently securing the network connectivity between application services deployed using Linux container management platforms like Docker and Kubernetes. At the foundation of Cilium is a new Linux kernel technology called BPF, which enables the dynamic insertion of powerful security visibility and control logic within Linux itself. Because BPF runs inside the Linux kernel, Cilium security policies can be applied and updated without any changes to the application code or container conﬁguration.

1.2 Why Cilium?

The development of modern datacenter applications has shifted to a service-oriented architecture often referred to as microservices, wherein a large application is split into small independent services that communicate with each other via APIs using lightweight protocols like HTTP. Microservices applications tend to be highly dynamic, with individual containers getting started or destroyed as the application scales out / in to adapt to load changes and during rolling updates that are deployed as part of continuous delivery. This shift toward highly dynamic microservices presents both a challenge and an opportunity in terms of securing connectivity between microservices. Traditional Linux network security approaches (e.g., iptables) filter on IP address and TCP/UDP ports, but IP addresses frequently churn in dynamic microservices environments. The highly volatile life cycle of containers causes these approaches to struggle to scale side by side with the application as load balancing tables and access control lists carrying hundreds of thousands of rules that need to be updated with a continuously growing frequency. Protocol ports (e.g. TCP port 80 for HTTP traffic) can no longer be used to differentiate between application traffic for security purposes as the port is utilized for a wide range of messages across services. An additional challenge is the ability to provide accurate visibility as traditional systems are using IP addresses as primary identification vehicle which may have a drastically reduced lifetime of just a few seconds in microservices architectures. By leveraging Linux BPF, Cilium retains the ability to transparently insert security visibility + enforcement, but does so in a way that is based on service / pod / container identity (in contrast to IP address identification in traditional systems) and can filter on application-layer (e.g. HTTP). As a result, Cilium not only makes it simple to apply security policies

2 Cilium Documentation, Release 1.0.0-rc9 in a highly dynamic environment by decoupling security from addressing, but can also provide stronger security isolation by operating at the HTTP-layer in addition to providing traditional Layer 3 and Layer 4 segmentation. The use of BPF enables Cilium to achieve all of this in a way that is highly scalable even for large-scale environments.

1.3 Functionality Overview

1.3.1 Protect and secure APIs transparently

Ability to secure modern application protocols such as REST/HTTP, gRPC and Kafka. Traditional ﬁrewalls operates at Layer 3 and 4. A protocol running on a particular port is either completely trusted or blocked entirely. Cilium provides the ability to ﬁlter on individual application protocol requests such as: • Allow all HTTP requests with method GET and path /public/.*. Deny all other requests. • Allow service1 to produce on Kafka topic topic1 and service2 to consume on topic1. Reject all other Kafka messages. • Require the HTTP header X-Token: [0-9]+ to be present in all REST calls. See the section Layer 7 Policy in our documentation for the latest list of supported protocols and examples on how to use it.

1.3.2 Secure service to service communication based on identities

Modern distributed applications rely on technologies such as application containers to facilitate agility in deployment and scale out on demand. This results in a large number of application containers to be started in a short period of time. Typical container firewalls secure workloads by filtering on source IP addresses and destination ports. This concept requires the firewalls on all servers to be manipulated whenever a container is started anywhere in the cluster. In order to avoid this situation which limits scale, Cilium assigns a security identity to groups of application containers which share identical security polices. The identity is then associated with all network packets emitted by the application containers, allowing to validate the identity at the receiving node. Security identity management is performed using a key-value store.

1.3.3 Secure access to and from external services

Label based security is the tool of choice for cluster internal access control. In order to secure access to and from external services, traditional CIDR based security policies for both ingress and egress are supported. This allows to limit access to and from application containers to particular IP ranges.

1.3.4 Simple Networking

A simple ﬂat Layer 3 network with the ability to span multiple clusters connects all application containers. IP alloca- tion is kept simple by using host scope allocators. This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: • Overlay: Encapsulation based virtual network spawning all hosts. Currently VXLAN and Geneve are baked in but all encapsulation formats supported by Linux can be enabled.

1.3. Functionality Overview 3 Cilium Documentation, Release 1.0.0-rc9

When to use this mode: This mode has minimal infrastructure and integration requirements. It works on almost any network infrastructure as the only requirement is IP connectivity between hosts which is typically already given. • Native Routing: Use of the regular routing table of the Linux host. The network is required to be capable to route the IP addresses of the application containers. When to use this mode: This mode is for advanced users and requires some awareness of the underlying networking infrastructure. This mode works well with: – Native IPv6 networks – In conjunction with cloud network routers – If you are already running routing daemons

1.3.5 Load balancing

Distributed load balancing for trafﬁc between application containers and to external services. The loadbalancing is implemented using BPF using efﬁcient hashtables allowing for almost unlimited scale and supports direct server return (DSR) if the loadbalancing operation is not performed on the source host.

1.3.6 Monitoring and Troubleshooting

The ability to gain visibility and to troubleshoot issues is fundamental to the operation of any distributed system. While we learned to love tools like tcpdump and ping and while they will always ﬁnd a special place in our hearts, we strive to provide better tooling for troubleshooting. This includes tooling to provide: • Event monitoring with metadata: When a packet is dropped, the tool doesn’t just report the source and destination IP of the packet, the tool provides the full label information of both the sender and receiving among a lot of other information. • Policy decision tracing: Why is a packet being dropped or a request rejected. The policy tracing framework allows to trace the policy decision process for both, running workloads and based on arbitrary label deﬁnitions. • Metrics export via Prometheus: Key metrics are exported via Prometheus for integration with your existing dashboards.

1.3.7 Integrations

• Network plugin integrations: CNI, libnetwork • Container runtime events: containerd • Kubernetes: NetworkPolicy, Labels, Ingress, Service • Logging: syslog, ﬂuentd

4 Chapter 1. Introduction to Cilium CHAPTER 2

Getting Started Guides

This document serves as the easiest introduction to using Cilium. If you are new to Cilium it is recommended to read the Introduction to Cilium section ﬁrst to learn about the basic concepts and motivation. The following guides which each takes an estimated time of 10-15 minutes to complete will help you to get started in your area of choice:

2.1 Getting Started Using Minikube

This guide uses minikube to demonstrate deployment and operation of Cilium in a single-node Kubernetes cluster. The minikube VM requires approximately 2 GB of RAM and supports hypervisors like VirtualBox that run on Linux, macOS, and Windows. If you instead want to understand the details of deploying Cilium on a full ﬂedged Kubernetes cluster, then go straight to Installation Guide. If you haven’t read the Introduction to Cilium yet, we’d encourage you to do that ﬁrst. The best way to get help if you get stuck is to ask a question on the Cilium Slack channel. With Cilium contributors across the globe, there is almost always someone available to help.

2.1.1 Step 0: Install kubectl & minikube

1. Install kubectl version >= 1.7.0 as described in the Kubernetes Docs. 2. Install one of the hypervisors supported by minikube. 3. Install minikube >= 0.22.3 as described on minikube’s github page. Boot a minikube cluster with the Container Network Interface (CNI) network plugin, the localkube bootstrapper. The localkube bootstrapper provides etcd >= 3.1.0, a cilium dependency.

$ minikube start --network-plugin=cni --bootstrapper=localkube --memory=4096 --extra-

˓→config=apiserver.Authorization.Mode=RBAC

5 Cilium Documentation, Release 1.0.0-rc9

After minikube has ﬁnished setting up your new Kubernetes cluster, you can check the status of the cluster by running kubectl get cs:

$ kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health": "true"}

Bind the Kubernetes system account to the cluster-admin role to enable the kube-dns service to run with RBAC enabled:

$ kubectl create clusterrolebinding kube-system-default-binding-cluster-admin --

˓→clusterrole=cluster-admin --serviceaccount=kube-system:default

4. Install etcd as a dependency of cilium in minikube by running: $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/addons/etcd/standalone-etcd.yaml service "etcd-cilium" created statefulset.apps "etcd-cilium" created To check that all pods are Running and 100% ready, including kube-dns and etcd-cilium-0 run:

$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default etcd-cilium-0 1/1 Running 0 1m kube-system etcd-minikube 1/1 Running 0 3m kube-system kube-addon-manager-minikube 1/1 Running 0 4m kube-system kube-apiserver-minikube 1/1 Running 0 3m kube-system kube-controller-manager-minikube 1/1 Running 0 3m kube-system kube-dns-86f4d74b45-lhzfv 3/3 Running 0 4m kube-system kube-proxy-tcd7h 1/1 Running 0 4m kube-system kube-scheduler-minikube 1/1 Running 0 4m kube-system storage-provisioner 1/1 Running 0 4m

If you see output similar to this, you are ready to proceed to the next step.

Note: The output might differ between minikube versions, you should expect to have all pods in READY state before continuing.

2.1.2 Step 1: Installing Cilium

The next step is to install Cilium into your Kubernetes cluster. Cilium installation leverages the Kubernetes Daemon Set abstraction, which will deploy one Cilium pod per cluster node. This Cilium pod will run in the kube-system namespace along with all other system relevant daemons and services. The Cilium pod will run both the Cilium agent and the Cilium CNI plugin. To deploy Cilium, run: K8s 1.7 K8s 1.8 K8s 1.9 K8s 1.10

6 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

K8s 1.11 $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/1.7/cilium.yaml clusterrole "cilium" created serviceaccount "cilium" created clusterrolebinding "cilium" created configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset "cilium" created $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/1.8/cilium.yaml clusterrole "cilium" created serviceaccount "cilium" created clusterrolebinding "cilium" created configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset "cilium" created $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/1.9/cilium.yaml clusterrole "cilium" created serviceaccount "cilium" created clusterrolebinding "cilium" created configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset "cilium" created $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/1.10/cilium.yaml clusterrole "cilium" created serviceaccount "cilium" created clusterrolebinding "cilium" created configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset "cilium" created $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/1.11/cilium.yaml clusterrole "cilium" created serviceaccount "cilium" created clusterrolebinding "cilium" created configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset "cilium" created Kubernetes is now deploying Cilium with its RBAC, ConﬁgMap and Daemon Set as a pod on all cluster nodes. This operation is performed in the background. Run the following command to check the progress of the deployment:

$ kubectl get pods --namespace kube-system NAME READY STATUS RESTARTS AGE cilium-1c2cz 1/1 Running 0 21m kube-addon-manager-minikube 1/1 Running 0 23m kube-dns-910330662-jqdjk 3/3 Running 0 23m kubernetes-dashboard-g8nzs 1/1 Running 0 23m

2.1. Getting Started Using Minikube 7 Cilium Documentation, Release 1.0.0-rc9

Wait until the Cilium pod shows the STATUS as Running, like above. In this tutorial, it’s okay to move forward with the READY status as 0.

2.1.3 Step 2: Deploy the Demo Application

Now that we have Cilium deployed and kube-dns operating correctly we can deploy our demo application. In our simple example, there are three microservices applications: app1, app2, and app3. App1 runs an HTTP web- service on port 80, which is exposed as a Kubernetes Service that load-balances requests to app1 to be across two pod replicas. App2 and app3 exist so that we can test different security policies for allowing applications to access app1. Application Topology for Cilium and Kubernetes

8 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

The file demo.yaml contains a Kubernetes Deployment for each of the three applications, with each deployment identified using the Kubernetes labels id=app1, id=app2, and id=app3. It also include a app1-service, which load- balances traffic to all pods with label id=app1. $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/minikube/demo.yaml service "app1-service" created deployment "app1" created pod "app2" created pod "app3" created Kubernetes will deploy the pods and service in the background. Running kubectl get svc,pods will inform you about the progress of the operation. Each pod will go through several states until it reaches Running at which point the pod is ready.

$ kubectl get pods,svc NAME READY STATUS RESTARTS AGE po/app1-3720119688-1h7c5 1/1 Running 0 37s po/app1-3720119688-jzqx2 1/1 Running 0 37s po/app2 1/1 Running 0 37s po/app3 1/1 Running 0 37s

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/app1-service 10.0.0.199 80/TCP 37s svc/kubernetes 10.0.0.1 443/TCP 27m

Each pod will be represented in Cilium as an Endpoints. We can invoke the cilium tool inside the Cilium pod to list them:

$ kubectl -n kube-system get pods -l k8s-app=cilium NAME READY STATUS RESTARTS AGE cilium-1c2cz 1/1 Running 0 26m

$ kubectl -n kube-system exec cilium-1c2cz cilium endpoint list ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS

˓→(source:key[=value]) IPv6 IPv4 STATUS ENFORCEMENT ENFORCEMENT 250 Disabled Disabled 262 k8s:id=app2

˓→ f00d::a0f:0:0:fa 10.15.132.130 ready k8s:io.kubernetes.pod.

˓→namespace=default 4698 Disabled Disabled 264 k8s:id=app3

˓→ f00d::a0f:0:0:125a 10.15.86.236 ready k8s:io.kubernetes.pod.

˓→namespace=default 28950 Disabled Disabled 263 k8s:id=app1

˓→ f00d::a0f:0:0:7116 10.15.51.177 ready k8s:io.kubernetes.pod.

˓→namespace=default 32138 Disabled Disabled 263 k8s:id=app1

˓→ f00d::a0f:0:0:7d8a 10.15.150.193 ready k8s:io.kubernetes.pod.

˓→namespace=default

Both ingress and egress policy enforcement is still disabled on all of these pods because no network policy has been imported yet which select any of the pods.

2.1. Getting Started Using Minikube 9 Cilium Documentation, Release 1.0.0-rc9

2.1.4 Step 3: Apply an L3/L4 Policy

When using Cilium, endpoint IP addresses are irrelevant when defining security policies. Instead, you can use the labels assigned to the VM to define security policies, which are automatically applied to any container with that label, no matter where or when it is run within a container cluster. We’ll start with a simple example where we allow app2 to reach app1 on port 80, but disallow the same connectivity from app3 to app1. This is a simple policy that filters only on IP protocol (network layer 3) and TCP protocol (network layer 4), so it is often referred to as an L3/L4 network security policy. Note: Cilium performs stateful connection tracking, meaning that if policy allows the frontend to reach backend, it will automatically allow all required reply packets that are part of backend replying to frontend within the context of the same TCP/UDP connection. L4 Policy with Cilium and Kubernetes

10 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

We can achieve that with the following CiliumNetworkPolicy:

apiVersion:"cilium.io/v2" kind: CiliumNetworkPolicy description:"L3-L4 policy for getting started using Kubernetes guide" metadata: name:"rule1" spec: endpointSelector: matchLabels: id: app1 ingress: - fromEndpoints: - matchLabels: id: app2 toPorts: - ports: - port:"80" protocol: TCP

CiliumNetworkPolicies match on pod labels using “endpointSelector” to identify the sources and destinations to which the policy applies. The above policy whitelists trafﬁc sent from app2 pods to app1 pods on TCP port 80. To apply this L3/L4 policy, run: $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/minikube/l3_l4_policy.yaml If we run cilium endpoint list again we will see that the pods with the label id=app1 now have ingress policy enforcement enabled inline with the policy above.

$ kubectl -n kube-system exec cilium-1c2cz cilium endpoint list ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS

˓→(source:key[=value]) IPv6 IPv4 STATUS ENFORCEMENT ENFORCEMENT 250 Disabled Disabled 262 k8s:id=app2

˓→ f00d::a0f:0:0:fa 10.15.132.130 ready k8s:io.kubernetes.pod.

˓→namespace=default 4698 Disabled Disabled 264 k8s:id=app3

˓→ f00d::a0f:0:0:125a 10.15.86.236 ready k8s:io.kubernetes.pod.

˓→namespace=default 28950 Enabled Disabled 263 k8s:id=app1

˓→ f00d::a0f:0:0:7116 10.15.51.177 ready k8s:io.kubernetes.pod.

˓→namespace=default 32138 Enabled Disabled 263 k8s:id=app1

˓→ f00d::a0f:0:0:7d8a 10.15.150.193 ready k8s:io.kubernetes.pod.

˓→namespace=default

2.1.5 Step 4: Test L3/L4 Policy

We can now verify the network policy that was imported. You can now launch additional containers that represent other services attempting to access backend. Any new container with label id=app2 will be allowed to access the app1 on port 80, otherwise the network request will be dropped.

2.1. Getting Started Using Minikube 11 Cilium Documentation, Release 1.0.0-rc9

To test this out, we’ll make an HTTP request to app1 from both app2 and app3 pods:

$ kubectl exec app2 -- curl -s app1-service.default.svc.cluster.local

It works!

This works as expected. Now the same request run from an app3 pod will fail:

$ kubectl exec app3 -- curl -s app1-service.default.svc.cluster.local

This request will hang, so press Control-C to kill the curl request, or wait for it to time out. You can observe the policy via kubectl

$ kubectl get cnp NAME AGE rule1 42s

$ kubectl describe cnp rule1 Name: rule1 Namespace: default Labels: Annotations: API Version: cilium.io/v2 Kind: CiliumNetworkPolicy Metadata: Cluster Name: Creation Timestamp: 2018-04-04T22:46:16Z Generation: 0 Initializers: Resource Version: 17204 Self Link: /apis/cilium.io/v2/namespaces/default/ciliumnetworkpolicies/

˓→rule1 UID: fc04f184-3859-11e8-9fba-08002792155c Spec: Endpoint Selector: Match Labels: Any : Id: app1 Ingress: From Endpoints: Match Labels: Any : Id: app2 To Ports: Ports: Port: 80 Protocol: TCP Status: Nodes: Minikube: Enforcing: true Last Updated: 2018-04-04T22:46:20.462630226Z Local Policy Revision: 6 Ok: true Events:

12 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

2.1.6 Step 5: Apply and Test HTTP-aware L7 Policy

In the simple scenario above, it was sufﬁcient to either give app2 / app3 full access to app1’s API or no access at all. But to provide the strongest security (i.e., enforce least-privilege isolation) between microservices, each service that calls app1’s API should be limited to making only the set of HTTP requests it requires for legitimate operation. For example, consider an extremely simple scenario where app1 has only two API calls: • GET /public • GET /private Continuing with the example from above, if app2 requires access only to the GET /public API call, the L3/L4 policy along has no visibility into the HTTP requests, and therefore would allow any HTTP request from app2 (since all HTTP is over port 80). To see this, run:

$ kubectl exec app2 -- curl -s app1-service.default.svc.cluster.local/public { 'val': 'this is public' }

and

$ kubectl exec app2 -- curl -s app1-service.default.svc.cluster.local/private { 'val': 'this is private' }

L7 Policy with Cilium and Kubernetes

2.1. Getting Started Using Minikube 13 Cilium Documentation, Release 1.0.0-rc9

Cilium is capable of enforcing HTTP-layer (i.e., L7) policies to limit what URLs app2 is allowed to reach. Here is an example policy ﬁle that extends our original policy by limiting app2 to making only a GET /public API call, but disallowing all other calls (including GET /private).

apiVersion:"cilium.io/v2" kind: CiliumNetworkPolicy description:"L7 policy for getting started using Kubernetes guide" metadata: name:"rule1" spec: endpointSelector: matchLabels: id: app1 ingress: - fromEndpoints:

14 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

- matchLabels: id: app2 toPorts: - ports: - port:"80" protocol: TCP rules: http: - method:"GET" path:"/public"

Update the existing rule to apply L7-aware policy to protect app1 using: $ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/minikube/l3_l4_l7_policy.yaml We can now re-run the same test as above, but we will see a different outcome:

$ kubectl exec app2 -- curl -s app1-service.default.svc.cluster.local/public { 'val': 'this is public' } and

$ kubectl exec app2 -- curl -s app1-service.default.svc.cluster.local/private Access denied

$ kubectl get ciliumnetworkpolicies NAME AGE rule1 4m $ kubectl describe ciliumnetworkpolicies rule1 Name: rule1 Namespace: default Labels: Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"cilium.

˓→io/v2","description":"L7 policy for getting started using Kubernetes guide","kind":

˓→"CiliumNetworkPolicy","metadata":{"annotations... API Version: cilium.io/v2 Kind: CiliumNetworkPolicy Metadata: Cluster Name: Creation Timestamp: 2018-04-04T22:46:16Z Generation: 0 Initializers: Resource Version: 17527 Self Link: /apis/cilium.io/v2/namespaces/default/ciliumnetworkpolicies/

˓→rule1 UID: fc04f184-3859-11e8-9fba-08002792155c Spec: Endpoint Selector: Match Labels: Any : Id: app1 Ingress: From Endpoints: Match Labels:

2.1. Getting Started Using Minikube 15 Cilium Documentation, Release 1.0.0-rc9

Any : Id: app2 To Ports: Ports: Port: 80 Protocol: TCP Rules: Http: Method: GET Path: /public Status: Nodes: Minikube: Enforcing: true Last Updated: 2018-04-04T22:49:34.744363299Z Local Policy Revision: 9 Ok: true Events: and cilium CLI:

$ kubectl -n kube-system exec cilium-qh5l2 cilium policy get [ { "endpointSelector": { "matchLabels": { "any:id": "app1", "k8s:io.kubernetes.pod.namespace": "default" } }, "ingress": [ { "fromEndpoints": [ { "matchLabels": { "any:id": "app2", "k8s:io.kubernetes.pod.namespace": "default" } } ], "toPorts": [ { "ports": [ { "port": "80", "protocol": "TCP" } ], "rules": { "http": [ { "path": "/public", "method": "GET" } ] } } ] }

16 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

], "labels": [ { "key": "io.cilium.k8s.policy.name", "value": "rule1", "source": "unspec" }, { "key": "io.cilium.k8s.policy.namespace", "value": "default", "source": "unspec" } ] } ] Revision: 10

We hope you enjoyed the tutorial. Feel free to play more with the setup, read the rest of the documentation, and reach out to us on the Cilium Slack channel with any questions!

2.1.7 Step 6: Clean-Up

You have now installed Cilium, deployed a demo app, and tested both L3/L4 and L7 network security policies.

$ minikube delete

After this, you can re-run this guide from Step 1.

2.1.8 Extra: Metrics

To try out the metrics exported by cilium, simply install the example prometheus spec ﬁle: K8s 1.7 K8s 1.8 K8s 1.9 K8s 1.10 K8s 1.11 $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/plugins/prometheus/prometheus.yaml $ kubectl replace --force -f https://raw.githubusercontent.com/cilium/cilium/

˓→HEAD/examples/kubernetes/1.7/cilium.yaml $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/plugins/prometheus/prometheus.yaml $ kubectl replace --force -f https://raw.githubusercontent.com/cilium/cilium/

˓→HEAD/examples/kubernetes/1.8/cilium.yaml $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/plugins/prometheus/prometheus.yaml $ kubectl replace --force -f https://raw.githubusercontent.com/cilium/cilium/

˓→HEAD/examples/kubernetes/1.9/cilium.yaml

2.1. Getting Started Using Minikube 17 Cilium Documentation, Release 1.0.0-rc9

$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/plugins/prometheus/prometheus.yaml $ kubectl replace --force -f https://raw.githubusercontent.com/cilium/cilium/

˓→HEAD/examples/kubernetes/1.10/cilium.yaml $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes/plugins/prometheus/prometheus.yaml $ kubectl replace --force -f https://raw.githubusercontent.com/cilium/cilium/

˓→HEAD/examples/kubernetes/1.11/cilium.yaml This will create a barebones prometheus installation that you can use to inspect metrics from the agent, then restart cilium so it can consume the new prometheus conﬁguration. Navigate to the web ui with:

$ minikube service prometheus -n prometheus

2.2 Getting Started Using Istio

This document serves as an introduction to using Cilium to enforce security policies in Kubernetes micro-services managed with Istio. It is a detailed walk-through of getting a single-node Cilium + Istio environment running on your machine. If you haven’t read the Introduction to Cilium yet, we’d encourage you to do that ﬁrst. The best way to get help if you get stuck is to ask a question on the Cilium Slack channel. With Cilium contributors across the globe, there is almost always someone available to help. If you haven’t read the Introduction to Cilium yet, we’d encourage you to do that ﬁrst. The best way to get help if you get stuck is to ask a question on the Cilium Slack channel. With Cilium contributors across the globe, there is almost always someone available to help.

2.2.1 Step 0: Install kubectl & minikube

$ minikube start --network-plugin=cni --bootstrapper=localkube --memory=4096 --extra-

˓→config=apiserver.Authorization.Mode=RBAC

After minikube has ﬁnished setting up your new Kubernetes cluster, you can check the status of the cluster by running kubectl get cs:

$ kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health": "true"}

Bind the Kubernetes system account to the cluster-admin role to enable the kube-dns service to run with RBAC enabled:

18 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

$ kubectl create clusterrolebinding kube-system-default-binding-cluster-admin --

˓→clusterrole=cluster-admin --serviceaccount=kube-system:default

4. Install etcd as a dependency of cilium in minikube by running: $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

If you see output similar to this, you are ready to proceed to the next step.

Note: The output might differ between minikube versions, you should expect to have all pods in READY state before continuing.

2.2.2 Step 1: Install Cilium

˓→kubernetes/1.7/cilium.yaml | \ sed -e 's/sidecar-http-proxy: "false"/sidecar-http-proxy: "true"/' | \ kubectl create -f - configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset.extensions "cilium" created clusterrolebinding.rbac.authorization.k8s.io "cilium" created

2.2. Getting Started Using Istio 19 Cilium Documentation, Release 1.0.0-rc9 clusterrole.rbac.authorization.k8s.io "cilium" created serviceaccount "cilium" created $ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes/1.8/cilium.yaml | \ sed -e 's/sidecar-http-proxy: "false"/sidecar-http-proxy: "true"/' | \ kubectl create -f - configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset.extensions "cilium" created clusterrolebinding.rbac.authorization.k8s.io "cilium" created clusterrole.rbac.authorization.k8s.io "cilium" created serviceaccount "cilium" created $ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes/1.9/cilium.yaml | \ sed -e 's/sidecar-http-proxy: "false"/sidecar-http-proxy: "true"/' | \ kubectl create -f - configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset.extensions "cilium" created clusterrolebinding.rbac.authorization.k8s.io "cilium" created clusterrole.rbac.authorization.k8s.io "cilium" created serviceaccount "cilium" created $ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes/1.10/cilium.yaml | \ sed -e 's/sidecar-http-proxy: "false"/sidecar-http-proxy: "true"/' | \ kubectl create -f - configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset.extensions "cilium" created clusterrolebinding.rbac.authorization.k8s.io "cilium" created clusterrole.rbac.authorization.k8s.io "cilium" created serviceaccount "cilium" created $ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes/1.11/cilium.yaml | \ sed -e 's/sidecar-http-proxy: "false"/sidecar-http-proxy: "true"/' | \ kubectl create -f - configmap "cilium-config" created secret "cilium-etcd-secrets" created daemonset.extensions "cilium" created clusterrolebinding.rbac.authorization.k8s.io "cilium" created clusterrole.rbac.authorization.k8s.io "cilium" created serviceaccount "cilium" created Kubernetes is now deploying Cilium with its RBAC settings, ConfigMap and DaemonSet as a pod on minikube. This operation is performed in the background. Note that this Cilium configuration requires deploying Istio with sidecar proxies in order to filter HTTP traffic at Layer-7 for any pod.

20 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

Run the following command to check the progress of the deployment:

$ kubectl get daemonsets -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE cilium 1 1 0 1 0 6s

Wait until the cilium Deployment shows a CURRENT count of 1 like above (a READY value of 0 is OK for this tutorial).

2.2.3 Step 2: Install Istio

Download Istio version 0.7.0:

$ export ISTIO_VERSION=0.7.0 $ curl -L https://git.io/getLatestIstio | sh - $ export ISTIO_HOME=`pwd`/istio-${ISTIO_VERSION} $ export PATH="$PATH:${ISTIO_HOME}/bin"

Deploy Istio on Kubernetes, with a Cilium-speciﬁc variant of Pilot which injects the Cilium network policy ﬁlters into each Istio sidecar proxy:

$ sed -e 's,docker\.io/istio/pilot:,docker.io/cilium/istio_pilot:,' \ < ${ISTIO_HOME}/install/kubernetes/istio.yaml | \ kubectl create -f -

Conﬁgure Istio’s sidecar injection to use Cilium’s Docker images for the sidecar proxies: $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes-istio/istio-sidecar-injector-configmap-release.yaml Check the progress of the deployment (every service should have an AVAILABLE count of 1):

$ kubectl get deployments -n istio-system NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE istio-ca 1 1 1 1 2m istio-ingress 1 1 1 1 2m istio-mixer 1 1 1 1 2m istio-pilot 1 1 1 1 2m

Once all Istio pods are ready, we are ready to install the demo application.

2.2.4 Step 3: Deploy the Bookinfo Application V1

Now that we have Cilium and Istio deployed, we can deploy version v1 of the services of the Istio Bookinfo sample application. The BookInfo application is broken into four separate microservices: • productpage. The productpage microservice calls the details and reviews microservices to populate the page. • details. The details microservice contains book information. • reviews. The reviews microservice contains book reviews. It also calls the ratings microservice. • ratings. The ratings microservice contains book ranking information that accompanies a book review. In this demo, each specific version of each microservice is deployed into Kubernetes using separate YAML files which define: • A Kubernetes Service.

2.2. Getting Started Using Istio 21 Cilium Documentation, Release 1.0.0-rc9

• A Kubernetes Deployment specifying the microservice’s pods, specific to each service version. • A Cilium Network Policy limiting the traffic to the microservice, specific to each service version.

Each Deployment must be packaged with Istio’s Envoy sidecar proxy in order to be managed by Istio, by running the istioctl kube-inject command on each YAML file. The resulting YAML files must then be adapted to mount Cilium’s API Unix domain sockets into the sidecar to allow Cilium’s Envoy filters to query the Cilium agent. This adaptation can be done with the cilium-kube-inject.sed script: $ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes-istio/cilium-kube-inject.sed > ./cilium-kube-inject.sed To package the Istio sidecar proxy and generate ﬁnal YAML speciﬁcations, run: $ for service in productpage-service productpage-v1 details-v1 reviews-v1; do

˓→\ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes-istio/bookinfo-${service}.yaml | \ istioctl kube-inject --injectConfigMapName istio-inject -f - | \ sed -f ./cilium-kube-inject.sed | \ kubectl create -f - ; done service "productpage" created ciliumnetworkpolicy "productpage-v1" created deployment "productpage-v1" created service "details" created ciliumnetworkpolicy "details-v1" created deployment "details-v1" created service "reviews" created ciliumnetworkpolicy "reviews-v1" created deployment "reviews-v1" created Check the progress of the deployment (every service should have an AVAILABLE count of 1):

$ kubectl get deployments -n default NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE details-v1 1 1 1 1 6m productpage-v1 1 1 1 1 6m reviews-v1 1 1 1 1 6m

To obtain the URL to the frontend productpage service, run:

22 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

$ export PRODUCTPAGE=`minikube service productpage -n default --url` $ echo "Open URL: ${PRODUCTPAGE}/productpage"

Open that URL in your web browser and check that the application has been successfully deployed.

2.2.5 Step 4: Canary and Deploy the Reviews Service V2

We will now deploy version v2 of the reviews service. In addition to providing reviews from readers, reviews v2 queries a new ratings service for book ratings, and displays each rating as 1 to 5 black stars. As a precaution, we will use Istio’s service routing feature to canary the v2 deployment to prevent breaking the end-to-end application completely if it is faulty. Before deploying v2, to prevent any trafﬁc from being routed to it for now, we will create this Istio route rules to route 100% of the reviews trafﬁc to v1: apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: reviews-default spec: destination: name: reviews precedence:1 route: - labels: version: v1

Apply this route rule: $ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes-istio/route-rule-reviews-v1.yaml routerule "reviews-default" created

2.2. Getting Started Using Istio 23 Cilium Documentation, Release 1.0.0-rc9

Deploy the ratings v1 and reviews v2 services: $ for service in ratings-v1 reviews-v2; do \ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes-istio/bookinfo-${service}.yaml | \ istioctl kube-inject --injectConfigMapName istio-inject -f - | \ sed -f ./cilium-kube-inject.sed | \ kubectl create -f - ; done service "ratings" created ciliumnetworkpolicy "ratings-v1" created deployment "ratings-v1" created ciliumnetworkpolicy "reviews-v2" created deployment "reviews-v2" created Check the progress of the deployment (every service should have an AVAILABLE count of 1):

$ kubectl get deployments -n default NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE details-v1 1 1 1 1 6m productpage-v1 1 1 1 1 6m ratings-v1 1 1 1 1 57s reviews-v1 1 1 1 1 6m reviews-v2 1 1 1 1 57s

Check in your web browser that no stars are appearing in the Book Reviews, even after refreshing the page several times. This indicates that all reviews are retrieved from reviews v1 and none from reviews v2.

The ratings-v1 CiliumNetworkPolicy explicitly whitelists access to the ratings API only from productpage and reviews v2: apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: ratings-v1 namespace: default specs: - endpointSelector: matchLabels: "k8s:app": ratings "k8s:version": v1 ingress: - fromEndpoints: - matchLabels: "k8s:app": productpage "k8s:version": v1 toPorts:

24 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

- ports: - port:"9080" protocol: TCP rules: http: - method: GET path:"/ratings/[0-9] *" - fromEndpoints: - matchLabels: "k8s:app": reviews "k8s:version": v2 toPorts: - ports: - port:"9080" protocol: TCP rules: http: - method: GET path:"/ratings/[0-9] *"

Check that reviews v1 may not be able to access the ratings service, even if it were compromised or suffered from a bug, by running curl from within the pod:

$ export POD_REVIEWS_V1=`kubectl get pods -n default -l app=reviews,version=v1 -o

˓→jsonpath='{.items[0].metadata.name}'` $ kubectl exec ${POD_REVIEWS_V1} -c istio-proxy -- curl --connect-timeout 5 http://

˓→ratings:9080/ratings/0 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 curl: (28) Connection timed out after 5000 milliseconds

Update the Istio route rule to send 50% of reviews trafﬁc to v1 and 50% to v2: apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: reviews-default spec: destination: name: reviews precedence:1 route: - labels: version: v1 weight: 50 - labels: version: v2 weight: 50

2.2. Getting Started Using Istio 25 Cilium Documentation, Release 1.0.0-rc9

Apply this route rule: $ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes-istio/route-rule-reviews-v1-v2.yaml routerule "reviews-default" configured Check in your web browser that stars are appearing in the Book Reviews roughly 50% of the time. This may require refreshing the page for a few seconds to observe. Queries to reviews v2 result in reviews containing ratings displayed as black stars:

Finally, update the route rule to send 100% of reviews trafﬁc to v2: apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: reviews-default spec: destination: name: reviews precedence:1

26 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

route: - labels: version: v2

Apply this route rule: $ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes-istio/route-rule-reviews-v2.yaml routerule "reviews-default" configured Refresh the product page in your web browser several times to verify that stars are now appearing in the Book Reviews on every page refresh. All the reviews are now retrieved from reviews v2 and none from reviews v1.

2.2.6 Step 5: Deploy the Product Page Service V2

We will now deploy version v2 of the productpage service, which brings two changes: • It is deployed with a more restrictive CiliumNetworkPolicy, which restricts access to a subset of the HTTP URLs, at Layer-7. • It implements a new authentication audit log into Kafka.

2.2. Getting Started Using Istio 27 Cilium Documentation, Release 1.0.0-rc9

Because productpage v2 sends messages into Kafka, we must ﬁrst deploy a Kafka broker: $ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes-istio/kafka-v1.yaml | \ istioctl kube-inject --injectConfigMapName istio-inject -f - | \ sed -f ./cilium-kube-inject.sed | \ kubectl create -f - service "kafka" created ciliumnetworkpolicy "kafka-authaudit" created statefulset "kafka-v1" created Wait until the kafka-v1-0 pod is ready, i.e. until it has a READY count of 2/2:

$ kubectl get pods -n default -l app=kafka NAME READY STATUS RESTARTS AGE kafka-v1-0 2/2 Running 0 21m

Create the authaudit Kafka topic, which will be used by productpage v2:

$ kubectl exec kafka-v1-0 -c kafka -- bash -c '/opt/kafka_2.11-0.10.1.0/bin/kafka-

˓→topics.sh --zookeeper localhost:2181/kafka --create --topic authaudit --partitions

˓→1 --replication-factor 1' Created topic "authaudit".

We are now ready to deploy productpage v2. The policy for v1 currently allows read access to the full HTTP REST API, under the /api/v1 HTTP URI path: • /api/v1/products: Returns the list of books and their details. • /api/v1/products/: Returns details about a specific book. • /api/v1/products//reviews: Returns reviews for a specific book. • /api/v1/products//ratings: Returns ratings for a specific book. Check that the full REST API is currently accessible in v1 and returns valid JSON data:

28 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

$ export PRODUCTPAGE=`minikube service productpage -n default --url` $ for APIPATH in /api/v1/products /api/v1/products/0 /api/v1/products/0/reviews /api/

˓→v1/products/0/ratings; do echo ; curl -s -S "${PRODUCTPAGE}${APIPATH}" ; echo ; done

[{"descriptionHtml": "

˓→Wikipedia Summary: The Comedy of Errors is one of William Shakespeare's

˓→early plays. It is his shortest and one of his most farcical comedies, with a major

˓→part of the humour coming from slapstick and mistaken identity, in addition to puns

˓→and word play.", "id": 0, "title": "The Comedy of Errors"}]

{"publisher": "PublisherA", "language": "English", "author": "William Shakespeare",

˓→"id": 0, "ISBN-10": "1234567890", "ISBN-13": "123-1234567890", "year": 1595, "type

˓→": "paperback", "pages": 200}

{"reviews": [{"reviewer": "Reviewer1", "rating": {"color": "black", "stars": 5}, "text

˓→": "An extremely entertaining play by Shakespeare. The slapstick humour is

˓→refreshing!"}, {"reviewer": "Reviewer2", "rating": {"color": "black", "stars": 4},

˓→"text": "Absolutely fun and entertaining. The play lacks thematic depth when

˓→compared to other plays by Shakespeare."}], "id": "0"}

{"ratings": {"Reviewer2": 4, "Reviewer1": 5}, "id": 0}

We realized that the REST API to get the book reviews and ratings was meant only for consumption by other internal services, and will be blocked from external clients using the updated Layer-7 CiliumNetworkPolicy in productpage v2, i.e. only the /api/v1/products and /api/v1/products/ HTTP URLs will be whitelisted: apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: productpage-v2 namespace: default specs: - endpointSelector: matchLabels: "k8s:app": productpage "k8s:version": v2 ingress: - toPorts: - ports: - port:"9080" protocol: TCP rules: http: - method: GET path:"/" - method: GET path:"/index.html" - method: POST path:"/login" - method: GET path:"/logout" - method: GET path:"/productpage" - method: GET path:"/api/v1/products" - method: GET path:"/api/v1/products/[0-9] *" # - method: GET

2.2. Getting Started Using Istio 29 Cilium Documentation, Release 1.0.0-rc9

# path: "/api/v1/products/[0-9]*/reviews" # - method: GET # path: "/api/v1/products/[0-9]*/ratings"

Create the productpage v2 service and its updated CiliumNetworkPolicy and delete productpage v1: $ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes-istio/bookinfo-productpage-v2.yaml | \ istioctl kube-inject --injectConfigMapName istio-inject -f - | \ sed -f ./cilium-kube-inject.sed | \ kubectl create -f - ciliumnetworkpolicy "productpage-v2" created deployment "productpage-v2" created

$ kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/HEAD/

˓→examples/kubernetes-istio/bookinfo-productpage-v1.yaml Check the progress of the deployment (every service should have an AVAILABLE count of 1):

$ kubectl get deployments -n default NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE details-v1 1 1 1 1 15m productpage-v2 1 1 1 1 1m ratings-v1 1 1 1 1 10m reviews-v1 1 1 1 1 15m reviews-v2 1 1 1 1 10m

Check that the product REST API is still accessible, and that Cilium now denies at Layer-7 any access to the reviews and ratings REST API:

$ export PRODUCTPAGE=`minikube service productpage -n default --url` $ for APIPATH in /api/v1/products /api/v1/products/0 /api/v1/products/0/reviews /api/

˓→v1/products/0/ratings; do echo ; curl -s -S "${PRODUCTPAGE}${APIPATH}" ; echo ; done

[{"descriptionHtml": "

˓→Wikipedia Summary: The Comedy of Errors is one of William Shakespeare's

˓→early plays. It is his shortest and one of his most farcical comedies, with a major

˓→part of the humour coming from slapstick and mistaken identity, in addition to puns

˓→and word play.", "id": 0, "title": "The Comedy of Errors"}]

{"publisher": "PublisherA", "language": "English", "author": "William Shakespeare",

˓→"id": 0, "ISBN-10": "1234567890", "ISBN-13": "123-1234567890", "year": 1595, "type

˓→": "paperback", "pages": 200}

Access denied

This demonstrated that requests to the /api/v1/products//reviews and /api/v1/products/ /ratings URIs now result in Cilium returning HTTP 403 Forbidden HTTP responses. productpage v2 also implements an authorization audit logging. On every user login or logout, it produces into Kafka topic authaudit a JSON-formatted message which contains the following information: • event: login or logout • username

30 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

• client IP address • timestamp To observe the Kafka messages sent by productpage, we will run an additional authaudit-logger service. This service fetches and prints out all messages from the authaudit Kafka topic. Start this service: $ curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/

˓→kubernetes-istio/authaudit-logger-v1.yaml | \ istioctl kube-inject --injectConfigMapName istio-inject -f - | \ sed -f ./cilium-kube-inject.sed | \ kubectl apply -f - Check the progress of the deployment (every service should have an AVAILABLE count of 1):

$ kubectl get deployments -n default NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE authaudit-logger-v1 1 1 1 1 23s details-v1 1 1 1 1 16m productpage-v2 1 1 1 1 2m ratings-v1 1 1 1 1 11m reviews-v1 1 1 1 1 16m reviews-v2 1 1 1 1 11m

Every login and logout on the product page will result in a line in this service’s log:

$ export POD_LOGGER_V1=`kubectl get pods -n default -l app=authaudit-logger,

˓→version=v1 -o jsonpath='{.items[0].metadata.name}'` $ kubectl logs ${POD_LOGGER_V1} -c authaudit-logger ... {"timestamp": "2017-12-04T09:34:24.341668", "remote_addr": "10.15.28.238", "event":

˓→"login", "user": "richard"} {"timestamp": "2017-12-04T09:34:40.943772", "remote_addr": "10.15.28.238", "event":

˓→"logout", "user": "richard"} {"timestamp": "2017-12-04T09:35:03.096497", "remote_addr": "10.15.28.238", "event":

˓→"login", "user": "gilfoyle"} {"timestamp": "2017-12-04T09:35:08.777389", "remote_addr": "10.15.28.238", "event":

˓→"logout", "user": "gilfoyle"}

As you can see, the user-identiﬁable information sent by productpage in every Kafka message is sensitive, so access to this Kafka topic must be protected using Cilium. The CiliumNetworkPolicy conﬁgured on the Kafka broker enforces that: • only productpage v2 is allowed to produce messages into the authaudit topic; • only authaudit-logger can fetch messages from this topic; • no service can access any other topic.

apiVersion:"cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: kafka-authaudit specs: - endpointSelector: matchLabels: "k8s:app": kafka ingress: - fromEndpoints: - matchLabels:

2.2. Getting Started Using Istio 31 Cilium Documentation, Release 1.0.0-rc9

"k8s:app": productpage "k8s:version": v2 toPorts: - ports: - port:"9092" protocol: TCP rules: kafka: - apiKey:"produce" topic:"authaudit" - apiKey:"apiversions" - apiKey:"metadata" - apiKey:"heartbeat" - fromEndpoints: - matchLabels: app: kafka - fromEndpoints: - matchLabels: "k8s:app": authaudit-logger toPorts: - ports: - port:"9092" protocol: TCP rules: kafka: - apiKey:"fetch" topic:"authaudit" - apiKey:"apiversions" - apiKey:"metadata" - apiKey:"findcoordinator" - apiKey:"joingroup" - apiKey:"leavegroup" - apiKey:"syncgroup" - apiKey:"offsets" - apiKey:"offsetcommit" - apiKey:"offsetfetch" - apiKey:"heartbeat"

Check that Cilium prevents the authaudit-logger service from writing into the authaudit topic (enter a message followed by ENTER, e.g. test message):

$ export POD_LOGGER_V1=`kubectl get pods -n default -l app=authaudit-logger,

˓→version=v1 -o jsonpath='{.items[0].metadata.name}'` $ kubectl exec ${POD_LOGGER_V1} -c authaudit-logger -ti -- /opt/kafka_2.11-0.10.1.0/

˓→bin/kafka-console-producer.sh --broker-list=kafka:9092 --topic=authaudit test message [2017-12-07 02:13:47,020] ERROR Error when sending message to topic authaudit with

˓→key: null, value: 12 bytes with error: (org.apache.kafka.clients.producer.internals.

˓→ErrorLoggingCallback) org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access

˓→topics: [authaudit]

This demonstrated that Cilium sent a response with an authorization error for any Produce request from this service. Create another topic named credit-card-payments, meant to transmit highly-sensitive credit card payment requests:

32 Chapter 2. Getting Started Guides Cilium Documentation, Release 1.0.0-rc9

$ kubectl exec kafka-v1-0 -c kafka -- bash -c '/opt/kafka_2.11-0.10.1.0/bin/kafka-

˓→topics.sh --zookeeper localhost:2181/kafka --create --topic credit-card-payments --

˓→partitions 1 --replication-factor 1' Created topic "credit-card-payments".

Check that Cilium prevents the authaudit-logger service from fetching messages from this topic:

$ export POD_LOGGER_V1=`kubectl get pods -n default -l app=authaudit-logger,

˓→version=v1 -o jsonpath='{.items[0].metadata.name}'` $ kubectl exec ${POD_LOGGER_V1} -c authaudit-logger -ti -- /opt/kafka_2.11-0.10.1.0/

˓→bin/kafka-console-consumer.sh --bootstrap-server=kafka:9092 --topic=credit-card-

˓→payments [2017-12-07 03:08:54,513] WARN Not authorized to read from topic credit-card-payments.

˓→ (org.apache.kafka.clients.consumer.internals.Fetcher) [2017-12-07 03:08:54,517] ERROR Error processing message, terminating consumer

˓→process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access

˓→topics: [credit-card-payments] Processed a total of 0 messages

This demonstrated that Cilium sent a response with an authorization error for any Fetch request from this service for any topic other than authaudit.

2.2.7 Step 6: Clean Up

You have now installed Cilium and Istio, deployed a demo app, and tested both Cilium’s L3-L7 network security policies and Istio’s service route rules. To clean up, run:

$ minikube delete

After this, you can re-run the tutorial from Step 0.

2.3 Getting Started Securing Kafka

This document serves as an introduction to using Cilium to enforce Kafka-aware security policies. It is a detailed walk-through of getting a single-node Cilium environment running on your machine. It is designed to take 15-30 minutes. If you haven’t read the Introduction to Cilium yet, we’d encourage you to do that ﬁrst. The best way to get help if you get stuck is to ask a question on the Cilium Slack channel. With Cilium contributors across the globe, there is almost always someone available to help.

2.3.1 Step 0: Install kubectl & minikube

2.3. Getting Started Securing Kafka 33 Cilium Documentation, Release 1.0.0-rc9