Let Your Software Supply Chain Ride with Kubernetes CI/CD
Total Page:16
File Type:pdf, Size:1020Kb
Let Your Software Supply Chain Ride with Kubernetes CI/CD Ricardo Aravena @raravena80 LISA 19 Who Am I? Ricardo Aravena (rico) Work @Rakuten Cloud Operations & Kubernetes Kata Containers Contributor @raravena80 K8s & CI/CD & GitOps @ Rakuten Where? 05 DBs and Big Data Almost everywhere Standalone 01 04 Tools K8s in prod Flux/Helm ~1 year Terraform Multiple services 02 03 CI/CD Jenkins @raravena80 Outline ✗ CI/CD History and Why GitOps? ✗ Tools and Security ✗ Developer -> Draft, Flux, Others ✗ Image Building -> Buildkit, Bazel ✗ Templating -> Helm, Kustomize ✗ Specifically Sec -> Notary, TUF, Trivy ✗ CI/CD -> JenkinsX, Tekton ✗ GitOps Future ✗ Takeaways @raravena80 CI/CD History 2003 2011 2013 2015 2017 buildbot Jenkins/Travis Cloud CI/CD Kubernetes GitOps First release First releases CircleCI, 1.0 Release Kubernetes AppVeyor, etc CI/CD @raravena80 Version Control Use a tool Both Infra & Code Compare current state to desired state Why GitOps? Automate Deployments Gain reversibility, have an audit trail and transparency @raravena80 GitOps Infra Kubernetes Clusters CI dev prod Config Code Repos config Repos sync tool Prod PRs dev sync tool port-forward remote apps pre-prod dev & testing stream remote logs @raravena80 Why Care About Security? @raravena80 GitOps Infra & Kubernetes Clusters CI dev prod Config Code Repos config Repos sync tool Prod PRs dev sync tool port-forward remote apps pre-prod dev & testing stream remote logs @raravena80 GitOps Vulnerabilities ✗ Package dependencies in OSS ✗ Glibc, Bash -> Shellshock ✗ Container backdoors ✗ CVE-2019-5736: runc container breakout ✗ Fake downloads or typosquatting ✗ Trojans ✗ Kubernetes ✗ CVE-2018-1002105 - Privilege escalation ✗ Tools ✗ Developer, image building, CI/CD @raravena80 GitOps Developer Tools & Security @raravena80 Draft Who? ✗ MS Open Source ✗ App development and What? deployment How? ✗ Draft packs/cli Local and Remote mgmt ✗ Yes - ‘draft up/connect’ @raravena80 https://github.com/Azure/draft Draft Languages @raravena80 https://github.com/Azure/draft Draft Languages & @raravena80 14 https://github.com/Azure/draft Draft Pros Cons ● Local development ● Supports many languages ● Uses git and its auth model ○ Security con ● Integrated docker image ● Requires tiller -- Helm v2 builder ● 0.16.x experimental release ● Support: active community ● No native RBAC @raravena80 https://github.com/Azure/draft Flux Who? ✗ Weaveworks What? ✗ App Deployment ✗ git push How? ✗ fluxctl cmd ✗ git for local Local and remote mgmt ✗ fluxctl for remote @raravena80 https://github.com/fluxcd/flux Flux Pros Cons ✗ Has its own controller and ✗ No native RBAC CRD ✗ Flux container image needs ✗ Leverages git auth signing ✗ Mature 1.15.x release ✗ Automatic sync could be ✗ Support: active community exposed to MIM ✗ No local development @raravena80 https://github.com/fluxcd/flux Flux @raravena80 https://github.com/fluxcd/flux Flux & @raravena80 https://github.com/fluxcd/flux Other GitOps Developer Tools Skaffold garden gitkube DevSpace Tilt @raravena80 https://github.com/ramitsurana/awesome-kubernetes#developer-platform Help! @raravena80 Image Building Tools @raravena80 Kaniko Build Support for container Google Cloud local fs, S3, What? images in K8s GS and Others Kaniko Build context Unnested executor How? (directory) containers image Use gVisor or Docker with Pod Kata seccomp + Security? SecurityPolicies Containers Apparmor @raravena80 https://github.com/GoogleContainerTools/kaniko BuildKit Build OCI images Use Dockerfiles or buildpacks What? Buildkit daemon Supports containerd or runc backends buildctl cli How? Use other tools built on top (img) unpriviledged Security? rootless builds clone SSH repo in build @raravena80 https://github.com/moby/buildkit Bazel ✗ Build images What? ✗ Build fast ✗ Bazel rules How? ✗ Knative templates ✗ Deterministic Security?✗ Reproducible ✗ Unprivileged @raravena80 https://bazel.build Other Build Tools kpack/buildpacks buildah img Knative Build Umoci/Orca @raravena80 Templating tools @raravena80 Helm ✗ Manage packages for K8s clusters What? ✗ Go templating ✗ Extendable w/Lua (v3) ✗ Helm CLI How? ✗ Tiller (v2) Going away with v3 ✗ Helm charts ✗ SSL ✗ Other mechanism for secrets Security? ✗ No security per se (helps with workflow) ✗ Tie together with other tools @raravena80 https://helm.sh/ Kustomize What? Change existing k8s manifests Leaves original YAML untouched How? kustomize.yaml kustomize cli Many fields to change k8s manifests Supported by Flux Security? Leverages kubectl v1.14 security Provides secret generator @raravena80 Security Tools @raravena80 TUF/Notary ✗ Moby/Docker ✗ TUF ✗ Client/Server ✗ Standard ✗ Signer service ✗ Notary ✗ Add to CI/CD ✗ Metadata ✗ Publishers can sign signatures content oine ✗ Allows delegation Who? What? How? @raravena80 https://github.com/theupdateframework Trivy Who? ✗ Aqua Security What? ✗ Container image scanning ✗ Application audit ✗ Policy enforcement How? ✗ Trivy cli ✗ Using tar or local docker container ✗ Using CI/CD tool @raravena80 https://github.com/aquasecurity/trivy in-toto Who? IBM Open Source What? Secure Workflows Secure/Verify Artifacts Python CLI How? Asymmetric Creates links Can work with Verification of Can be used Layouts crypto between steps containers tampering with CI/CD tool @raravena80 Other Security Tools Falco - Container Activity Monitor Harbor - Secure registry (Notary) Anchore, Aqua, Clair, Dagda - Image scanning Grafeas - Signing Kritis, Portieris - Admission Controller Kube hunter - K8s Vulnerabilities Open Policy Agent - Config enforcer Trireme - Netpol mgmt Image encryption - Encrypt cont imgs Tern, Snyk - Package Compliance @raravena80 CI/CD Tools @raravena80 JenkinsX ● Jenkins ● ChartMuseum (Helm) What? ● Nexus ● Monocular (Chart disc) ● Draft ● Skaffold ● jx command line How? ● jenkins-x.yml file for pipelines ● Leverages draft to manage code ● Skaffold to build and push images Security? ● Auto upgrade w/GitOps and Tekton ● Jenkins built in sec @raravena80 https://jenkins-x.io/ Tekton K8s CRDs Tasks K8s Native K8s native Pipelines • Pod Security How? What? Policies CI/CD Resources • Network K8s Jobs • Images ecurity? Policies S • Git repos • git SSL support @raravena80 https://tekton.dev Other CI/CD Tools Spinnaker Argo CD Drone CI GoCD Gitlab TeamCity Screwdriver CD Rundeck Habitat @raravena80 Secure Software Supply Chain @raravena80 Jenkins X @raravena80 https://jenkins-x.io/ Jenkins X + @raravena80 https://jenkins-x.io/ GitOps Future @raravena80 42 GitOps Future... ✗ GitOps tools self updates ✗ Ie. Devs can’t build if Draft/Flux not up to date ✗ Container runtime support ✗ Kata, gVisor for GitOps and image building ✗ GitOps with Ignite and VMs ✗ New configuration languages. I.e Dhall, Cue ✗ Integration with Open Policy Agent ✗ K8s admission (Gatekeeper) + Uniform configs ✗ GitOps for Kubernetes cluster management ✗ Integrations, integrations and integrations @raravena80 Takeaways @raravena80 Takeaways... Separate environments ✗ Dev, QA, Stage, Prod Security first! runc ✗ Patch regularly!! ✗ Use seccomp profiles Alternatively use Kata, or gVisor for pipelines ✗ Check performance and compatibilities Always use unprivileged to build cont. images ✗ Use rootless with BuildKit for extra security @raravena80 https://sysdig.com/blog/33-kubernetes-security-tools/ Takeaways... Use GitOps CI/CD Tool to upgrade packages ✗ Container packages and tools themselves Leverage in-toto (or Grafeas, Notary) Scan your images - Trivy/Anchore/Claire/Dagda Use K8s mechanisms ✗ Admission ctrl (Kritis/Portieris), Authentication w/RBAC Configure Network Policies in K8s ✗ Trireme, or overlay @raravena80 https://sysdig.com/blog/33-kubernetes-security-tools/ Resources ✘ GitOps ○ https://www.weave.works/technologies/gitops/ ✘ Security ○ K8s Sec Tools https://sysdig.com/blog/33-kubernetes-security-tools/ ○ Harbor https://github.com/goharbor/harbor ○ In-toto https://in-toto.github.io/ ○ Trivy https://github.com/aquasecurity/trivy ✘ CI/CD ○ JenkinsX https://jenkins.io/projects/jenkins-x/ ○ Tekton https://tekton.dev/ ○ Spinnaker https://www.spinnaker.io/ @raravena80 Thanks! You can find me at: ✘ @raravena80 ✘ https://blog.serverbooter.com @raravena80.