sign in sign up
<<
Home , Bazel (software)

Let Your Software Supply Chain Ride with CI/CD

Ricardo Aravena @raravena80 LISA 19 Who Am I? Ricardo Aravena (rico) Work @Rakuten Cloud Operations & Kubernetes Kata Containers Contributor

@raravena80 K8s & CI/CD & GitOps @ Rakuten

Where? 05 DBs and Big Data Almost everywhere Standalone

01 04 Tools K8s in prod Flux/Helm ~1 year Terraform Multiple services

02 03 CI/CD Jenkins

@raravena80 Outline

✗ CI/CD History and Why GitOps? ✗ Tools and Security ✗ Developer -> Draft, Flux, Others ✗ Image Building -> Buildkit, Bazel ✗ Templating -> Helm, Kustomize ✗ Specifically Sec -> Notary, TUF, Trivy ✗ CI/CD -> JenkinsX, Tekton ✗ GitOps Future ✗ Takeaways

@raravena80 CI/CD History

2003 2011 2013 2015 2017

buildbot Jenkins/Travis Cloud CI/CD Kubernetes GitOps

First release First releases CircleCI, 1.0 Release Kubernetes AppVeyor, etc CI/CD

@raravena80 Version Control Use a tool

Both Infra & Code Compare current state to desired state Why GitOps?

Automate Deployments

Gain reversibility, have an audit trail and transparency

@raravena80 GitOps Infra

Kubernetes Clusters CI

dev

prod

Config Code Repos config Repos sync tool

Prod PRs

dev sync tool port-forward remote apps pre-prod dev & testing stream remote logs

@raravena80 Why Care About Security?

@raravena80 GitOps Infra &

Kubernetes Clusters CI

dev

prod

Config Code Repos config Repos sync tool

Prod PRs

dev sync tool port-forward remote apps pre-prod dev & testing stream remote logs

@raravena80 GitOps Vulnerabilities

✗ Package dependencies in OSS ✗ Glibc, Bash -> Shellshock ✗ Container backdoors ✗ CVE-2019-5736: runc container breakout ✗ Fake downloads or typosquatting ✗ Trojans ✗ Kubernetes ✗ CVE-2018-1002105 - Privilege escalation ✗ Tools ✗ Developer, image building, CI/CD

@raravena80 GitOps Developer Tools & Security

@raravena80 Draft

Who? ✗ MS Open Source

✗ App development and What? deployment

How? ✗ Draft packs/cli

Local and Remote mgmt ✗ Yes - ‘draft up/connect’

@raravena80 https://github.com/Azure/draft Draft Languages

@raravena80 https://github.com/Azure/draft Draft Languages &

@raravena80 14 https://github.com/Azure/draft Draft

Pros Cons ● Local development ● Supports many languages ● Uses git and its auth model ○ Security con ● Integrated docker image ● Requires tiller -- Helm v2 builder ● 0.16.x experimental release ● Support: active community ● No native RBAC

@raravena80 https://github.com/Azure/draft Flux

Who? ✗ Weaveworks

What? ✗ App Deployment

✗ git push How? ✗ fluxctl cmd ✗ git for local Local and remote mgmt ✗ fluxctl for remote

@raravena80 https://github.com/fluxcd/flux Flux

Pros Cons ✗ Has its own controller and ✗ No native RBAC CRD ✗ Flux container image needs ✗ Leverages git auth signing ✗ Mature 1.15.x release ✗ Automatic sync could be ✗ Support: active community exposed to MIM ✗ No local development

@raravena80 https://github.com/fluxcd/flux Flux

@raravena80 https://github.com/fluxcd/flux Flux &

@raravena80 https://github.com/fluxcd/flux Other GitOps Developer Tools

Skaffold garden gitkube DevSpace Tilt

@raravena80 https://github.com/ramitsurana/awesome-kubernetes#developer-platform Help!

@raravena80 Image Building Tools

@raravena80 Kaniko

Build Support for container Cloud local fs, S3, What? images in K8s GS and Others

Kaniko Build context Unnested executor How? (directory) containers image

Use gVisor or Docker with Pod Kata seccomp + Security? SecurityPolicies Containers Apparmor

@raravena80 https://github.com/GoogleContainerTools/kaniko BuildKit

Build OCI images

Use Dockerfiles or buildpacks What? Buildkit daemon

Supports containerd or runc backends

buildctl cli How? Use other tools built on top (img)

unpriviledged

Security? rootless builds

clone SSH repo in build

@raravena80 https://github.com/moby/buildkit Bazel

✗ Build images What? ✗ Build fast

✗ Bazel rules How? ✗ Knative templates

✗ Deterministic Security?✗ Reproducible ✗ Unprivileged

@raravena80 https://bazel.build Other Build Tools

kpack/buildpacks buildah img Knative Build Umoci/Orca

@raravena80 Templating tools

@raravena80 Helm

✗ Manage packages for K8s clusters What? ✗ Go templating ✗ Extendable w/Lua (v3)

✗ Helm CLI How? ✗ Tiller (v2) Going away with v3 ✗ Helm charts

✗ SSL ✗ Other mechanism for secrets Security? ✗ No security per se (helps with workflow) ✗ Tie together with other tools

@raravena80 https://helm.sh/ Kustomize

What? Change existing k8s manifests

Leaves original YAML untouched

How? kustomize.yaml kustomize cli Many fields to change k8s manifests Supported by Flux Security? Leverages kubectl v1.14 security

Provides secret generator

@raravena80 Security Tools

@raravena80 TUF/Notary

✗ Moby/Docker ✗ TUF ✗ Client/Server ✗ Standard ✗ Signer service ✗ Notary ✗ Add to CI/CD ✗ Metadata ✗ Publishers can sign signatures content oine ✗ Allows delegation

Who? What? How?

@raravena80 https://github.com/theupdateframework Trivy

Who? ✗ Aqua Security What? ✗ Container image scanning ✗ Application audit ✗ Policy enforcement How?

✗ Trivy cli ✗ Using tar or local docker container ✗ Using CI/CD tool

@raravena80 https://github.com/aquasecurity/trivy in-toto

Who?

IBM Open Source

What?

Secure Workflows Secure/Verify Artifacts Python CLI

How? Asymmetric Creates links Can work with Verification of Can be used Layouts crypto between steps containers tampering with CI/CD tool

@raravena80 Other Security Tools

Falco - Container Activity Monitor Harbor - Secure registry (Notary) Anchore, Aqua, Clair, Dagda - Image scanning Grafeas - Signing Kritis, Portieris - Admission Controller Kube hunter - K8s Vulnerabilities Open Policy Agent - Config enforcer Trireme - Netpol mgmt Image encryption - Encrypt cont imgs Tern, Snyk - Package Compliance

@raravena80 CI/CD Tools

@raravena80 JenkinsX

● Jenkins ● ChartMuseum (Helm) What? ● Nexus ● Monocular (Chart disc) ● Draft ● Skaffold

● jx command line How? ● jenkins-x.yml file for pipelines ● Leverages draft to manage code ● Skaffold to build and push images

Security? ● Auto upgrade w/GitOps and Tekton ● Jenkins built in sec

@raravena80 https://jenkins-x.io/ Tekton

K8s CRDs Tasks K8s Native

K8s native Pipelines • Pod Security How?

What? Policies CI/CD Resources • Network

K8s Jobs • Images ecurity? Policies S • Git repos • git SSL support

@raravena80 https://tekton.dev Other CI/CD Tools

Spinnaker Argo CD Drone CI GoCD Gitlab TeamCity Screwdriver CD Rundeck Habitat

@raravena80 Secure Software Supply Chain

@raravena80 Jenkins X

@raravena80 https://jenkins-x.io/ Jenkins X +

@raravena80 https://jenkins-x.io/ GitOps Future

@raravena80 42 GitOps Future...

✗ GitOps tools self updates ✗ Ie. Devs can’t build if Draft/Flux not up to date ✗ Container runtime support ✗ Kata, gVisor for GitOps and image building ✗ GitOps with Ignite and VMs ✗ New configuration languages. I.e Dhall, Cue ✗ Integration with Open Policy Agent ✗ K8s admission (Gatekeeper) + Uniform configs ✗ GitOps for Kubernetes cluster management ✗ Integrations, integrations and integrations

@raravena80 Takeaways

@raravena80 Takeaways...

Separate environments ✗ Dev, QA, Stage, Prod

Security first!

runc ✗ Patch regularly!! ✗ Use seccomp profiles Alternatively use Kata, or gVisor for pipelines ✗ Check performance and compatibilities

Always use unprivileged to build cont. images ✗ Use rootless with BuildKit for extra security

@raravena80 https://sysdig.com/blog/33-kubernetes-security-tools/ Takeaways...

Use GitOps CI/CD Tool to upgrade packages ✗ Container packages and tools themselves

Leverage in-toto (or Grafeas, Notary)

Scan your images - Trivy/Anchore/Claire/Dagda

Use K8s mechanisms ✗ Admission ctrl (Kritis/Portieris), Authentication w/RBAC

Configure Network Policies in K8s ✗ Trireme, or overlay

@raravena80 https://sysdig.com/blog/33-kubernetes-security-tools/ Resources

✘ GitOps ○ https://www.weave.works/technologies/gitops/ ✘ Security ○ K8s Sec Tools https://sysdig.com/blog/33-kubernetes-security-tools/ ○ Harbor https://github.com/goharbor/harbor ○ In-toto https://in-toto.github.io/ ○ Trivy https://github.com/aquasecurity/trivy ✘ CI/CD ○ JenkinsX https://jenkins.io/projects/jenkins-x/ ○ Tekton https://tekton.dev/ ○ Spinnaker https://www.spinnaker.io/

@raravena80 Thanks!

You can find me at: ✘ @raravena80 ✘ https://blog.serverbooter.com

@raravena80