OvS manipulation with Go at DigitalOcean Matt Layher, Software Engineer OvSCon 2017 ● Software Engineer at DO for ~3.5 years ● Focused on virtual networking primitives ● Huge fan of Go programming language ● GitHub + Twitter: @mdlayher digitalocean.com Cloud computing designed for developers ● Cloud provider with focus on simplicity ● “Droplet” product is a virtual machine ● Compute, storage, networking, and monitoring primitives ○ Load Balancers as a Service ○ Floating IPs ○ Cloud Firewalls (learn more at Kei Nohguchi’s talk!) digitalocean.com DO is powered by Open vSwitch! ● 10,000+ of instances of OvS! ● One of the most crucial components in our entire stack. digitalocean.com Open vSwitch at DigitalOcean: The Past digitalocean.com Open vSwitch and Perl ● Events (create droplet, power on, etc.) reach a hypervisor ● Perl event handlers pick up events and performs a series of actions to prepare a droplet ● Perl builds flow strings and calls ovs-ofctl digitalocean.com Building flows with Perl my $ipv4_flow_rules = [ [ # Flow priority. 2020, # Hash of flow matches. { dl_src => $mac, in_port => $ovs_port_id, ip => undef, nw_src => "${ip}/32", }, # Literal string of flow actions. "mod_vlan_vid:${ipv4vlan},resubmit(,1)" ], # … many more flows ] digitalocean.com Applying flows with Perl # Build comma-separated matches from hash. my $flow = _construct_flow_string($flow_hash); # Build the flow string with usual fields. my $cmd = "priority=${priority},idle_timeout=${timeout},${flow},actions=${actions}"; # Once a flow is added, we need a way to delete it later on! if ($add_delete_hook && defined($delete_hook_handle)) { # Flows written into a libvirt hook to be deleted later. if (_write_flow_to_be_deleted($bridge, $delete_hook_handle, $flow) != PASS) { return FAIL; } } # Shell out to ovs-ofctl and do the work! return _run_ovs_cmd("ovs-ofctl add-flow ${bridge} '${cmd}'"); digitalocean.com Conclusions: Open vSwitch and Perl ● Pros: ○ Straightforward code ○ Perl is well-suited to manipulating strings ● Cons: ○ No sanity checking (other than OvS applying the flow) ○ Lacking test coverage ○ libvirt flow deletion hooks for individual flows proved problematic ○ Shell out once per flow; no atomicity digitalocean.com Open vSwitch at DigitalOcean: The Present digitalocean.com Open vSwitch and Go ● Events reach a hypervisor ● Perl/Go (it depends) systems perform a series of actions to prepare a droplet ● Go builds flow strings and calls ovs-ofctl digitalocean.com package ovs Package ovs is a client library for Open vSwitch which enables programmatic control of the virtual switch. digitalocean.com package ovs ● Go package for manipulating Open vSwitch ● No DigitalOcean-specific code! ● Open source (soon)! ○ https://github.com/digitalocean/go-openvswitch digitalocean.com Building flows with Go flow := &ovs.Flow{ // Commonly used flow pieces are struct fields. Priority: 2000, Protocol: ovs.ProtocolIPv4, InPort: droplet.PortID, // Matches and Actions are Go interfaces; functions create a // type that implements the interface. Matches: []ovs.Match{ ovs.DataLinkSource(r.HardwareAddr.String()), ovs.NetworkSource(r.IP.String()), }, Actions: []ovs.Action{ ovs.Resubmit(0, tableL2Rewrite), }, } digitalocean.com Building flows with Go (cont.) ● Our example flow marshaled to textual format: priority=2000,ip,in_port=1,dl_src=de:ad:be:ef:de:ad, \ nw_src=192.168.1.1,table=0,idle_timeout=0,actions=resubmit(,10) ● Mostly string manipulation behind the scenes; just like Perl ● Go is statically typed, reducing chance of programmer errors ● Can validate each match and action for correctness without hitting OvS digitalocean.com The ovs.Match Go interface type Match interface { // MarshalText() (text []byte, err error) encoding.TextMarshaler } ● Because of the way Go interfaces work, any type with a MarshalText method can be used as an ovs.Match ● The error return value can be used to catch any bad input ● ovs.Action’s definition is identical digitalocean.com The ovs.Client Go type // Configure ovs.Client with our required OpenFlow flow format and protocol. client := ovs.New(ovs.FlowFormat("OXM-OpenFlow14"), ovs.Protocols("OpenFlow14")) // $ ovs-vsctl --may-exist add-br br0 err := client.VSwitch.AddBridge("br0") // $ ovs-ofctl add-flow --flow-format=OXM-OpenFlow14 --protocols=OpenFlow14 br0 ${flow} err = client.OpenFlow.AddFlow("br0", exampleFlow()) ● ovs.Client is a wrapper around ovs-vsctl and ovs-ofctl commands ● ovs.New constructor uses “functional options” pattern for sane defaults ● We can still only apply one flow at a time… right? digitalocean.com ovs.Client flow bundle transactions // Assume we want to apply a new flow set and remove old one. add, remove := newFlows(), oldFlows() // We can apply all of these flows atomically using a flow bundle! err := client.OpenFlow.AddFlowBundle(bridge, func(tx *ovs.FlowTransaction) error { // $ echo -e “delete priority=10,cookie=1,actions=drop\n” >> mem tx.Delete(remove...) // $ echo -e “add priority=10,cookie=1,actions=output:1\n” >> mem tx.Add(add...) // $ cat mem | ovs-ofctl --bundle add-flow --flow-format=OXM-OpenFlow14 --protocols=OpenFlow14 br0 - return tx.Commit() }) ● Flow bundle stored in memory, passed directly from buffer to ovs-ofctl ● Modifications are processed by OvS in a single, atomic transaction digitalocean.com package hvflow Package hvflow provides Open vSwitch flow manipulation at the hypervisor level. digitalocean.com package hvflow ● DigitalOcean-specific wrapper for package ovs ● Provides higher-level constructs, such as: ○ enable public IPv4 and IPv6 connectivity ○ reset and apply security policies ○ disable all connectivity digitalocean.com The hvflow.Client Go type // Configure hvflow.Client to modify bridge “br0”. client, err := hvflow.NewClient("br0", ovs.New( ovs.FlowFormat("OXM-OpenFlow14"), ovs.Protocols("OpenFlow14"), )) ● hvflow.Client is a high-level wrapper around ovs.Client ● hvflow.NewClient constructor uses “functional options” pattern for sane defaults digitalocean.com Network parameters - “netparams” ● Encode all necessary information about { "droplet_id": 1, how to enable networking for a given "vnics": [ VNIC { "mac": "de:ad:be:ef:de:ad", ● Carries IPv4 and IPv6 addresses, firewall "enabled": 1, configurations, floating IPs… "name": "tapext1", "interface_type": "public", "addresses": { "ipv4": [ { ● netparams used to configure OvS with "ip_address": "10.0.0.10", "masklen": 20, hvflow.Client.Transaction method "gateway": "10.0.0.1" } ] } } ] } digitalocean.com hvflow.Client transactions // Assume a netparams structure similar to the one just shown. params, ifi := networkParams(), "public" err := client.Transaction(ctx, func(ctx context.Context, tx *hvflow.Transaction) error { // Convert netparams into hvflow simplified representation. req, ok, err := hvflow.NewIPv4Request(params, ifi) if err != nil { return err } if ok { // If IPv4 configuration present, apply it! if err := tx.EnableIPv4(ctx, req); err != nil { return wrapError(err, "failed to enable IPv4 networking") } } return tx.Commit() }) digitalocean.com hvflow.Client transactions (cont.) ● Each operation accumulates additional // IPv4 configuration. flows to be applied within the context of err := tx.EnableIPv4(ctx, req4) the transaction. // IPv6 configuration. ● Flow set sizes can vary from a couple err = tx.EnableIPv6(ctx, req6) dozen to several hundred flows. // Floating IPv4 configuration. err = tx.EnableFloatingIPv4(ctx, req4F) ● Flows are always applied using a flow // Disable networking on an interface. bundle; non-transactional err = tx.Disable(ctx, 10, "public") hvflow.Client API was deleted! // Apply flow set to OvS. err = tx.Commit() digitalocean.com The hvflow.Cookie Go interface type Cookie interface { Marshal() (uint64, error) Unmarshal(i uint64) error } ● Cookie structs packed and unpacked from uint64 form ● Cookies are versioned using a 4-bit identifier ● Used to store identification metadata about a flow ● Easy deletions of flows; much simpler deletion hooks with libvirt digitalocean.com hvflowctl and hvflowd gRPC client and server that manipulate Open vSwitch digitalocean.com hvflowctl and hvflowd ● gRPC client and server written in Go ● hvflowctl passes netparams and other data to hvflowd ● hvflowd manipulates OvS flows via hvflow package digitalocean.com hvflowd’s gRPC interface ● gRPC uses protocol buffers // The set of RPCs that make up the “HVFlow” service. service HVFlow { (“protobuf”) for RPC // Add flows using the parameters specified in request. communication rpc AddFlows(AddFlowsRequest) returns (AddFlowsResponse); ● RPCs accept one message type } and return another // RPC parameters encoded within a request message. ● netparamspb package for message AddFlowsRequest { // netparams have a protobuf representation too. encoding netparams in netparamspb.NetworkParams network_params = 1; protobuf string interface_type = 2; } // No need to return any data on success. message AddFlowsResponse {} digitalocean.com hvflowd AddFlows RPC // AddFlows requests that hvflowd add flows to enable connectivity for one or more droplets. func (s *server) AddFlows(ctx context.Context, req *hpb.AddFlowsRequest) (*hpb.AddFlowsResponse, error) { // Fetch netparams from gRPC request message. params := req.GetNetworkParams() // Perform the necessary transaction logic to establish connectivity. err := s.hvflowc.Transaction(ctx, func(ctx context.Context, tx *hvflow.Transaction) error { // hvflow.Client transaction logic … return tx.Commit() }) // Inform the caller if the request was successful. return &hvflowpb.AddFlowsResponse{},