Vulcand REST API Documentation Release 2.0

Vulcand Inc

Nov 07, 2018

Contents

1 Quick Start 3 1.1 Example: setting up Vulcand...... 4

2 User Manual 5 2.1 Glossary...... 5 2.2 Conﬁguration...... 7 2.3 TLS...... 25 2.4 Metrics...... 30 2.5 Logging...... 31 2.6 Process management...... 32 2.7 Installation...... 33

3 Middlewares 35 3.1 Middlewares...... 35 3.2 Middleware Chains...... 35 3.3 Vbundle...... 36

4 API Reference 43 4.1 Status...... 43 4.2 Log severity...... 43 4.3 Host...... 44 4.4 Listener...... 45 4.5 Backend...... 46 4.6 Server...... 47 4.7 Frontend...... 48 4.8 Rate limit...... 50 4.9 Connection limit...... 51

i ii Vulcand REST API Documentation, Release 2.0

Vulcand is a reverse proxy for HTTP API management and microservices. It is inspired by Hystrix. It uses Etcd as a conﬁguration backend, so changes to conﬁguration take effect immediately without restarting the service.

Warning: Status: Under active development. Used at Mailgun on moderate workloads.

Contents 1 Vulcand REST API Documentation, Release 2.0

2 Contents CHAPTER 1

Quick Start

Vulcand uses Etcd as a conﬁguration backend. See Etcd getting started guide for instructions. The easiest way to install Vulcand is to pull the trusted build from the hub.docker.com and launch it in the container:

# download

˓→vulcand from the trusted build docker pull

˓→mailgun/vulcand:v0.8.0-beta.2

# launch vulcand in a container docker run -

˓→p 8182:8182 -p 8181:8181 mailgun/

˓→vulcand:v0.8.0-beta.2 /go/

˓→bin/vulcand --apiInterface=0.0.0.

˓→0 --etcd=http://172.17.42.1:4001

You can check if Vulcand is running by checking the logs of the container:

# check out

˓→the output from the container: docker logs $(docker ps| grep

˓→vulcand| awk '{ print $1 }' )

Dec 25 20:22:24.

˓→002: WARN PID:1[supervisor.

˓→go:350] No frontends found

That was Vulcand complaining that there are no frontends speciﬁed

3 Vulcand REST API Documentation, Release 2.0

1.1 Example: setting up Vulcand

# Upsert

˓→backend and add a server to it etcdctl set /vulcand/backends/

˓→b1/backend '{"Type": "http"}' etcdctl set /vulcand/

˓→backends/b1/servers/srv1'

˓→{"URL": "http://localhost:5000"}'

# Upsert a frontend connected to

˓→backend "b1" that matches path / etcdctl set

˓→/vulcand/frontends/f1/frontend

˓→'{"Type": "http", "BackendId

˓→": "b1", "Route": "Path(`/`)"}'

# Upsert

˓→backend and add a server to it vctl backend upsert -id b1 vctl server upsert -b b1 -id

˓→srv1 -url http://localhost:5000

# Upsert a frontend connected to

˓→backend "b1" that matches path / vctl frontend upsert

˓→-id f1 -b b1 -route 'Path("/")'

# Upsert

˓→backend and add a server to it curl -X POST -H "Content-

˓→Type: application/json" http:/

˓→/localhost:8182/v2/backends\ -d '{"Backend

˓→": {"Id":"b1", "Type":"http"}}' curl -X POST

˓→-H "Content-Type: application/

˓→json" http://localhost:8182/

˓→v2/backends/b1/servers\ -d '{"Server": {"Id":"srv1",

˓→ "URL":"http://localhost:5000"}}'

# Upsert a frontend connected to

˓→backend "b1" that matches path / curl -X POST -H "Content-

˓→Type: application/json" http:/

˓→/localhost:8182/v2/frontends\ -d '{"Frontend": {"Id":"f1

˓→", "Type": "http", "BackendId":

˓→"b1", "Route": "Path(\"/\")"}}'

4 Chapter 1. Quick Start CHAPTER 2

User Manual

2.1 Glossary

Familiarizing with the glossary would help to understand the rest of this guide.

2.1.1 Host

Hostname is deﬁned by incoming Host header. E.g. curl http://example.com/alice generates the following request:

GET /alice HTTP/1.1 User-Agent: curl/7.35.0 Host: example.com

Vulcand hosts contain associated information and settings, such as SNI options and TLS certiﬁcates.

2.1.2 Listener

Listener is a dynamic socket that can be attached or detached to Vulcand without restart. Vulcand can have multiple http and https listeners attached to it, providing service on multiple interfaces and protocols.

2.1.3 Frontend

Frontends match the requests and forward it to the backends. Each frontend deﬁnes a route - a special expression that matches the request, e.g. Path("/v1/path"). Frontends are linked to backend and Vulcand will use the servers from this backend to serve the request.

5 Vulcand REST API Documentation, Release 2.0

2.1.4 Backend

Backend is a collection of servers, they control connection pools to servers and transport options, such as connection, read and write timeouts.

2.1.5 Server

Server is a ﬁnal destination of the incoming request, each server is deﬁned by URL :// :, e.g. http://localhost:5000.

2.1.6 Middleware

Vulcand supports pluggable middlewares. Middlewares can intercept or transform the request to any frontend. Exam- ples of the supported middlewares are rate limits and connection limits. You can add or remove middlewares using command line, API or directly via backends.

2.1.7 Circuit Breaker

Circuit breakers are special type of middlewares that observe various metrics for a particular frontend and can activate failover scenario whenever the condition matches e.g. error rate exceeds the threshold.

2.1.8 Secret storage

Vulcand supports secret storage - running process acts like encryption/decryption service every time it reads and writes sensitive data, e.g. TLS certiﬁcates to the backend. To use this feature, users generate sealKey using command line utility and pass this key to the process for encryption and decryption of the data in the backends.

2.1.9 Failover Predicates

Sometimes it is handy to retry the request on error. The good question is what constitutes an error? Sometimes it’s a read/write timeout, and somethimes it’s a special error code. Failover predicates are expressions that deﬁne when the request can be failed over, e.g. IsNetworkError() && Attempts <= 2

IsNetworkError() # failover on network error Attempts()<=1 # allows only 1 failover attempt RequestMethod()== "GET" # failover for GET requests only ResponseCode()== 408 # failover on 408 HTTP response code

Warning: if you omit Attempts, failover will max out after 10 attempts.

2.1.10 Route

Route is a simple routing language for matching http requests with Go syntax: Method("POST") && Path("/ path"), here are a couple of examples:

6 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

Host(".example.com") // Match by host with trie syntax HostRegexp(".*.example.com") // Match by host with regexp syntax

Path("/v1/users/") // Match by path with trie syntax Method("POST")&& Path("/v1/users") // Match by method and path with trie syntax

PathRegexp("/v1/users/.*") // Match by path with ˓→regexp syntax MethodRegexp("DELETE|GET")&& PathRegexp("/v1/users/. *") // Match by method and ˓→path with regexp syntax

Header("Content-Type", "application/") // trie-based matcher for

˓→headers HeaderRegexp("Content-Type", "application/.*") // regexp based matcher ˓→for headers

2.2 Conﬁguration

Vulcand can be conﬁgured via Etcd, API or command line tool - vctl. You can switch between different conﬁguration examples using the samples switch.

2.2.1 Backends and servers

Backend is a collection of servers. Vulcand load-balances requests within the backend and keeps the connection pool to every server. Fron- tends using the same backend will share the connections. Adding and removing servers to the backend will change the trafﬁc in real-time, removing the backend will lead to graceful drain off of the connections.

# Upsert

˓→backend and add a server to it etcdctl set /vulcand/backends/

˓→b1/backend '{"Type": "http"}' etcdctl set /vulcand/

˓→backends/b1/servers/srv1'

˓→{"URL": "http://localhost:5000"}'

# Upsert

˓→backend and add a server to it vctl backend upsert -id b1 vctl server upsert -b b1 -id

˓→srv1 -url http://localhost:5000

# Upsert

˓→backend and add a server to it curl -X POST -H "Content-

˓→Type: application/json" http:/

˓→/localhost:8182/v2/backends\ (continues on next page)

2.2. Conﬁguration 7 Vulcand REST API Documentation, Release 2.0

(continued from previous page) -d '{"Backend

˓→": {"Id":"b1", "Type":"http"}}' curl -X POST

˓→-H "Content-Type: application/

˓→json" http://localhost:8182/

˓→v2/backends/b1/servers\ -d '{"Server": {"Id":"srv1",

˓→ "URL":"http://localhost:5000"}}'

Backend settings Backends deﬁne the conﬁguration options to the servers, such as the amount of idle connections and timeouts. Backend options are represented as JSON dictionary.

{ "Timeouts":{ "Read": "1s", /

˓→/ Socket read timeout (before we

˓→receive the first reply header) "Dial":

˓→ "2s", // Socket connect timeout "TLSHandshake

˓→": "3s", // TLS handshake timeout }, "KeepAlive":{ "Period":

˓→ "4s", // Keepalive

˓→period for idle connections "MaxIdleConnsPerHost

˓→":3, // How many idle

˓→connections will be kept per host } }

You can update the settings at any time, that will initiate graceful reload of the underlying settings in Vulcand.

etcdctl set /vulcand/backends/b1/

˓→backend '{"Type": "http",

˓→"Settings": {"KeepAlive": {

˓→"MaxIdleConnsPerHost": 128,

˓→"Period": "4s"}}}'

vctl backend upsert -id b1\ -readTimeout=1s -

˓→dialTimeout=2s -

˓→handshakeTimeout=3s\ -keepAlivePeriod=4s -

˓→maxIdleConns=128

curl -X POST -H "Content-Type:

˓→application/json" http://

˓→localhost:8182/v2/backends\ (continues on next page)

8 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

(continued from previous page) -d '{"Backend": {"Id":"b1",

˓→"Type":"http", "Settings": {

˓→"KeepAlive": {

˓→"MaxIdleConnsPerHost": 128,

˓→"Period": "4s"}}}}'

Server heartbeat Heartbeat allows to automatically de-register the server when it crashes or wishes to be de-registered. Server can heartbeat it’s presense, and once the heartbeat is stopped, Vulcand will gracefully remove the server from the rotation.

# Upsert a server with TTL 5 seconds etcdctl set --ttl5 /vulcand/backends/b1/servers/srv2 '{"URL": "http://localhost:5001

˓→"}'

# Upsert a server with TTL 5 seconds vctl server upsert -b b1 -id srv2 -ttl 5s -url http://localhost:5002

# Upsert a server with TTL 5 seconds curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/backends/b1/

˓→servers\ -d '{"Server": {"Id":"srv2", "URL":"http://localhost:5001"}, "TTL": "5s"}'

2.2.2 Frontends

If request matches a frontend route it is redi- rected to one of the servers of the as- so- ci- ated backend. It is

2.2. Conﬁguration 9 Vulcand REST API Documentation, Release 2.0

rec- om- mended to specify a frontend per API method, e.g. Host("api.example.com") && Method("POST") && Path("/v1/users"). Route can be any valid route expression, e.g. Path("/v1/users") will match for all hosts and Host("api. example.com") && Path("/v1/users") will match only for api.example.com.

# upsert frontend connected to backend b1 and matching path "/" etcdctl set /vulcand/frontends/f1/frontend '{"Type": "http", "BackendId": "b1", "Route

˓→": "Path(`/`)"}'

# upsert frontend connected to backend b1 and matching path "/" vctl frontend upsert -id f1 -b b1 -route 'Path("/")'

# upsert frontend connected to backend b1 and matching path "/" curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/frontends\ -d '{"Frontend": {"Id":"f1", "Type": "http", "BackendId": "b1", "Route": "Path(\

˓→"/\")"}}'

Frontend settings Frontends control various limits, forwarding and failover settings.

{ "Limits":{ "MaxMemBodyBytes": 12, // Maximum request body size to keep in memory before

˓→buffering to disk "MaxBodyBytes": 400, // Maximum request body size to allow for this frontend }, "FailoverPredicate": "IsNetworkError() && Attempts() <= 1", // Predicate that

˓→defines when requests are allowed to failover "Hostname": "host1", // Host to set in

˓→forwarding headers "TrustForwardHeader": true, // Time provider

˓→(useful for testing purposes) }

Setting frontend settings upates the limits and parameters for the newly arriving requests in real-time. etcdctl set /vulcand/frontends/f1/frontend '{"Id": "f1", "Type": "http", "BackendId":

˓→"b1", "Route": "Path(`/`)", "Settings": {"FailoverPredicate":"(IsNetworkError() ||

˓→ResponseCode() == 503) && Attempts() <= 2"}}' vctl frontend upsert\ -id=f1\ -route='Path("/")'\ -b=b1\ -maxMemBodyKB=6 -maxBodyKB=7\ -failoverPredicate='IsNetworkError()'\ -trustForwardHeader\ -forwardHost=host1 curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/frontends\ -d '{"Frontend": {"Id": "f1", "Type": "http", "BackendId": "b1", "Route":

˓→"Path(`/`)", "Settings": {"FailoverPredicate":"(IsNetworkError() || ResponseCode() ˓→== 503) && Attempts() <= 2"}}}' (continues on next page)

10 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

(continued from previous page)

Switching backends Updating frontend’s backend property gracefully re-routes the trafﬁc to the new servers assigned to this backend:

# redirect the traffic of the frontend "loc1" to the servers of the backend "b2" etcdctl set /vulcand/frontends/f1/frontend '{"Type": "http", "BackendId": "b2", "Route

˓→": "Path(`/`)"}'

# redirect the traffic of the frontend "f1" to the servers of the backend "b2" vctl frontend upsert -id=f1 -route='Path("/")' -b=b2

# redirect the traffic of the frontend "loc1" to the servers of the backend "up2" curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/frontends -

˓→d '{"Frontend": {"Id": "f1", "Type": "http", "BackendId": "b2", "Route": "Path(`/`)

˓→"}}'

Note: you can add and remove servers to the existing backend, and Vulcand will start redirecting the trafﬁc to them automatically

2.2.3 Hosts

One can use Host entries to specify host-related settings, such as TLS certificates and SNI options. TLS Certificates Certificates are stored as encrypted JSON dictionaries. Updating a certificate will gracefully reload it for all running HTTP servers.

# Set keypair etcdctl set /vulcand/hosts/localhost/host '{"Settings": {"KeyPair": {...}}}' vctl host upsert -name -cert= -privateKey= curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/hosts\ -d '{"Host": { "Name": "localhost", "Settings": {"KeyPair": {"Cert": "base64",

˓→Key: "base64"}}}}'

Note: When setting keypair via Etcd you need to encrypt keypair. This is explained in TLS section of this document.

OCSP Online Certiﬁcate Status Protocol is a protocol for certiﬁcate revocation checking. Vulcand checks OCSP status in the background and includes the OCSP staple response in the TLS handshake when this feature turned on. Read more about turning OCSP for hosts in OCSP section of this document.

2.2. Conﬁguration 11 Vulcand REST API Documentation, Release 2.0

2.2.4 Routing Language

Vulcand uses a special type of a routing language to match requests - called route and implemented as a standalone library It uses Go syntax to route http requests by hostname, method, path and headers. Every Vulcand frontend has a special Route ﬁeld for routing requests. Here is the syntax explained:

Matcher("value") // matches value using trie Matcher(".value") // uses trie-based matching for a.value and b.value MatcherRegexp(".*value") // uses regexp-based matching

Host matcher:

Host(".localhost") // trie-based matcher for a.localhost, b.localhost, etc. HostRegexp(".*localhost") // regexp based matcher

Path matcher:

Path("/hello/") // trie-based matcher for raw request path PathRegexp("/hello/.*") // regexp-based matcher for raw request path

Method matcher:

Method("GET") // trie-based matcher for request method MethodRegexp("POST|PUT") // regexp based matcher for request method

Header matcher:

Header("Content-Type", "application/") // trie-based matcher for headers HeaderRegexp("Content-Type", "application/.*") // regexp based matcher for headers

Matchers can be combined using && operator:

Host("localhost")&& Method("POST")&& Path("/v1")

Vulcan will join the trie-based matchers into one trie matcher when possible, for example:

Host("localhost")&& Method("POST")&& Path("/v1") Host("localhost")&& Method("GET")&& Path("/v2")

Will be combined into one trie for performance. If you add a third route:

Host("localhost")&& Method("GET")&& PathRegexp("/v2/. *")

It wont be joined ito the trie, and would be matched separately instead.

Warning: Vulcan can not merge regexp-based routes into efﬁcient structure, so if you have hundreds/thousands of frontends, use trie-based routes!

Host based routing

Vulcand does not require host-speciﬁc routing, e.g. the frontend with the following route will match all requests regardless of their hostname:

12 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

PathRegexp("/.*")

curl -H "Host:example.com" http://localhost/hello # works curl -H "Host:hello.com" http://localhost/hello # also works

In case if you need Host-based routing (just as Apache’s VHost or Nginx’s Server names), you can use the routes:

Host("example.com")&& PathRegexp("/. *")

curl -H "Host:example.com" http://localhost/hello # works curl -H "Host:hello.com" http://localhost/hello # not found

Note: The example above do not set up host entries in Vulcand. You only need them when using HTTPS to supply certiﬁcates.

Method matching

Vulcand works better when creating a separate frontend for each HTTP method in your API:

Host("localhost")&& Method("POST")&& Path("/users") Host("localhost")&& Method("GET")&& Path("/users")

In this case each frontend collects separate set of realtime metrics that are different for creating and gettings users. This separation will provide separate histograms and separate load balancing logic for different request types what helps to understand the performance better.

2.2.5 Listeners

Listeners allow at- tach- ing and de- tach- ing sockets on var- i- ous in- ter- faces and net- works. Vulcand can have multiple listeners attached and share the same listener.

2.2. Conﬁguration 13 Vulcand REST API Documentation, Release 2.0

{ "Protocol":"http", // 'http' or 'https' "Scope":"", // optional scope field, read below for details "Address":{ "Network":"tcp", // 'tcp' or 'unix' "Address":"localhost:8183" // 'host:port' or '/path/to.socket' }, }

# Add http listener accepting requests on 127.0.0.1:8183 etcdctl set /vulcand/listeners/ls1\ '{"Protocol":"http", "Address":{"Network":"tcp", "Address":"127.0.0.1:8183

˓→"}}'

# Add http listener accepting requests on 127.0.0.1:80 vctl listener upsert --id ls1 --proto=http --net=tcp -addr=127.0.0.1:8080

# Add http listener accepting requests on 127.0.0.1:8183 curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/listeners\ -d '{"Listener":{"Id": "ls1", "Protocol":"http", "Address":{"Network":"tcp",

˓→"Address":"127.0.0.1:8183"}}}'

Listener scopes Listeners support scopes as the way to limit operational scope of socket. Scope ﬁeld uses Vulcand Routing Language. Here’s an example of Listener that only allows requests with hostname example.com

{ "Protocol":"http", // 'http' or 'https' "Scope": "Host(`example.com`)", // operational scope "Address":{ "Network":"tcp", // 'tcp' or 'unix' "Address":"0.0.0.0:8183" // 'host:port' or '/path/to.socket' }, }

E.g. if we have two frontends deﬁned:

Host("example.com")&& Path("/users") Host("localhost")&& Path("/users")

Only ﬁrst frontend is reachable for requests coming to port 8183.

2.2.6 Middlewares

Middlewares are allowed to observe, modify and intercept http requests and responses. Vulcand provides several middlewares. Users can write their own middlewares for Vulcand in Go. To specify execution order of the middlewares, one can deﬁne the priority. Middle- wares with smaller priority values will be executed ﬁrst.

14 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

2.2.7 Rate Limits

Vulcan supports controlling request rates. Rate can be checked against different request parameters and is set up via limiting variable.

client.ip

˓→ # client ip request.header.X-Special-

˓→Header # request header

Adding and removing middlewares will modify the frontend behavior in real time. One can set expiring middlewares as well.

# Update or set

˓→rate limit the request to

˓→frontend "f1" to 1 request

˓→per second per client ip # with bursts

˓→up to 3 requests per second. etcdctl set /vulcand/frontends/

˓→f1/middlewares/rl1'{ "Priority": 0, "Type": "ratelimit", "Middleware":{ "Requests":1, "PeriodSeconds":1, "Burst":3,

˓→ "Variable": "client.ip"}}'

# Update or set

˓→rate limit the request to

˓→frontend "f1" to 1 request

˓→per second per client ip # with bursts

˓→up to 3 requests per second. vctl ratelimit

˓→upsert -id=rl1 -frontend=f1

˓→-requests=1 -burst=3

˓→-period=1 --priority=0

# Update or set

˓→rate limit the request to

˓→frontend "f1" to 1 request

˓→per second per client ip # with bursts

˓→up to 3 requests per second. curl -X POST -

˓→H "Content-Type: application/

˓→json" http://localhost:8182/

˓→v2/frontends/f1/middlewares\ -d '{"Middleware": { "Priority": 0, "Type": "ratelimit", (continues on next page)

2.2. Conﬁguration 15 Vulcand REST API Documentation, Release 2.0

(continued from previous page) "Id": "rl1", "Middleware":{ "Requests":1, "PeriodSeconds":1, "Burst":3,

˓→ "Variable": "client.ip"}}}'

Programmatic rate limits Sometimes you have to change rate limits based on various parameters, e.g. account billing plan. Instead of setting hard-coded rate limits, Vulcand can accept rate limits set via headers for each individual request. Each HTTP header should contain a JSON-encoded list with rates in the following format:

[{"PeriodSeconds":1,

˓→ "Requests":2, "Burst":3}]

That means that you should write a middleware that sets the header to the right value and place it before the ratelimit middleware. After it’s done you can activate the ratelimit plugin:

# Update or set rate limit

˓→the request to frontend "f1"

˓→to get the rates from the X-

˓→Custom-Rates. # in case if the header

˓→is missing, ratelimit will

˓→default to 1 request per

˓→second per client ip # with bursts

˓→up to 3 requests per second. etcdctl set /vulcand/frontends/

˓→f1/middlewares/rl1'{ "Id":"rl1", "Priority":0, "Type":"ratelimit", "Middleware":{ "PeriodSeconds":1, "Requests":1, "Burst":3, "Variable":"client.ip", "RateVar":"request.

˓→header.X-Custom-Rates"}}'

# Update or set rate limit

˓→the request to frontend "f1"

˓→to get the rates from the X- (continues on next page) ˓→Custom-Rates.

16 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

(continued from previous page) # in case if the header

˓→is missing, ratelimit will

˓→default to 1 request per

˓→second per client ip # with bursts

˓→up to 3 requests per second. vctl ratelimit upsert -id=rl1

˓→-frontend=f1 -requests=1

˓→-burst=3 -period=1--

˓→priority=0 --rateVar=

˓→"request.header.X-Custom-

˓→Rates"

# Update or set rate limit

˓→the request to frontend "f1"

˓→to get the rates from the X-

˓→Custom-Rates. # in case if the header

˓→is missing, ratelimit will

˓→default to 1 request per

˓→second per client ip # with bursts

˓→up to 3 requests per second. curl -X POST -H "Content-Type:

˓→application/json" http:/

˓→/localhost:8182/v2/frontends/

˓→f1/middlewares -d'{ "Middleware": { "Id":"rl1", "Priority":0, "Type":"ratelimit", "Middleware":{ "PeriodSeconds":1, "Requests":1, "Burst":3,

˓→ "Variable":"client.ip", "RateVar":"request.

˓→header.X-Custom-Rates"}}}'

2.2.8 Connection Limits

Connection limits control the amount of simultaneous connections per frontend. Frontends re-use the same variables as rate limits.

# limit the amount of connections per frontend to 16 per client ip etcdctl set /vulcand/frontends/f1/middlewares/cl1\ '{"Priority": 0, "Type": "connlimit", "Middleware":{"Connections":16, "Variable

˓→": "client.ip"}}'

# limit the amount of connections per frontend to 16 per client ip vctl connlimit upsert -id=cl1 -frontend=f1 -connections=1 --priority=0--

˓→variable=client.ip

2.2. Conﬁguration 17 Vulcand REST API Documentation, Release 2.0

# limit the amount of connections per frontend to 16 per client ip curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/frontends/

˓→f1/middlewares\ -d '{"Middleware": {"Id": "cl1", "Priority": 0, "Type": "connlimit", "Middleware

˓→":{"Connections":16, "Variable": "client.ip"}}}'

2.2.9 Rewrites and redirects

Rewrite plugin enables rewriting request URLs, returning redirect responses and changing response bodies. Rewrites

# remove /foo prefix from the url etcdctl set /vulcand/frontends/f1/middlewares/r1'{ "Id":"r1", "Priority":1, "Type":"rewrite", "Middleware":{ "Regexp":"/foo(.*)", "Replacement":"$1", "RewriteBody":false, "Redirect":false}}'

# remove /foo prefix from the url, note the single quotes for '$1' vctl rewrite upsert -f f1 -id r1 --regexp="/foo(.*)" --replacement='$1'

# remove /foo prefix from the url curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/frontends/

˓→f1/middlewares\ -d '{"Middleware": { "Id":"r1", "Priority":1, "Type":"rewrite", "Middleware":{ "Regexp":"/foo(.*)", "Replacement":"$1", "RewriteBody":false, "Redirect":false}}}'

Redirects Setting a redirect parameter to rewrite will make it to generate 302 Found response with Location header set to the new URL:

# remove /foo prefix from the url etcdctl set /vulcand/frontends/f1/middlewares/r1'{ "Id":"r1", "Priority":1, "Type":"rewrite", "Middleware":{ "Regexp":"^http://localhost/(.*)", "Replacement":"https://localhost/$1", "RewriteBody":false, "Redirect":true}}'

18 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

# redirect http requests to https location vctl rewrite upsert -f f1 -id r1 --regexp="^http://localhost/(.*)" --replacement= ˓→'https://localhost/$1' --redirect

# remove /foo prefix from the url curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/frontends/

˓→f1/middlewares\ -d '{"Middleware": { "Id":"r1", "Priority":1, "Type":"rewrite", "Middleware":{ "Regexp":"^http://localhost/(.*)", "Replacement":"https://localhost/$1", "RewriteBody":false, "Redirect":true}}}'

Templating Rewrite can treat the response body as a template. Consider the following example:

# treat response body as a template etcdctl set /vulcand/frontends/f1/middlewares/r1'{ "Id":"r1", "Priority":1, "Type":"rewrite", "Middleware":{"RewriteBody":true}}'

# treat response body as a template vctl rewrite upsert -f f1 -id r1 --rewriteBody

# treat response body as a template curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/frontends/

˓→f1/middlewares\ -d '{"Middleware": { "Id":"r1", "Priority":1, "Type":"rewrite", "Middleware":{"RewriteBody":true}}}'

The backend server can now reply: handler := http.HandlerFunc(func(w http.ResponseWriter, req *http.Request){ w.WriteHeader(200) w.Write([]byte(`{"foo": "{{.Request.Header.Get "variable-value"}}"}`)) })

And the client will get as a response:

{"foo": "variable-value"}

2.2. Conﬁguration 19 Vulcand REST API Documentation, Release 2.0

2.2.10 Structured logs

Warning: We are still polishing the log format, so it may change soon. trace plugin supports output in syslog-compatible format of the structured logs to UDP or Unix socket. Here’s the example of the log entry:

Jan 13 15:07:51 vulcan pid:[3634]: @cee:{"request":{"method":"GET","url":"http://

˓→h:5000"},"response":{"code":404,"roundtrip":0.333712}}

The preﬁx is a standard syslog preﬁx, and the part after @cee: is a structured log entry. Here’s the entry format explained:

{ "request":{ "method": "GET", // request method "url": "http://localhost:5000", // request URL "headers":{ // optional captured request headers "User-Agent":[ // captured request User-Agent header values "curl\/7.35.0" ] }, "tls":{ // tls is an optonal field, used when it's a

˓→TLS connection "version": "TLS12", // TLS version used "resume": false, // whether it's a session resumed with session

˓→ticket "cipher_suite": "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", // cipher used in a

˓→connection "server": "vulcand.io" // server name used in SNI } }, "response":{ "code": 404, // response code "roundtrip": 0.408372, // roundtrip in milliseconds, part after '.' is

˓→microseconds "headers":{ // optional captured response headers "Content-Type":[ "text\/plain; charset=utf-8" // captured response Content-Type header values ] } } }

Adding and removing trace middleware will turn on/off tracing in real time.

# turn tracing on, pointing output to unix syslog facility. # capture request header values 'X-A' and 'X-B' and response headers 'X-C' and 'X-D' etcdctl set /vulcand/frontends/f1/middlewares/t1'{ "Id":"t1", "Priority":1, "Type":"trace", "Middleware":{ "ReqHeaders":["X-A","X-B"], (continues on next page)

20 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

(continued from previous page) "RespHeaders":["X-C","X-D"], "Addr":"syslog://"}}'

# turn tracing on, pointing output to unix syslog facility. # capture request header values 'X-A' and 'X-B' and response headers 'X-C' and 'X-D' vctl trace upsert -f f1 -id t1 --addr='syslog://'\ --reqHeader=X-A --reqHeader=X-B --respHeader=X-C --respHeader=X-D

# turn tracing on, pointing output to unix syslog facility. # capture request header values 'X-A' and 'X-B' and response headers 'X-C' and 'X-D' curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/frontends/

˓→f1/middlewares -d'{ "Middleware": { "Id":"t1", "Priority":1, "Type":"trace", "Middleware":{ "ReqHeaders":["X-A","X-B"], "RespHeaders":["X-C","X-D"], "Addr":"syslog://"}}}'

Controlling output You can control output using the following form of address values:

# UDP socket formats syslog://localhost:5000 # host localhost, port 5000, LOG_

˓→LOCAL0 facility syslog://localhost:5000?f=MAIL&sev=INFO # host localhost, port 5000, MAIL

˓→facility, INFO severity syslog://localhost:5000?f=MAIL # host localhost, port 5000, MAIL

˓→facility, INFO severity syslog://localhost:5000?f=LOG_LOCAL0&sev=DEBUG # host localhost, port 5000, LOG_

˓→LOCAL0 facility, INFO severity

# unixgram socket format syslog:///tmp/out.sock # /tmp/out.sock unixgram socket syslog:///tmp/out.sock?f=MAIL # /tmp/out.sock unixgram socket

# default syslog syslog:// # default OS-specific unix/unixgram socket syslog://?f=LOG_LOCAL0&sev=INFO # default OS-specific unix/unixgram socket

2.2.11 Circuit Breakers

Circuit breaker is a special middleware that is designed to provide a fail-over action in case if service has degraded. It is very helpful to prevent cascading failures - where the failure of the one service leads to failure of another. Circuit breaker observes requests statistics and checks the stats against special error condition. In case if condition matches, CB activates the fallback scenario: returns the response code or

2.2. Conﬁguration 21 Vulcand REST API Documentation, Release 2.0

redirects the request to another frontend. Circuit Breaker states CB provides a set of explicit states and transitions explained below:

´ Initial state is Standby. CB observes the statistics and does not modify the request. In case if condition matches, CB enters Tripped state, where it responds with predefines code or redirects to another frontend. CB can execute the special HTTP callback when going from Standby to Tripped state CB sets a special timer that defines how long does it spend in the Tripped state Once Tripped timer expires, CB enters Recovering state and resets all stats In Recovering state Vulcand will start routing the portion of the traffic linearly increasing it over the period specified in Recovering timer. In case if the condition matches in Recovering state, CB enters Tripped state again In case if the condition does not match and recovery timer expries, CB enters Standby state. CB can execute the special HTTP callback when going from Recovering to Standby state Conditions CB defines a simple language that allows us to specify simple con- di- tions that watch the

22 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

stats for a frontend:

•••NetworkErrorRatio•••••• ()> 0.5 // watch error ratio over 10 second sliding window for ˓→a frontend LatencyAtQuantileMS(50.0)> 50 // watch latency at quantile in milliseconds. ResponseCodeRatio(500, 600,0, 600)> 0.5 // ratio of response codes in range [500-

˓→600) to [0-600)

Note: Quantiles should be provided as ﬂoats - don’t forget to add .0 to hint it as ﬂoat

Response fallback Response fallback will tell CB to reply with a predeﬁned response instead of forwarding the request to the backend

{ "Type": "response", "Action":{ "ContentType": "text/plain", "StatusCode": 400, "Body": "Come back later" } }

Redirect fallback Redirect fallback will redirect the request to another frontend.

Note: It won’t work for frontends not deﬁned in the Vulcand conﬁg.

{ "Type": "redirect", "Action":{ "URL": "https://example.com/fallback" } }

Webhook Action Circuit breaker can notify external sources on it’s state transitions, e.g. it can create a pager duty incident by issuing a webhook:

{ "Body":{ "client": "Sample Monitoring Service", "client_url": "https://example.com", "description": "FAILURE for production/HTTP on machine srv01.acme.com", "event_type": "trigger", "incident_key": "srv01/HTTP", "service_key": "-pager-duty-service-key" }, "Headers":{ (continues on next page)

2.2. Conﬁguration 23 Vulcand REST API Documentation, Release 2.0

(continued from previous page) "Content-Type":[ "application/json" ] }, "Method": "POST", "URL": "https://events.pagerduty.com/generic/2010-04-15/create_event.json" }

Setup Circuit breaker setup is can be done via Etcd, command line or API: etcdctl set /vulcand/frontends/f1/middlewares/cb1'{ "Id":"cb1", "Priority":1, "Type":"cbreaker", "Middleware":{ "Condition":"NetworkErrorRatio() > 0.5", "Fallback":{"Type": "response", "Action": {"StatusCode": 400, "Body":

˓→"Come back later"}}, "FallbackDuration": 10000000000, "RecoveryDuration": 10000000000, "CheckPeriod": 100000000 } }' vctl cbreaker upsert\ --frontend=f1\ --id=cb1\ --condition="NetworkErrorRatio() > 0.5"\ --fallback='{"Type": "response", "Action": {"StatusCode": 400, "Body

˓→": "Come back later"}}' curl -X POST -H "Content-Type: application/json"\ http://localhost:8182/v2/frontends/f1/middlewares\ -d'{ "Middleware": { "Id":"cb1", "Priority":1, "Type":"cbreaker", "Middleware":{ "Condition":"NetworkErrorRatio() > 0.5", "Fallback":{ "Type": "response", "Action": {"StatusCode": 400, "Body": "Come back later"} }, "FallbackDuration": 10000000000, "RecoveryDuration": 10000000000, "CheckPeriod": 100000000 } } }'

24 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

2.3 TLS

Vulcand supports HTTPS via SNI, certiﬁcate management and multiple HTTPS servers per running process. This sections below contain all the steps required to enable TLS support in Vulcand

2.3.1 Managing certiﬁcates

Vulcand encrypts certiﬁcates when storing them in the backends and uses Nacl secretbox to seal the data. The running server acts as an encryption/decryption point when reading and writing certiﬁcates. This special key has to be generated by Vulcand using command line utility: Setting up seal key

$ vctl secret new_key

Once we got the key, we can pass it to the running daemon. This key will be used by Vulcand to encrypt and decrypt stored certiﬁcates and private keys.

$ vulcand -sealKey="the-seal-key"

Note: Add space before command to avoid leaking seal key in bash history, or use HISTIGNORE

Warning: Vulcand needs the sealKey to use TLS, without it simply will refuse to set the certiﬁcates for host.

Setting host keypair Setting certiﬁcate via etcd is slightly different from CLI and API:

# Read the private key and certificate and returns back the encrypted version that

˓→can be passed to etcd $ vctl secret seal_keypair -sealKey -cert= -privateKey=

˓→

# Once we got the certificate sealed, we can pass it to the Etcd: etcdctl set /vulcand/hosts/mailgun.com/host '{"Settings": {"KeyPair": {..encrypted

˓→data...}}}'

# Connect to Vulcand Update the TLS certificate. # In this case we don't need to supply seal key, as in this case the CLI talks to the

˓→Vulcand directly $ vctl host upsert -name -cert= -privateKey=

# In this case we don't need to supply seal key, as in this case the CLI talks to the

˓→Vulcand directly curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/hosts\ -d '{"Host": {"Name": "localhost", "Settings": {"KeyPair": {"Cert": "base64-

˓→encoded-certificate", "Key": "base64-encoded-key-string"}}}}'

Note: To update the certiﬁcate in the live mode just repeat the steps with the new certiﬁcate, vulcand will gracefully

2.3. TLS 25 Vulcand REST API Documentation, Release 2.0 reload the TLS conﬁg for running server

2.3.2 OCSP

Online Certificate Status Protocol is a protocol for certificate revocation checking. Vulcand checks OCSP status in the background and includes the OCSP staple response in the TLS handshake when this feature turned on. By default it is turned off, mostly because it provides questionable benefits.

# Set keypair and OCSP settings etcdctl set /vulcand/hosts/localhost/host '{"Settings": {"KeyPair": {...}, "OCSP":{"Enabled":true,"Period":"1h0m0s","Responders":[],

˓→"SkipSignatureCheck":false}}}'

# set keypair and OCSP settings # --ocsp // turn OCSP on # --ocspSkipCheck // insecure: skip OCSP signature check # --ocspPeriod='1h' // override OCSP check period defined in the certificate, use

˓→'1h','30m' as time periods # --ocspResponder // optional OCSP responder, use multiple args for responder

˓→list vctl host upsert -name example.com -cert= -privateKey=

˓→key>\ --ocsp --ocspSkipCheck --ocspPeriod='1h' --ocspResponder="http://example.com/respond"

#set keypair and OCSP settings curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/hosts\ -d '{"Host": { "Name": "localhost", "Settings": { "KeyPair": {"Cert": "base64", Key: "base64"}, "OCSP":{ "Enabled":true, "Period":"1h0m0s", "Responders":[], "SkipSignatureCheck":false}}}}}'

2.3.3 SNI

Not all clients support SNI, or sometimes host name is not available. In this case you can set the default certiﬁcate that will be returned in case if the SNI is not available:

# Set example.com as default host returned in case if SNI is not available etcdctl set /vulcand/hosts/example.com/host '{"Settings": {"Default": true, "KeyPair

˓→": {...}}}'

2.3.4 Session Tickets

Session tickets is a way to resume TLS connection, saving time on a TLS handshake. Vulcand supports in-memory session tickets cache for HTTPS listeners and backend pools. Session tickets are enabled by default

26 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

# Add http listener accepting requests on 127.0.0.1:9443, uses session ticket LRU

˓→cache of 1024 etcdctl set /vulcand/listeners/ls1\ '{"Id":"ls1","Protocol":"https","Address":{"Network":"tcp","Address":"127.

˓→0.0.1:9443"}, "Settings":{ "TLS":{ "SessionTicketsDisabled":false, "SessionCache":{"Type":"LRU","Settings":{"Capacity":1024}}}}}'

# Add http listener accepting requests on 127.0.0.1:9443, uses session ticket LRU

˓→cache of 1024 vctl listener upsert --id ls1 --proto=https --net=tcp -addr=127.0.0.1:9443\ -tlsSessionCache=LRU -tlsSessionCacheCapacity=1024

# Add http listener accepting requests on 127.0.0.1:443, uses session ticket LRU

˓→cache of 1024 curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/listeners\ -d '{"Listener": {"Id": "ls1", "Protocol":"https", "Address":{"Network":"tcp", "Address":"127.0.0.1:443"}, "Settings":{ "TLS":{ "SessionTicketsDisabled":false, "SessionCache":{"Type":"LRU","Settings":{"Capacity":1024}}}}}}'

2.3.5 Cipher Suites

Vulcand supports cipher suites currently implemented in go crypto/tls standard lib:

TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_RC4_128_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_RC4_128_SHA TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

By default, the following cipher suites are selected, in the order of preference:

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA

Here’s an example of how to conﬁgure cipher suites for HTTPS listener

2.3. TLS 27 Vulcand REST API Documentation, Release 2.0

# Add http listener accepting requests on 127.0.0.1:9443, uses session ticket LRU

˓→cache of 1024 etcdctl set /vulcand/listeners/ls1\ '{"Id":"ls1","Protocol":"https","Address":{"Network":"tcp","Address":"127.

˓→0.0.1:9443"}, "Settings":{ "TLS":{ "CipherSuites":[ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"]}}}'

# Add http listener accepting requests on 127.0.0.1:443 with customized cipher suite

˓→list vctl listener upsert --id ls1 --proto=https --net=tcp -addr=127.0.0.1:9443\ --tlsCS=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 --tlsCS=TLS_ECDHE_RSA_WITH_AES_

˓→128_GCM_SHA256

# Add http listener accepting requests on 127.0.0.1:443, uses session ticket LRU

˓→cache of 1024 curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/listeners\ -d '{"Listener": {"Id": "ls1", "Protocol":"https", "Address":{"Network":"tcp",

˓→"Address":"127.0.0.1:9443"}, "Settings":{ "TLS":{ "CipherSuites":[ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"]}}}}'

2.3.6 TLS options

Both HTTPS listeners and backends support some other TLS options: • Insecure: skipping certiﬁcate checks • Setting minimum and maximum supported version • Setting a server preference when selecting a cipher suite. Here’s an example on how to set these options for HTTPS listener. Note that you can use the same parameters for backends as well.

# Add http listener accepting requests on 127.0.0.1:9443, uses session ticket LRU

˓→cache of 1024 etcdctl set /vulcand/listeners/ls1'{ "Id":"ls1", "Protocol":"https", "Address":{"Network":"tcp","Address":"127.0.0.1:9443"}, "Settings":{ "TLS":{ "PreferServerCipherSuites":true, "InsecureSkipVerify":true, "MinVersion":"VersionTLS10", "MaxVersion":"VersionTLS11", "SessionTicketsDisabled":true}}}'

28 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

# Add http listener accepting requests on 127.0.0.1:9443 with customized cipher suite

˓→list vctl listener upsert --id ls1 --proto=https --net=tcp -addr=127.0.0.1:9443\ --tlsSkipVerify --tlsSessionTicketsOff --tlsMinV=VersionTLS10 --

˓→tlsMaxV=VersionTLS11 --tlsPreferServerCS

# Add http listener accepting requests on 127.0.0.1:443, uses session ticket LRU

˓→cache of 1024 curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/listeners\ -d '{"Listener": { "Id":"ls1", "Protocol":"https", "Address":{"Network":"tcp","Address":"127.0.0.1:9443"}, "Settings":{ "TLS":{ "PreferServerCipherSuites":true, "InsecureSkipVerify":true, "MinVersion":"VersionTLS10", "MaxVersion":"VersionTLS11", "SessionTicketsDisabled":true}}}}'

2.3.7 HTTPS listeners

Once we have the certiﬁcate set, we can create HTTPS listeners for the host:

# Add http listener accepting requests on 127.0.0.1:443 etcdctl set /vulcand/listeners/ls1\ '{"Protocol":"https", "Address":{"Network":"tcp", "Address":"127.0.0.1:443

˓→"}}'

# Add http listener accepting requests on 127.0.0.1:443 vctl listener upsert --id ls1 --proto=https --net=tcp -addr=127.0.0.1:443

# Add http listener accepting requests on 127.0.0.1:443 curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/listeners\ -d '{"Listener": {"Id": "ls1", "Protocol":"https", "Address":{"Network":"tcp", "Address":

˓→"127.0.0.1:443"}}}'

2.3.8 HTTPS Backends

Vulcand supports HTTPS backends out of the box, with default TLS settings. All TLS options described in the sections above, like session tickets, cipher suites and TLS versions are available for HTTPS backends as well. Here’s how you can modify TLS settings for a backend:

# Upsert https backend, choosing to ignore certificate checks and setting min and max

˓→TLS version etcdctl set /vulcand/backends/b1/backend '{"Id":"b1","Type":"http", "Settings":{ "TLS":{ "PreferServerCipherSuites":false, "InsecureSkipVerify":true, (continues on next page)

2.3. TLS 29 Vulcand REST API Documentation, Release 2.0

(continued from previous page) "MinVersion":"VersionTLS10", "MaxVersion":"VersionTLS11"}}}'

# Upsert https backend, choosing to ignore certificate checks and setting min and max

˓→TLS version vctl backend upsert -id b1 --tlsSkipVerify --tlsMinV="VersionTLS10"--

˓→tlsMaxV=VersionTLS11

# Upsert https backend, choosing to ignore certificate checks and setting min and max

˓→TLS version curl -X POST -H "Content-Type: application/json" http://localhost:8182/v2/backends\ -d '{"Backend": {"Id":"b1","Type":"http", "Settings":{ "TLS":{ "PreferServerCipherSuites":false, "InsecureSkipVerify":true, "MinVersion":"VersionTLS10", "MaxVersion":"VersionTLS11"}}}}'

2.4 Metrics

Metrics are provided for frontends and servers:

{ "Verdict":{ "IsBad":false, // Verdict will specify if there's something wrong with the

˓→server "Anomalies":null // Anomalies can be populated if Vulcand detects something

˓→unusual }, "Counters":{ // Counters in a rolling time window "Period":10000000000, // Measuring period in ns "NetErrors":6, // Network errors "Total":78, // Total requests "StatusCodes":[ { "Code":400, // Status codes recorded "Count":7 }, { "Code":429, "Count":67 } ] }, "LatencyBrackets":[ // Latency brackets recorded for the server or frontend { "Quantile":99, "Value":172000 // microsecond resolution }, { "Quantile":99.9, (continues on next page)

30 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

(continued from previous page) "Value":229000 } ] }

Vulcand provides real-time metrics via API and command line.

# top acts like a standard linux top command, refreshing top active frontends every

˓→second. vctl top

# top frontends curl http://localhost:8182/v2/top/frontends?limit=100

# top servers curl http://localhost:8182/v2/top/servers?limit=100

# vctl top acts like a standard linux top command, refreshing top active frontends

˓→every second. vctl top # -b flag will show top only for frontends and servers that are associated with

˓→backend b1 vctl top -b b1

2.5 Logging

Vulcand supports logging levels:

INFO # all output WARN # warnings and errors only (default) ERROR # errors only

You can change the real time logging output by using set_severity command: vctl log set_severity -s=INFO curl -X PUT http://localhost:8182/v1/log/severity -F severity=INFO

# vctl log set_severity -s=INFO

You can check current severity using get_severity command: vctl log get_severity curl http://localhost:8182/v1/log/severity

# vctl log get_severity

2.5. Logging 31 Vulcand REST API Documentation, Release 2.0

2.6 Process management

2.6.1 Startup and conﬁguration

Usage of vulcand vulcand

-apiInterface="": # apiInterface - interface for API -apiPort=8182 # apiPort - port for API

-etcd=[] # etcd - list of etcd discovery service API servers -etcdKey="vulcand" # etceKey - etcd key for reading configuration

-log="console" # log - syslog or console -logSeverity="WARN" # log severity, INFO, WARN or ERROR -pidPath="" # path to write PID

-sealKey="" # sealKey is used to store encrypted data in the

˓→backend, # use 'vctl secret new_key' to create a new key.

-statsdAddr="localhost:8185" # statsdAddr - address where Vulcand will emit statsd

˓→metrics -statsdPrefix="vulcand" # statsdPrefix is a prefix prepended to every metric

-serverMaxHeaderBytes=1048576 # Maximum size of request headers in server

2.6.2 Binary upgrades

In case if you need to upgrade the binary on the ﬂy, you can now use signals to reload the binary without downtime. Here’s how it works: • Replace the binary with a new version • Send USR2 signal to a running vulcand instance kill -USR2 $(pidof vulcand)

• Check that there are two instances running:

4938 pts/12 Sl+0:04 vulcand 10459 pts/12 Sl+0:01 vulcand

Parent vulcand process forks the child process and passes all listening sockets ﬁle descriptors to the child. Child process is now serving the requests along with parent process. • Check the logs for errors • If everything works smoothly, send SIGTERM to the parent process, so it will gracefully shut down: kill 4938

• On the other hand, if something went wrong, send SIGTERM to the child process and recover the old binary back.

32 Chapter 2. User Manual Vulcand REST API Documentation, Release 2.0

kill 10459

You can repeat this process multiple times.

2.6.3 Log control

You can controll logging verbosity by supplying logSeverity startup ﬂag with the supported values INFO, WARN and ERROR, default value is WARN. If you need to temporarily change the logging for a running process (e.g. to debug some issue), you can do that by using set_severity command:

vctl log set_severity -s=INFO OK: Severity has been updated to INFO

You can check the current logging seveirty by using get_severity command:

vctl log get_severity OK: severity: INFO

2.6.4 Metrics

Vulcand can emit metrics to statsd via UDP. To turn this feature on, supply statsdAddr and statsdPrefix parameters to vulcand executable. The service emits the following metrics for each frontend and server:

Metric type Metric Name counter each distinct response code counter failure and success occurence gauge runtime stats (number of goroutines, memory)

2.7 Installation

2.7.1 Docker builds

Here’s how you build vulcan in Docker:

docker build -t mailgun/vulcand .

Starting the daemon:

docker run -d -p 8182:8182 -p 8181:8181 mailgun/vulcand:v0.8.0-alpha.3 /go/bin/

˓→vulcand -apiInterface="0.0.0.0" --etcd=http://172.17.42.1:4001

Don’t forget to map the ports and bind to the proper interfaces, otherwise vulcan won’t be reachable from outside the container. Using the vctl from container:

docker run mailgun/vulcand:v0.8.0-alpha.3 /opt/vulcan/vctl status --vulcan 'http://

˓→172.17.42.1:8182'

2.7. Installation 33 Vulcand REST API Documentation, Release 2.0

Make sure you’ve speciﬁed --vulcan ﬂag to tell vctl where the running vulcand is. We’ve used lxc bridge interface in the example above.

2.7.2 Docker trusted build

There’s a trusted mailgun/vulcand build you can use. The recommended version is 0.8.0-alpha.3.

2.7.3 Manual installation

Note: You have to install go>=1.3.1 and Etcd before installing vulcand:

Install:

make install make run

34 Chapter 2. User Manual CHAPTER 3

Middlewares

Vulcand allows you to compile in middlewares that can change, intercept or reject the request or even alter the response, and provides vulcanbundle command line tool to make it easy.

3.1 Middlewares

Middlewares are allowed to observe, modify and intercept http requests and responses. Each middleware is fully compatible with Go standard library http.Handler interface:

type Handler interface { ServeHTTP(ResponseWriter, *Request) }

• ServeHTTP is called before the request is going to be forwarded to the server selected by the load balancer. This function can modify or intercept request before it gets to a ﬁnal destination and change the response.

3.2 Middleware Chains

Middleware chains deﬁne an order in which middlewares will be executed. Each Middleware handler will explicitly call next handler:

func (h *MyHandler) ServeHTTP(w http.ResponseWriter,r *Request){ // do something with the request and pass to the next handler h.next.ServeHTTP(w,r) }

In case if middleware decides to reject the request, it should not call next, and instead write it’s own response:

func (h *MyHandler) ServeHTTP(w http.ResponseWriter,r *Request){ io.WriteString(w, "access denied") (continues on next page)

35 Vulcand REST API Documentation, Release 2.0

(continued from previous page) io.WriteHeader(403) return }

Example of a request that passes auth and limiting middlewares and hits the server:

Example of a request that is rejected by limiting middleware: In this case server won’t process the request.

3.3 Vbundle

Vbundle is a very simple command line tool that creates a new version of vulcand daemon by writing a new main.go that imports the vulcand packages and your plugins: import ( "github.com/vulcand/vulcand" "github.com/example/extension" (continues on next page)

36 Chapter 3. Middlewares Vulcand REST API Documentation, Release 2.0

3.3. Vbundle 37 Vulcand REST API Documentation, Release 2.0

(continued from previous page) ) func main(){ vulcand.RegisterPlugin(extension.Spec()) vulcand.Run() }

Vbundle does not download anything from the internet, it just generates the program for you. To compile, the packages and dependencies should be in your working environment.

3.3.1 Example - Auth middleware

In this example, we will write a new fully functional (but insecure) middleware that will require all requests to be authorized with HTTP basic auth. For this demo, we’ve created auth package in our environment. Full source code is available at: http://github.com/vulcand/vulcand-auth

Describing your middleware

Vulcand looks for a special function GetSpec that returns all that it needs to know about this middleware. import ( "github.com/vulcand/vulcand/plugin" ) const Type= "auth" func GetSpec() *plugin.MiddlewareSpec{ return &plugin.MiddlewareSpec{ Type: Type, // A short name for the middleware FromOther: FromOther, // Tells vulcand how to create middleware from

˓→another one FromCli: FromCli, // Tells vulcand how to create middleware from

˓→CLI CliFlags: CliFlags(), // Vulcand will add this flags CLI command } }

We will get back to each one of this functions later, for now let’s create a middleware struct itself. Our Auth plugin is a struct that stores username and password.

Note: Your struct should contain only values expected in: http://golang.org/pkg/encoding/json/#Marshal

// AuthMiddleware struct holds configuration parameters and is used to // serialize/deserialize the configuration from storage engines. type AuthMiddleware struct { Password string Username string }

38 Chapter 3. Middlewares Vulcand REST API Documentation, Release 2.0

Handler

Let’s deﬁne ServeHTTP handler that will be called on each request and response.

// Auth middleware handler type AuthHandler struct { cfg AuthMiddleware next http.Handler }

// This function will be called each time the request hits the location with this

˓→middleware activated func (a *AuthHandler) ServeHTTP(w http.ResponseWriter,r *http.Request){ auth, err := utils.ParseAuthHeader(r.Header.Get("Authorization")) // Reject the request by writing forbidden response if err != nil ||a.cfg.Username != auth.Username ||a.cfg.Password != auth.

˓→Password{ w.WriteHeader(http.StatusForbidden) io.WriteString(w, "Forbidden") return } // Pass the request to the next middleware in chain a.next.ServeHTTP(w,r) }

Note: it is important to call next handler if you want to pass the request to the server

As you may noticed, AuthHandler is fully compatible with http.Handler. This is very handy as you may now re-use all these middlewares available in the internet, like this one: • https://github.com/codahale/http-handlers • https://github.com/vulcand/oxy

Utility functions

Let’s deﬁne some other important functions as the next step.

// This function is optional but handy, used to check input parameters when creating

˓→new middlewares func New(user, pass string)( *AuthMiddleware, error){ if user =="" || pass ==""{ return nil, fmt.Errorf("Username and password can not be empty") } return &AuthMiddleware{Username: user, Password: pass}, nil }

// This function is important, it's called by vulcand to create a new handler from

˓→the middleware config and put it into the // middleware chain. Note that we need to remember 'next' handler to call func (c *AuthMiddleware) NewHandler(next http.Handler)(http.Handler, error){ return &AuthHandler{next: next, cfg: *c}, nil }

// String() will be called by loggers inside Vulcand and command line tool. (continues on next page)

3.3. Vbundle 39 Vulcand REST API Documentation, Release 2.0

(continued from previous page) func (c *AuthMiddleware) String() string { return fmt.Sprintf("username=%v, pass=%v",c.Username," ********") }

Constructors

As the ﬁnal step, lets deﬁne the functions required by GetSpec, these ones will be called by Vulcand when it will need to create new auth middlewares:

// FromOther Will be called by Vulcand when engine or API will read the middleware

˓→from the serialized format. // It's important that the signature of the function will be exactly the same,

˓→otherwise Vulcand will // fail to register this middleware. // The first and the only parameter should be the struct itself, no pointers and

˓→other variables. // Function should return middleware interface and error in case if the parameters

˓→are wrong. func FromOther(c AuthMiddleware)(plugin.Middleware, error){ return New(c.Username,c.Password) }

// FromCli constructs the middleware from the command line func FromCli(c *cli.Context)(plugin.Middleware, error){ return New(c.String("user"),c.String("pass")) }

// CliFlags will be used by Vulcand construct help and CLI command for the vctl

˓→command func CliFlags()[]cli.Flag{ return []cli.Flag{ cli.StringFlag{"user, u","", "Basic auth username",""}, cli.StringFlag{"pass, p","", "Basic auth pass",""}, } }

Imports

Let’s take a look at all imports used in the http://github.com/vulcand/vulcand-auth

// Note that I import the versions bundled with vulcand. That will make our lives

˓→easier, as we'll use exactly the same versions used // by vulcand. We are escaping dependency management troubles thanks to Godep. import ( "fmt" "io" "net/http"

"github.com/vulcand/vulcand/vendor/github.com/codegangsta/cli" "github.com/vulcand/vulcand/vendor/github.com/vulcand/oxy/utils" "github.com/vulcand/vulcand/plugin" )

40 Chapter 3. Middlewares Vulcand REST API Documentation, Release 2.0

Vulcand uses Godep to manage it dependencies, we encourage you to read it’s docs. That’s why all imports to shared libraries are pointing to versions vendored with the version of vulcand you are using.

Getting it together

Let’s create a folder in GOPATH environment that will be used for your version of Vulcand compiled with the new middleware. In our case it looks like this:

/home/alex/goworld/src/github.com/vulcand/vulcand-bundle

In your case it would be something different. Now execute the vbundle command, you should see something like this:

$ vbundle init --middleware=github.com/vulcand/vulcand-auth/auth Dec 26 01:02:57.180: INFO PID:16442[main.go:51] SUCCESS: bundle vulcand and vctl

˓→completed

Note: --middleware=github.com/vulcand/vulcand-auth/auth ﬂag tells the tool to include our auth middleware into bundle.

Let us check that it actually did something:

$ ls main.go registry vctl

What just happened? Vbundle wrote a new version of main.go and vctl that have our auth middleware plugged in. The ﬁnal step would be to install the bundle. Let’s cd to the vulcand-bundle dir and execute the build commands

$ go build -o vulcand $ pushd vctl/&& go build -o vctl&& popd

Congrats! Now you have your version of vulcand and vctl Let’s use it and try a couple of tricks with the new middleware to ensure it actually works. We need to start the new binary and configure the server first. (If you have it configured, just skip configuration steps below)

# start the daemon $ ./vulcand -etcd http://localhost:4001

# add host, location and upstream with endpoints via newly compiled command line tool $ ./vctl/vctl backend upsert -id b1 $ ./vctl/vctl server upsert -id srv1 -b b1 -url http://localhost:5000 $ ./vctl/vctl frontend upsert -id f1 -b b1 -route 'Path("/")'

Using Auth middleware

Now to the fun part, you can conﬁgure the new authorization. Make sure that auth command is now available:

$ ./vctl/vctl auth --help $ ./vctl/vctl auth upsert --help

Let’s play with the new feature!

3.3. Vbundle 41 Vulcand REST API Documentation, Release 2.0

# add the auth $ ./vctl/vctl auth upsert -f f1 -user=hello -pass=world OK: auth upserted

# try it out $ curl http://localhost:8181/ Forbidden

# ok, here you go: $ curl -u hello:world http://localhost:8181/ ok

Etcd

Note that you can use Etcd to conﬁgure the new middleware, vulcand will load it from json representation:

$ etcdctl set /vulcand/frontends/f1/middlewares/auth1 '{"Type": "auth", "Middleware":{

˓→"Username": "user", "Password": "secret1"}}'

Testing

We were in a hurry trying to get it all working, didn’t we? Now we can step back and cover it all by tests: https://github.com/vulcand/vulcand-auth/blob/master/auth/auth_test.go

Maintenance

Note that vbundle simply writes the new vulcanbundle for you, and does not try to get in your way of managing your environment. However, here are a couple of tips if you feel like you need ones: • Your new bundle is just a go program like any other one out there. • This means that you can add it to your repo, and update periodically by calling vbundle init with new parameters. • You can use godep save -r ./... with new bundle as well to make sure your bundle’s deps are always there.

42 Chapter 3. Middlewares CHAPTER 4

API Reference

Vulcan’s HTTP API is the best way to configure one or several instances of Vulcan at the same time. Essentially it’s a tiny wrapper around the backend. Multiple Vulcand instances listening to the same prefix would detect changes simultaneously and reload configuration.

4.1 Status

Status endpoint is handy when you need to integrate Vulcand with another load balancer that can poll Vulcand to route the trafﬁc based on the healthcheck response.

4.1.1 Check status

GET /v2/status

Returns: 200 OK

{ "Status": "ok" }

4.2 Log severity

Log severity endpoint allows to change the logging severity for a running instance

4.2.1 Get severity

43 Vulcand REST API Documentation, Release 2.0

GET /v2/log/severity

Returns: 200 OK

{ "Severity": "WARN" }

4.2.2 Set severity

PUT 'multipart/form-data' /v2/log/severity

Parameter Description severity Severity - WARN, INFO or ERROR

Returns: 200 OK

{ "Message": "Severity has been updated to INFO" }

4.3 Host

4.3.1 Get hosts

GET /v2/hosts

Example response:

{ "Hosts":[ { "Name":"localhost", "Settings":{ "KeyPair":null, "Default":false } } ] }

4.3.2 Upsert host

POST 'application/json' /v2/hosts

Add a host to the proxy.

44 Chapter 4. API Reference Vulcand REST API Documentation, Release 2.0

{ "Host": { "Name": "localhost", // hostname "Settings": { // settings are optional "KeyPair": {"Cert": "base64", Key: "base64"}, // base64 encoded key-pair

˓→certificate "Default": false , // default host for SNI } }

} Example responses:

{ "Name": "localhost", "Settings":{ "KeyPair": null, "Default": false } }

4.3.3 Delete host

DELETE /v2/hosts/

Delete a host.

4.4 Listener

4.4.1 Upsert listener

POST 'application/json' /v2/listeners

Upsert listener

{ "Listener": { "Id": "l1", "Protocol": "https", // http or https "Address": { "Network":"tcp", // unix or tcp "Address":"localhost:8184" } } }

Example response:

{ "Id": "12", (continues on next page)

4.4. Listener 45 Vulcand REST API Documentation, Release 2.0

(continued from previous page) "Protocol": "https", "Address": { "Network":"tcp", "Address":"localhost:8184" } }

4.4.2 Delete listener

DELETE /v2/listeners/

Delete a listener

4.5 Backend

4.5.1 Get backends

GET /v2/backends

Retrieve the existing upstreams. Example response:

{ "Backends":[ { "Id": "b1", "Type": "http", "Settings":{ "Timeouts":{ "Read":"", "Dial":"", "TLSHandshake":"" }, "KeepAlive":{ "Period":"", "MaxIdleConnsPerHost":0 } } } ] }

4.5.2 Upsert backend

POST 'application/json' /v2/backends

{ "Backend":{ "Id": "b1", (continues on next page)

46 Chapter 4. API Reference Vulcand REST API Documentation, Release 2.0

(continued from previous page) "Type": "http", "Settings":{ "Timeouts":{ "Read": "5s", "Dial": "5s", "TLSHandshake": "10s" }, "KeepAlive":{ "Period": "30s", "MaxIdleConnsPerHost": 12 } } } }

Example response:

{ "Id": "b1", "Type": "http", "Settings":{ "Timeouts":{ "Read": "5s", "Dial": "5s", "TLSHandshake": "10s" }, "KeepAlive":{ "Period": "30s", "MaxIdleConnsPerHost": 12 } } }

4.5.3 Delete backend

DELETE /v2/backends/

4.6 Server

4.6.1 Get servers

GET /v2/backends//servers

Retrieve the servers of the backend. Example response:

{ "Servers":[ { "Id": "srv1", "URL": "http://localhost:5000" }, (continues on next page)

4.6. Server 47 Vulcand REST API Documentation, Release 2.0

(continued from previous page) { "Id": "srv2", "URL": "http://localhost:5003" } ] }

4.6.2 Get server

GET /v2/backends//servers/

Retrieve the particular server with id server-id

4.6.3 Upsert server

POST /v2/backends//servers

Upsert server to the backend

{ "Server":{ "Id": "srv1", "URL": "http://localhost:5000" } }

Example response:

{ "Id": "e4", "Url": "http://localhost:5004", "Stats": null }

4.6.4 Delete server

DELETE /v2/backends//servers/

Delete a server.

4.7 Frontend

4.7.1 Get frontends

GET /v2/frontends

Retrieve the frontends. Example response:

48 Chapter 4. API Reference Vulcand REST API Documentation, Release 2.0

{ "Frontends":[ { "Id": "f1", "Route": "Path(`/`)", "Type": "http", "BackendId": "b1", "Settings":{ "Limits":{ "MaxMemBodyBytes":0, "MaxBodyBytes":0 }, "FailoverPredicate":"", "Hostname":"", "TrustForwardHeader": false } } ] }

4.7.2 Get frontend

GET /v2/frontends/

Retrieve the particular frontend with id frontend-id

{ "Id": "f1", "Route": "Path(`/`)", "Type": "http", "BackendId": "b1", "Settings":{ "Limits":{ "MaxMemBodyBytes":0, "MaxBodyBytes":0 }, "FailoverPredicate":"", "Hostname":"", "TrustForwardHeader": false } }

4.7.3 Upsert frontend

POST 'application/json' /v2/frontends

Add a frontend to the host. Params:

{ "Frontend":{ "Id": "f1", "Route": "Path(`\/`)", "Type": "http", (continues on next page)

4.7. Frontend 49 Vulcand REST API Documentation, Release 2.0

(continued from previous page) "BackendId": "b1", "Settings":{ "Limits":{ "MaxMemBodyBytes":0, "MaxBodyBytes":0 }, "FailoverPredicate":"", "Hostname":"", "TrustForwardHeader": false } } }

Example response:

4.7.4 Delete frontend

DELETE /v2/frontends/

Delete a frontend.

4.8 Rate limit

4.8.1 Get rate limit

GET /v2/frontends//middlewares/

Retrieve the particular rate of frontend with id frontend-id and rate id rate-id Example response:

{ "Id": "rl1", "Priority":0, "Type": "ratelimit", "Middleware":{ "PeriodSeconds":1, (continues on next page)

50 Chapter 4. API Reference Vulcand REST API Documentation, Release 2.0

(continued from previous page) "Burst":3, "Variable": "client.ip", "Requests":1 } }

4.8.2 Upsert rate limit

POST 'application/json' /v2/frontends/middlewares

Add a rate limit to the frontend, will take effect immediately.

{ "Middleware":{ "Id": "rl1", "Priority":0, "Type": "ratelimit", "Middleware":{ "PeriodSeconds":1, "Burst":3, "Variable": "client.ip", "Requests":1 } } }

Json parameters explained:

Parameter Description Id Optional rate id, will be generated if omitted Requests Required amount of allowed requests PeriodSeconds Required period in seconds for counting the requests Burst Required allowed burst of the requests (additional requests exceeding the rate) Variable Variable for rate limiting e.g. client.ip or request.header.My-Header

4.8.3 Delete a rate limit

DELETE /v2/frontends//middlewares/

Deletes rate limit from the frontend.

4.9 Connection limit

4.9.1 Get connection limit

GET /v2/frontends//middlewares/

Retrieve the particular connection limit of frontend with id frontend-id and connection limit id conn-id. Ex- ample response:

4.9. Connection limit 51 Vulcand REST API Documentation, Release 2.0

{ "Id": "cl1", "Priority":0, "Type": "connlimit", "Middleware":{ "Connections":3, "Variable": "client.ip" } }

4.9.2 Upsert connection limit

POST 'application/json' /v2/frontends//middlewares

Upsert a connection limit to the frontend. Example response:

{ "Middleware":{ "Id": "cl1", "Priority":0, "Type": "connlimit", "Middleware":{ "Connections":3, "Variable": "client.ip" } } }

JSON parameters explained

Parameter Description Id Optional limit id, will be generated if omitted.| Connections Required maximum amount of allowed simultaneous connections| Variable Variable for limiting e.g. client.ip or request.header.My-Header

4.9.3 Delete connection limit

DELETE /v2/frontends//middlewares/

Delete a connection limit from the frontend.

52 Chapter 4. API Reference