The Varnish Book Request Restart Frontend (From Client) Workthread vcl_recv lookup vcl_hash hit miss purge pipe busy vcl_hit pass hit-for-pass vcl_miss vcl_purge vcl_pipe waiting vcl_pass Backend (To Server) Workthread vcl_backend_fetch read beresp(headers) vcl_backend_response vcl_backend_error cacheable? yes hit-for-pass cache do not cache vcl_deliver vcl_synth Done RESTART cnt_restart: Request received ESI_REQ ok? max_restarts? cnt_recv: vcl_recv{} req.* SYNTH hash purge pass pipe synth cnt_recv: vcl_hash{} req.* lookup cnt_pipe: cnt_lookup: filter req.*->bereq.* cnt_purge: hash lookup (waitinglist) req.* vcl_purge{} req.* vcl_pipe{} hit? miss? hit-for-pass? busy? bereq.* synth restart pipe synth cnt_lookup: req.* send bereq, vcl_hit{} obj.* copy bytes until close deliver miss restart synth pass cnt_miss: vcl_miss{} req.* fetch synth restart pass parallel if obj expired cnt_pass: vcl_pass{} req.* fetch synth restart BGFETCH FETCH see backend graph FETCH_DONE FETCH_FAIL SYNTH cnt_deliver: cnt_synth: Filter obj.->resp. req.* stream? req.* vcl_synth{} vcl_deliver{} resp.* body resp.* deliver restart restart deliver synth V1D_Deliver DONE FETCH BGFETCH RETRY vbf_stp_startfetch: vcl_backend_fetch{} bereq.* abandon fetch send bereq, read beresp (headers) vbf_stp_startfetch: bereq.* vcl_backend_response{} beresp.* retry deliver abandon max? ok? 304? other? vbf_stp_error: vbf_stp_condfetch: vbf_stp_fetch: bereq.* vcl_backend_error{} copy obj attr setup VFPs error abandon beresp.* RETRY steal body fetch retry deliver fetch_fail? ok? fetch_fail? error? ok? max? ok? "backend synth" FETCH_FAIL RETRY FETCH_DONE cached object t_origin TTL grace keep time fresh object stale object If-Modified-Since Object Lifetime Authors: Francisco Velázquez (Varnish Software), Kristian Lyngstøl, Tollef Fog Heen, Jérôme Renard Copyright: Varnish Software AS 2010-2015, Redpill Linpro AS 2008-2009 Versions: Documentation version 4.x-2-g757256c / Tested for Varnish plus-4.1.2r1 Date: 2016-07-28 License: The material is available under a CC-BY-NC-SA license. See http://creativecommons.org/licenses/by-nc-sa/3.0/ for the full license. For questions regarding what we mean by non-commercial, please contact [email protected]. Contact: For any questions regarding this training material, please contact [email protected]. Web: http://www.varnish-software.com/book/ Source: http://github.com/varnish/Varnish-Book/ Contents 1 Introduction 19 1.1 What is Varnish? 20 1.1.1 Varnish is Flexible 21 1.2 Varnish Cache and Varnish Plus 22 1.3 Varnish Cache and Varnish Software Timeline 24 1.4 What Is New in Varnish 4? 26 2 Design Principles 28 2.1 How objects are stored 30 2.2 Object Lifetime 31 3 Getting Started 32 3.1 Varnish Distribution 33 3.2 Exercise: Install Varnish 34 3.3 Exercise: Configure Varnish 36 3.3.1 VCL Reload 38 3.3.2 Test Varnish Using Apache as Backend 41 3.4 The Management Interface varnishadm 42 3.5 More About Varnish Configuration 44 3.6 Command Line Configuration 46 3.7 Defining a Backend in VCL 48 3.8 Exercise: Use the administration interface to learn, review and set Varnish 49 parameters 3.9 Exercise: Fetch Data Through Varnish 50 4 Examining Varnish Server's Output 51 4.1 Log Data Tools 52 4.2 Log Layout 53 4.3 Transactions 54 4.3.1 Transaction Groups 56 4.3.2 Example of Transaction Grouping with varnishlog 57 4.4 Query Language 58 4.5 Exercise: Filter Varnish Log Records 60 4.6 varnishstat 61 4.6.1 Notable Counters 65 4.7 Exercise: Try varnishstat and varnishlog together 67 5 Tuning 68 5.1 Varnish Architecture 69 5.1.1 The Parent Process: The Manager 71 5.1.2 The Child Process: The Cacher 72 5.1.3 VCL Compilation 73 5.2 Storage Backends 74 5.3 The Varnish Shared memory Log (VSL) 76 5.4 Tunable Parameters 77 5.5 Varnish Tuner 79 5.5.1 Varnish Tuner Persistence 80 5.5.2 Install Varnish Tuner 81 5.6 Threading Model 82 5.7 Threading Parameters 83 5.7.1 Details of Threading Parameters 85 5.7.2 Time Overhead per Thread Creation 86 5.8 System Parameters 87 5.9 Timers 88 5.10 Exercise: Tune first_byte_timeout 90 5.11 Exercise: Configure Threading 91 6 HTTP 92 6.1 Protocol Basics 93 6.1.1 Resources and Representations 94 6.1.2 Requests and Responses 95 6.1.3 Request Example 96 6.1.4 Response Example 97 6.2 HTTP Characteristics 98 6.3 Cache-related Headers Fields 99 6.4 Constructing Responses from Caches 100 6.5 Cache Matching 101 6.5.1 Vary 102 6.5.2 ETag 103 6.5.3 Last-Modified 104 6.5.4 If-None-Match 105 6.5.5 If-Modified-Since 106 6.6 Allowance 108 6.6.1 Cache-Control 109 6.6.2 Pragma 111 6.7 Freshness 112 6.7.1 Age 113 6.7.1.1 Exercise: Use article.php to test Age 113 6.7.2 Expires 114 6.8 Availability of Header Fields 115 6.9 Exercise: Test Various Cache Headers Fields with a Real Browser 116 7 VCL Basics 117 7.1 Varnish Finite State Machine 118 7.2 Detailed Varnish Request Flow for the Client Worker Thread 121 7.3 The VCL Finite State Machine 123 7.4 VCL Syntax 124 7.5 Built-in vcl_recv 125 7.6 VCL Built-in Functions and Keywords 127 7.7 Legal Return Actions 128 7.8 Variables in VCL subroutines 129 7.9 Detailed Varnish Request Flow for the Backend Worker Thread 130 7.10 VCL – vcl_backend_response 131 7.10.1 vcl_backend_response 132 7.10.2 The Initial Value of beresp.ttl 133 7.10.3 Example: Setting TTL of .jpg URLs to 60 seconds 135 7.10.4 Example: Cache .jpg for 60 seconds only if s-maxage is not present 136 7.10.5 Exercise: Avoid Caching a Page 137 7.10.6 Exercise: Either use s-maxage or set TTL by file type 138 7.11 Waiting State 139 7.12 Summary of VCL Basics 140 8 VCL Subroutines 141 8.1 VCL – vcl_recv 142 8.1.1 Revisiting built-in vcl_recv 144 8.1.2 Example: Basic Device Detection 145 8.1.3 Exercise: Rewrite URL and Host Header Fields 146 8.2 VCL – vcl_pass 147 8.2.1 hit-for-pass 148 8.3 VCL – vcl_backend_fetch 149 8.4 VCL – vcl_hash 150 8.5 VCL – vcl_hit 151 8.6 VCL – vcl_miss 152 8.7 VCL – vcl_deliver 153 8.8 VCL – vcl_synth 154 8.8.1 Example: Redirecting requests with vcl_synth 156 8.9 Exercise: Modify the HTTP response header fields 157 8.10 Exercise: Change the error message 158 9 Cache Invalidation 159 9.1 Purge - Bans - Cache Misses - Surrogate Keys 161 9.2 HTTP PURGE 163 9.2.1 VCL – vcl_purge 164 9.2.2 Example: PURGE 165 9.2.3 Exercise: PURGE an article from the backend 166 9.2.4 PURGE with restart return action 167 9.3 Softpurge 169 9.4 Banning 170 9.4.1 Lurker-Friendly Bans 173 9.5 Exercise: Write a VCL program using purge and ban 175 9.6 Force Cache Misses 176 9.7 Hashtwo/Xkey (Varnish Software Implementation of Surrogate Keys) 177 9.7.1 Example Using Hashtwo or Xkey 179 10 Saving a Request 181 10.1 Directors 182 10.1.1 Random Directors 184 10.2 Health Checks 185 10.2.1 Analyzing health probes 187 10.2.2 Demo: Health Probes 189 10.3 Grace Mode 190 10.3.1 Timeline Example 192 10.3.2 Exercise: Grace 193 10.4 retry Return Action 194 10.5 Saint Mode 195 10.6 Tune Backend Properties 197 10.7 Access Control Lists (ACLs) 198 10.8 Compression 200 11 Content Composition 202 11.1 A Typical Website 203 11.2 Cookies 204 11.2.1 Vary and Cookies 205 11.2.2 Best Practices for Cookies 206 11.2.3 Exercise: Handle Cookies with Vary and hash_data with HTTPie 207 11.3 Edge Side Includes 208 11.3.1 Basic ESI usage 209 11.3.2 Example: Using ESI 210 11.3.3 Exercise: Enable ESI and Cookies 211 11.3.4 Testing ESI without Varnish 212 11.4 Masquerading AJAX requests 213 11.4.1 Exercise: write a VCL that masquerades XHR calls 214 12 Varnish Plus Software Components 215 12.1 Varnish Administration Console (VAC) 216 12.1.1 Overview Page of the Varnish Administration Console 217 12.1.2 Configuration Page of the Varnish Administration Console 218 12.1.3 Banning Page of the Varnish Administration Console 219 12.2 Varnish Custom Statistics (VCS) 220 12.2.1 VCS Data Model 222 12.2.2 VCS API 225 12.2.3 Screenshots of GUI 227 12.3 Varnish High Availability (VHA) 228 12.4 SSL/TLS frontend support with hitch 229 13 Appendix A: Resources 231 14 Appendix B: Varnish Programs 232 14.1 varnishtop 233 14.2 varnishncsa 234 14.3 varnishhist 235 14.4 Exercise: Try varnishstat, varnishlog and varnishhist 236 14.5 varnishtest 237 14.5.1 The Varnish Test Case (VTC) Language 238 14.5.2 Synchronization in Varnish Tests 240 14.5.3 Running Your Varnish Test Cases 242 14.5.4 Exercise: Test Apache as Backend with varnishtest 243 14.5.5 Setting Parameters in varnishtest 244 14.5.6 Fetch Data with varnishtest 246 14.5.7 Understanding Expires in varnishtest 247 14.5.8 Example of Transactions in varnishtest 248 14.5.9 logexpect 249 14.5.10 Exercise: Assert Counters in varnishtest 251 14.5.11 Understanding Vary in varnishtest 252 14.5.12 Understanding Last-Modified and If-Modified-Since in 254 varnishtest 14.5.13 Understanding Cache-Control in varnishtest 256 14.5.14 VCL in varnishtest 258 14.5.15 PURGE in varnishtest 259 14.5.16 Cache Invalidation in varnishtest 261 14.5.17 Understanding Grace using varnishtest 262 14.5.18 Exercise: Handle Cookies with Vary and hash_data() in 264 varnishtest 14.5.19 Understanding ESI in varnishtest 265 15 Appendix C: Extra Material 267 15.1 ajax.html 268 15.2 article.php 269 15.3 cookies.php 270 15.4 esi-top.php 271 15.5 esi-user.php 272 15.6 httpheadersexample.php 274 15.7 purgearticle.php 277 15.8 test.php 278 15.9 set-cookie.php 279 15.10 VCL Migrator from Varnish 3 to Varnish 4 280 16 Appendix D: VMOD Development 281 16.1 VMOD Basics 282 16.2 varnishtest script program 283 16.2.1 VTC 284 16.2.2 Run Your Varnish Tests 286 16.3 Hello, World! VMOD 287 16.3.1 Declaring and Documenting Functions 288 16.3.2 Implementing Functions 290