Samba's Cloudy Future Jeremy Allison Samba Team [email protected]

Samba Team Jeremy Allison Jeremy [email protected] Samba's Cloudy Future

O p e n i n g W i n d o w s t o a W i d e r W o r l d

apps. Isn't Isn't cloud storage the future? Yes, but not usable for many existing Yes, O p e n i n g W i n d o w s t o a W i d e r W o r l d Cloud Storage blobis a store Apps are changing to cope with no random access semantics of cloud stores, but this will take time. Blob stores don't map very well onto the open/read/write/close random access semantics of most applications. ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d Samba can cope with with cope can Samba theory Might we become a FUSE host ? host FUSE a become we Might No, because FUSE system calls calls system FUSE because No, long awfully an for block can time.. In async. syscalls all making by this So can't Windows clients access clients Windows can't So a of top on running Samba via ? filesystem FUSE Doesn't FUSE solve this ? – – – – No FUSE on Windows clients. Mostly, for Linux andMacOS X Mostly, applications. ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d :-) need a way to ensure we sync to the the sync to we ensure to way a :-)need just File/directory creation/deletion, . creation/deletion, File/directory We We update metadata on cloud remote – – Enter Enter flexibleSamba What should we do with theextra metadata (ACLs etc.) ? Already contains code to cache all reads/writes onto local files. The VFS can save us ! easily modifiable user-space code. Stackable, modular, ● ● ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d To cloud To (https) cloud provider backend Pluggable daemon Cloud gateway Local file access Into Cache Disk Metadata sync protocol VFS Regular VFS Possible Architecture Possible smbd Cloud Local disk cache (shared amongst smbds) To client To (SMB2) O p e n i n g W i n d o w s t o a W i d e r W o r l d provide ? If a client writes then closes, they are guaranteed to to guaranteed are they closes, then writes client a If storage. cloud to uploaded version that have synced. not files Open Latency going to the the cloud would make this this make would cloud the the to going Latency unusable. Samba- on-premises same the to going clients Only semantics. Windows provide gateway Cloud – – – – full Windows semantics. What level ofWhat can we coherence level Good-enough” semantics for many apps. “ Coherence onclose/unlink/mkdir/create only. NOT ● ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d Under the covers On non COW filesystems, physical copies of files files of copies physical filesystems, COW non On needed. – can be synchronized out to cloud storage. If we have a underlying btrfs copy-on-write filesystem, use file cloning on close to create instant copies that move to/from private namespace into smbd-exported namespace. physical device. On sync use atomic file operations (rename, unlink) to Cloud services daemon reads/writes into private namespace hidden fromsmbd, but onthe same ● ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d the cloud. the here. help can FILE_ATTRIBUTE_OFFLINE At least fetching file metadata (size etc.) must be be must (size etc.) metadata file fetching least At synchronous. be then can (read/write) operations access data File from synchronized is data the until async go to forced On create success local file/directory created. file/directory local success create On ? operations cloud ignore TMP, marked For files Requires synchronous call to cloud backend. cloud to call synchronous Requires – – – – – – Operations – Operations Open/Create/Mkdir– from cloud. For open existing, open operation must triggerread Create operations must sync with cloud daemon. ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d

Operations–

No cloud communication needed. communication cloud No – Read/Write/Truncate/Unlink/Rename cloud, but should be a reasonably quick operation (just round-trip latency). Unlink and rename need synchronous operation to the Once the on-disk file is known to be ina valid state, reads, writes and truncates can proceed as normal. ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d Operations Operations - Close Should we add a delay time here ? Only guarantee guarantee Only ? here time delay a add we Should ? minutes 5 after coherence Once copy is complete, smbd can reply to the close close the to reply can smbd complete, is copy Once request. Use COW if available. if COW Use – – – cloud. Cloud sync daemon then pushes changed file up to the If file modified then make a sync call to the cloud sync daemon to make a copy of this file. Reference count open files – on last close then expensive synchronous operations need to happen. ● ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d The cloud daemonsync Just expect credentials file to have been magically magically been have to file credentials expect Just administrator. an by locally placed All smbd's will need to talk to it. to talk to need will smbd's All recovery. crash for transactions tdb Use – – – Many possible tunables – number of simultaneous connections, bandwidth limits etc. storage providers. Ignore key management/credentials issues. Create plugable back-ends to allow choice cloudof Using existing Samba technologies talloc, asynchronous tevent, with backing threadpool. ● ● ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d If so, how do we queue operations that occur during occur during that operations queue we do how so, If cloud outage ? outage cloud replay to order in allow we do queuing much How ? access client stopping before later operations Do we allow client access to continue whilst there is there whilst continue to access client allow we Do ? outage cloud a – – – Error recovery I don't currently have good answers to these problems. Shock, horror. Cloud file operations can fail, right in theShock, horror. middle of uploading via https. ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d ” - Bruce Schneier Schneier We have secured ourselveshave secured from the We fuse module. fuse cloud. to uploading before filesystem encfs to Copy vfs_encfs – Samba VFS module that emulates encfs encfs emulates that module VFS Samba – vfs_encfs – – know about or can't talk about.orcan't about know NSA, except for exceptNSA, parts the eitherthat don't we only way to reduce this risk. optionscan I see: GettingTwo this right is hard. Local encryption before uploading to the cloud is the encrypted withkeys held by the cloudstorage provider can be compromised. Data stored in the cloud unencrypted, or remotely Privacy: " Privacy: ● ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d Three CloudStore Vendors Create backends that backends Create target the Big

O p e n i n g W i n d o w s t o a W i d e r W o r l d code already. Unfortunately they're not releasing it it releasing not they're Unfortunately already. code asked). I've far, (so community the to back Proprietary cloud-gateways provide more complete complete more provide cloud-gateways Proprietary guarantees. service this has UK the in University Loughborough Yes. No – In doing this I'm trying to raise the bar on what what on bar the raise to trying I'm this doing In – No Samba. with provided “basic” services the are – – – Existing vendors Are you reinventing the wheel ? Are we trying to compete with ? them purpose. There are manyexisting cloud-gateway companies, some of which are already using Samba for this ● ● ● O p e n i n g W i n d o w s t o a W i d e r W o r l d - [email protected] Questions andQuestions Comments ?

future.odp ftp://samba.org/pub/samba/slides/sambaxp-2015-cloudy Slides available at: Email: O p e n i n g W i n d o w s t o a W i d e r W o r l d