Davix Data Management Library for WebDav/Http Devresse Adrien Furano Fabrizio IT-GT-DMS What is Davix ? Davix is NOT : - Yet an other HTTP Library - an other Grid specific software Davix is : - a toolkit for optimized remote I/O - support all HTTP based protocols → S3 / WebDav / (CDMI ? ) Why Davix ? Downloading Web Content ≠ Remote random I/O → Wget/Curl/Aria2c are good to download Full files.. → Efficient Remote random I/O is far more complex Doing efficient remote I/O implies → HTTP request tuning → Server implementation knowledges → Caching Why Davix ? We need a data-management toolkit for our customers if we wish to promote HTTP usage for Remote I/O → We need a tool similar to the xrd client libs / tools → One performant reference implementation → None of the current solutions are acceptables → No one want to reimplement the wheel Davix Goals → Transparent optimizations →Thread-safe HTTP/SSL session reuse → Transparent caching → Vector operations → Auto detection of server capabilities →Parallel stream support → Reliability of data access → Metalink → Configurable « retry policy » Davix Goals → Simplified file specific operations : → HTTP Staging file operation ? → Checksum Calculation → Right management : ACL ? → Security : proxy certificate + voms exts → Third party copy support → Cluster environment → redirection caching on every operation. Davix Goals → One Implementation for the « extensions » → Future support for metalink : → Multi-source and multi-link download. → Geolocalized download ? → Advanced Failover management Davix Implementation → Fully in C++ → Portable : → Debian / SL / Fedora , Mac OSX, Windows, FreeBSD → Already pkg on Fedora / Debian → Based on libneon for the low-level Http/Dav queries. → Reliable official http/webdav library. → No dependencies, just the minimum : → OpenSSL, libxml. → Keep It Simple Stupid Davix C++ Example # setup context Davix::Context handle; Davix::Posix pos(handle); # setup parameters Davix::RequestParams params ; ... # list directory content query DAVIX_DIR* d = pos.opendir(&params, « my_url »); struct dirent * dir = pos.readdir(d); # read file on remote Storage DAVIX_FD* fd = pos.open(« params », « my_url », flags ») ; Pos.read(fd, buffer, size ); Pos.close(fd) ; Davix Architecture Posix like API File like API Davix DAVIX Core C++ Request Abstraction Layer S3/Dav Xml Parser Dependencies libneon libxml Davix has been already tested on → LCGDM DPM/LFC Webdav frontend → DMLITE Webdav frontend → dCache DAV → python DavServer → Apache 2 standard mod-dav → Http based Federation : Ugr → will be tests with all common Webdav endpoints. → Automated functional tests. Davix Status Under active devlopment → evolve quickly, but stable API Already used by some projects → FTS 3.0 → GFAL 2.0 → UGR → https://svnweb.cern.ch/trac/lcgdm/wiki/Dynafeds Collaborative development : → available on GIT → all contributions are welcome Future Release Davix 0.3.0 - Vector and random write I/O - HTTP Patch - SOCKS 5 support - Improved caching for small chunks - Metalink transparent failover - Improved cmd line tools - Support for Zero Copy architecture ( splice() ) - code simplification TDavixFile → ROOT is an important actor in HEP for data analysis → Current implementation for HTTP for ROOT ( TWebFile / TS3File / … ) are quite poor and buggy → No Write support → Bug in redirection handle → No VOMS / Proxy cred support TDavixFile → New HTTP/WebDAV/S3 plugin relying on DAVIX → Implement everything that you dream about → Already Implemented and tested → should be merge in the next version TDavixSystem → Implementation of All meta-data operations for ROOT → readdir / stat / mkdir / unlink → Work with Webdav / Amazon S3 Questions ? Site : https://svnweb.cern.ch/trac/lcgutil/wiki/davix Contact : [email protected] Myself : [email protected] .