INTRODUCTION TO PYDIO CELLS

CS3 2019 Rome - Charles du Jeu

Pydio | Enterprise File Sharing Platform CS3 2019 - ROME Table of Contents

Introduction Pydio Overview

Breaking the monolith Building a future-proof architecture

Major Features Comparison with Pydio 8, a glance at the roadmap

Questions?

Pydio | Enterprise File Sharing Platform INTRODUCTION Pydio: sync & share open source software

Pydio | Enterprise File Sharing Platform INTRODUCTION Who am i ? [Github] cdujeu [Twitter] @charlesdujeu

Pydio CEO/CTO, I am a Software Engineer…

…turned Musician + libriste afficionado created AjaXplorer in 2008 (remember ajax?)

…turned Entrepreneur created company in 2012 (and rebranded to Pydio)

…turned Gopher! started rewritting Pydio from scratch in 2017

Pydio | Enterprise File Sharing Platform INTRODUCTION Pydio: a 10y.o. project legacy Open Source file sync and share platform

Pydio | Enterprise File Sharing Platform INTRODUCTION Pydio: a 10y.o. project legacy The PHP love/hat relationship

PHP has proven a good choice over the years Community easy enrollment Scripting language Perfs improvements in last version. More PSR’s & Frameworks over the years

But manipulating files requires more … Pydio is all about unstructured data (getting bigger everyday) PHP script approach: incompressible load-time Monolith is hard to maintain, hard to scale Dependencies-crippled software

Pydio | Enterprise File Sharing Platform Time to move on ! INTRODUCTION Rewriting scope Objectives and constraints

Features Iso-perimeter Enterprise-oriented features Rich Web UI: do not rewrite the ReactJS Frontend

Ease deployment 80% of our support is about environment, not Pydio

Scalability Data access virtualisation Ease Load balancing Big files management

Development Time / Cost Quality

Pydio | Enterprise File Sharing Platform BREAKING THE MONOLITH Introducing Pydio Cells

Pydio | Enterprise File Sharing Platform BREAKING THE MONOLITH Pydio Cells - Go Microservices

Pydio | Enterprise File Sharing Platform Solid foundations: micro-services best-practices / 12 factors

Pydio | Enterprise File Sharing Platform Data: scalability and redundancy

Pydio | Enterprise File Sharing Platform Identity management Role-based ACLs Policy-based ACLs

Authentication Authorizations OpenID Connect

Pydio | Enterprise File Sharing Platform Exposed using a unified gateway

Pydio | Enterprise File Sharing Platform BREAKING THE MONOLITH Pydio Cells

Pydio | Enterprise File Sharing Platform ARCHITECTURE Easing the deployment Embedded webserver - Using standards everywhere

S3 Protocol (data PUT/GET) OpenID Connect (authentication) OpenAPI/Swagger (Rest) ActivityStreams 2.0 GRPC/Protobuf

Pydio | Enterprise File Sharing Platform ARCHITECTURE Easing the deployment A one-liner command - No dependencies except MySQL

Pydio | Enterprise File Sharing Platform ARCHITECTURE Data accuracy and redundancy DataSources events are driven by the file system

HTTP PUT Object Request

One source of truth

Pydio | Enterprise File Sharing Platform ARCHITECTURE Data scalability Add as many datasources as needed

Pydio | Enterprise File Sharing Platform ARCHITECTURE We love DevOps! Integrating in modern infrastructures

JSON formatted logs to be Instrumented code for scraping Container-ready aggregated by ELK stack metrics in Prometheus / Grafana Docker / Kubernetes

Pydio | Enterprise File Sharing Platform BREAKING THE MONOLITH So we got rid of the LAMP stack

L A M P

Cross-compilation Std Lib net/http Each micro-service has 100% GO! No-dep binary External libs its own persistence layer Caddy Embedded

$ GOOS=darwin; go build .

Pydio | Enterprise File Sharing Platform FEATURES OVERVIEW Major changes compared with Pydio 8

Pydio | Enterprise File Sharing Platform FEATURES Cells Cells are like channels in Slack

Cells are created by end-users - From scratch - From one or many existing folders - With one or many users

Inside Cell, files authorizations still respect the admin-defined ACLs

Cells provide a chat room with users it is shared with.

Cells ownership can be shared/transfered to other users.

Pydio | Enterprise File Sharing Platform FEATURES Security Policies A generic framework to compute permissions

{ Following a pattern similar to IAM Resources of Amazon, the policy engine is "description": "One policy to rule them all.", aggregating a set of security rules to answer the following question: "subjects": ["users:", "users:maria"], "actions" : ["delete", ""], Who is able to do what on something given some context "effect": "allow", "resources": [

"resources:articles:<.*>", • Who: a user, a profile or a role identifier "resources:printer" ], • Able: The effect, which can be either "allow" or "deny". "conditions": { "remoteIP": { • What: An arbitrary action name, "delete", "create" or « scoped:action:something". "type": "CIDRCondition", "options": { • Something: An arbitrary unique resource name, for example "something", "cidr": "192.168.0.1/16" "resources.articles.1234" or some uniform resource name like « urn:isbn:3827370191". } } } • Context: The current context containing information about the environment such as } the IP Address, request date, the resource owner name, etc.

Roles + ACLs REST APIs Protection Users/Shares visibility

Pydio | Enterprise File Sharing Platform FEATURES Security Policies Admin Console

Pydio | Enterprise File Sharing Platform FEATURES Sharding Data accross datasources Template Paths can compute internal Paths dynamically based on the context

Default value used for « My Files » workspaces

Advanced example for sharding data based on user first name

Pydio | Enterprise File Sharing Platform ADVANCED FEATURES Versioning Policies Securing users data while maintaining storage usage reasonable

Files Versioning Activating file versioning will create a copy of a file at each modification. Currently there is no « delta » algorithm: the file is copied fully in a dedicated datasource.

This can create high storage usage and it is necessary to regulate the number of versions you may keep. DEDICATED DATASOURCE Versioning polices are defining Retention Periods. Policies are applied on a per-datasource basis.

Pydio | Enterprise File Sharing Platform FEATURES Auditing Tools GDPR Compliance and better control

Dedicated Audit Log - Exportable as XLS Shares Reports and Audits

Pydio | Enterprise File Sharing Platform FEATURES

And many other features….

Pydio | Enterprise File Sharing Platform ADVANCED FEATURES Roadmap What’s next ?

Very next version

Migration Tool from Pydio 8

Later this year

OpenIDConnect deeper integration: turn Cells to a proper IdP

Rewire Sync Client

IFTTT, Lambda functions and other event-based integrations

Pydio | Enterprise File Sharing Platform QUESTIONS?

github.com/cdujeu twitter.com/@charlesdujeu linkedin.com/in/charles-du-jeu-a679581/

Pydio | Enterprise File Sharing Platform BREAKING THE MONOLITH How we performed the transition How we ensured a smooth transition on a 1,5 year project

Pydio | Enterprise File Sharing Platform OTHER TOPICS Integration Tests How we achieve automatic deployments and tests

Pydio | Enterprise File Sharing Platform