Sympa Mailinglist Manager
Stefan Hornburg
GP/RW Erlangen 4th March 2020
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 1 / 56 Introduction About Sympa
Mailing list manager Almost 23 years old Legacy Perl
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 2 / 56 Introduction Applications
Mailing list manager Newsletter Marketing
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 3 / 56 Introduction Users Users
Universities Riseup (riseup.net) 20 thousand lists 3.3 million users 4.8 million subscriptions
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 4 / 56 Introduction Customers Customers
HTWK Leipzig Open Hardware Repository (CERN) Shop for riders eShop (US government) Social Security Law (USA)
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 5 / 56 Introduction Competition Email Marketing
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 6 / 56 Introduction Motivation Motivation
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 7 / 56 Introduction Motivation Motivation
Privacy Infrastructure Open Source Flexibility Costs
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 8 / 56 Introduction Flexibility Flexibility
Configuration Templates Scenarios Plugins
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 9 / 56 Sympa basics Sympa basics
List types List creation templates Scenarios
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 10 / 56 Sympa basics List types List types
Public discussion mailing list Newsletter Confidential list ...
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 11 / 56 Sympa basics List creation templates Template files
config.tt2 (Configuration) comment.tt2 (Description)
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 12 / 56 Sympa basics List creation templates Extract of newsletter configuration
visibility noconceal
send newsletter
process_archive on
archive web_access public mail_access owner
subscribe open
unsubscribe open ,notify
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 13 / 56 Sympa basics Scenarios Incoming email for newletters
# send.newsletter scenario
title.gettext Newsletter, restricted to moderators
is_editor([listname],[sender]) smtp,dkim,smime,md5 -> do_it
true() smtp,dkim,smime,md5 -> reject(reason='send_editor')
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 14 / 56 Sympa basics Scenarios Authentication methods
smtp From: address in email md Web interface authentication dkim Valid DKIM signature smime Valid S/MIME signature
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 15 / 56 Sympa in production Sympa in production
HTWK Leipzig Sympa university wide
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 16 / 56 Sympa in production Installation and requirements Installation
Sympa 6.2.16 PostgreSQL 9.4 Postfix Nginx
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 17 / 56 Sympa in production Installation and requirements Requirements
Migration from Majordomo Authentication AD/LDAP Automated mailing lists Listmasters Adaption of CD
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 18 / 56 Sympa in production Authentication AD/LDAP Authentication AD/LDAP
LDAP only
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 19 / 56 Sympa in production Authentication AD/LDAP Authentication AD/LDAP
ldap regexp .* host ldap.htwk-leipzig.de:636 bind_dn [email protected] bind_password nevairbe suffix dc=htwk-leipzig,dc=de get_dn_by_uid_filter (uid=[sender]) get_dn_by_email_filter (&(objectcategory=person)(mail=[sender]) (!(userAccountControl:1.2.840.113556.1.4.803:=2))) email_attribute mail scope sub use_tls ldaps ca_file /etc/ssl/certs/htwk-chain.pem ca_verify required
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 20 / 56 Sympa in production Automated mailing lists Automated mailing lists
LDAP queries List templates
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 21 / 56 Sympa in production Automated mailing lists LDAP query for specific seminar groups
(&(objectcategory=person)(extensionAttribute3=08FOO15) (mail=*)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 22 / 56 Sympa in production Automated mailing lists Get all seminar groups
(&(objectcategory=person)(extensionAttribute3=*)(mail=*) (!(userAccountControl:1.2.840.113556.1.4.803:=2)))
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 23 / 56 Sympa in production Automated mailing lists LDAP with Perl
Net::LDAP Net::LDAP::Control
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 24 / 56 Sympa in production Automated mailing lists LDAP with Perl
# apply page and sort controls $page = Net::LDAP::Control::Paged->new( size => 500 ); $sort = Net::LDAP::Control::Sort->new( order => "cn" );
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 25 / 56 Sympa in production Automated mailing lists Adjust list names
# replace / and ä ö ü ß use Encode 'decode'; use Lingua::DE::ASCII 'to_ascii';
$group =~ s/\//_/g; $group = lc( to_ascii( decode( 'UTF-8', $attribute ) ) );
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 26 / 56 Sympa in production Automated mailing lists List creation
List directory List from template Sync subscribers Update email aliases
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 27 / 56 Sympa in production Adaption of CD Adaption of CD
Title and Logo logo_html_definition CSS template web_tt2/css.tt2 Header template web_tt2/header.tt2
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 28 / 56 Sympa in production Problems Synchronisation problem
lots of lists at the same time LDAP/AD server error
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 29 / 56 Sympa in production Problems Synchronisation problem
ttl 3 hours random 0-59 minutes
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 30 / 56 Sympa in production Problems Caching problem
stale data owners, moderators permissions
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 31 / 56 Integration Integration
Authentication Data Sources
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 32 / 56 Integration Authentication Authentication
LDAP CAS SSO TLS Shibbolet
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 33 / 56 Integration Data Sources Data Sources
LDAP SQL Sympa List File
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 34 / 56 Integration Data Sources Data Sources
Subscribers Moderators Owners
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 35 / 56 Integration Data Sources Data Sources
Update in regular intervals (ttl, distribution_ttl) Marker for included records User can unsubscribe
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 36 / 56 Integration Data Sources Data Sources
include_file /home/sympa/subscribers.txt
[email protected] Test Tester [email protected] Foo Bar [email protected]
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 37 / 56 Features Features
Personalization Protection Hierarchy Scenarios Plugins
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 38 / 56 Features Personalization Enable personalization
Web interface: Sending/receiving setup List config: merge_feature on
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 39 / 56 Features Personalization Example
Welcome to our great workshop, [% user.gecos %]!
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 40 / 56 Features Personalization Example
Welcome to our great workshop, Stefan Hornburg!
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 41 / 56 Features Personalization Add custom attributes
custom_attribute id level name Your game level type integer
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 42 / 56 Features Personalization Add custom attributes
Welcome to our great workshop, [% user.gecos %]
Your game level is [% user.custom_attribute.level.value %].
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 43 / 56 Features Personalization Add custom attributes
Welcome to our great workshop, Stefan Hornburg!
Your game level is 25.
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 44 / 56 Features Protection Protection
DKIM DMARC / ARC S/MIME
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 45 / 56 Features Hierarchy Hierarchy
Server Virtual hosts (Robots) List
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 46 / 56 Features Plugins Plugins
Template plugins Modlist/Whitelist plugin Distribution notification plugin message_hook post_archive DistributionNotification
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 47 / 56 Challenges Challenges
Reverse DNS SPF/DKIM Bounces Address verification Log parsing (Sisimai) Blocking
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 48 / 56 Challenges Vetting email addresses Vetting email addresses
my $validator = Email::Valid->new( -mxcheck => 1, );
unless ($validator->address($email)) { warn "Email $email invalid: " . $validator->details . "\n"; }
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 49 / 56 Future Projects Projects
REST / GraphQL API Container Ansible Archive search
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 50 / 56 Future API API
SOAP API REST API GraphQL API DBIx::Class schema
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 51 / 56 Future Container Container
Sympa Database (PostgreSQL, MySQL, ...) MTA / IMAP / getmail Web frontend
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 52 / 56 Future Ansible Ansible
Provisioning Testing
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 53 / 56 Future Ansible Archive Search
setup Elasticsearch and index structure write indexer that parses the emails in the archive setup very simple website for search authentication for website is list membership
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 54 / 56 Finish Questions Questions
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 55 / 56 Finish The end The end
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 56 / 56