Security and Network Startup Resource Center Partially adopted from materials by These materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International license (http://creativecommons.org/licenses/by-nc/4.0/) Last updated 4 March 2018 Problem Statement: Security & ScienceDMZs Very large! Host-based security Network security Data Security How much ”security” do we need? Host-based security Overview – perfSONAR is built with security in mind – A few missing pieces, but included in docs – What about data, is it practical? Network Security Overview – DTNs as part of a ScienceDMZ – ScienceDMZ design includes: • Strong host-based security • Minimal host-based services • Use of ACLs not firewalls (in the network) • NIDs (Network Intrusion Detection systems) Securing data on DTNs: ACLs Aggressive access lists (network) – More useful with project-specific DTNs – Exchanging data with a small set of remote collaborators = ACL is fairly easy to manage – Large-scale data distribution servers = difficult/time consuming to handle (but then, the firewall ruleset for such a service would be, too) Limitation of the application set (host) – Makes it easier to protect – Keep unnecessary applications off the DTN (and watch for them anyway using a host IDS – take violations seriously) Securing data on DTNs: Network IDS Intrusion Detection Systems (IDS) – One example is Bro – https://www.bro.org/ – Bro is high-performance and battle-tested • Bro protects several high-performance national assets • Bro can be scaled with clustering: https://www.bro.org/sphinx/cluster/index.html • Other IDS solutions are available also Securing data on DTNs: Host IDS Using a Host IDS is recommended for hosts in a Science DMZ Several open source solutions have been recommended: • OSSec: https://ossec.github.io/ • Rkhunter: http://rkhunter.sourceforge.net/ • chkrootkit: http://www.chkrootkit.org/ • Logcheck: http://logcheck.org/ • Fail2ban: https://www.fail2ban.org/ • denyhosts: http://denyhosts.sourceforge.net/ ScienceDMZ ScienceDMZ Do we really need to go through all this trouble? “Big Data” vs. BIG Data One Biology experiment SnapChat Data produced at one Beamline by a team per day worldwide in 2016 of nine scientists: by millions of people = 38 TB = 114 TB (Photosystem II X-Ray Study) http://www.nature.com/articles/ncomms5371 10 – ESnet Science Engagement ([email protected]) - 3/4/18 Moving the BIG data w/ Firewall Inbound data transfer is filtered nuttcp -r -T 10 -i 1 -p 10200 bwctl.newy.net.internet2.edu 4.5625 MB / 1.00 sec = 38.1995 Mbps 13 retrans 4.8750 MB / 1.00 sec = 40.8956 Mbps 4 retrans 4.8750 MB / 1.00 sec = 40.8954 Mbps 6 retrans 6.4375 MB / 1.00 sec = 54.0024 Mbps 9 retrans 5.7500 MB / 1.00 sec = 48.2310 Mbps 8 retrans 5.8750 MB / 1.00 sec = 49.2880 Mbps 5 retrans 6.3125 MB / 1.00 sec = 52.9006 Mbps 3 retrans 5.3125 MB / 1.00 sec = 44.5653 Mbps 7 retrans 4.3125 MB / 1.00 sec = 36.2108 Mbps 7 retrans 5.1875 MB / 1.00 sec = 43.5186 Mbps 8 retrans 53.7519 MB / 10.07 sec = 44.7577 Mbps 0 %TX 1 %RX 70 retrans 8.29 msRTT 11 – ESnet Science Engagement ([email protected]) - 3/4/18 Moving the BIG data w/o Firewall Outbound data transfer not filtered nuttcp -T 10 -i 1 -p 10200 bwctl.newy.net.internet2.edu 92.3750 MB / 1.00 sec = 774.3069 Mbps 0 retrans 111.8750 MB / 1.00 sec = 938.2879 Mbps 0 retrans 111.8750 MB / 1.00 sec = 938.3019 Mbps 0 retrans 111.7500 MB / 1.00 sec = 938.1606 Mbps 0 retrans 111.8750 MB / 1.00 sec = 938.3198 Mbps 0 retrans 111.8750 MB / 1.00 sec = 938.2653 Mbps 0 retrans 111.8750 MB / 1.00 sec = 938.1931 Mbps 0 retrans 111.9375 MB / 1.00 sec = 938.4808 Mbps 0 retrans 111.6875 MB / 1.00 sec = 937.6941 Mbps 0 retrans 111.8750 MB / 1.00 sec = 938.3610 Mbps 0 retrans 1107.9867 MB / 10.13 sec = 917.2914 Mbps 13 %TX 11 %RX 0 retrans 8.38 msRTT Approximately 20-25 times performance improvement 12 – ESnet Science Engagement ([email protected]) - 3/4/18 On the open Internet • Public IPs • No firewall • Expectations of well-maintained hosts Scary! On the open Internet: Scary A few statistics… In 35 hours on our network on one host*: – 19,000+ attempted ssh logins – 350 different invalid user accounts tried – 17,500+ attempts against root account – 4 IP addresses accounted for most of the attempts – This with fail2ban throttling attempts over 1,920 times! *host = 134.164.48.209; dates = Feb 4 at 17:35 to Feb 5 at 04:35 On the open Internet… …What do you think might be the number one thing we can do or, perhaps, the most critical thing we can do to protect our hosts? What else? Host-based security strategies The usual suspects: – Authentication and access • Minimal access to minimal surface (users / services) – Minimal services – Minimal ports – Bind to localhost as appropriate – Auto updates – Consider host-based IDS – Log and log watching – Host-based firewall if needed • perfSONAR needs it and has it What’s running on your host? • firewall-cmd • netstat • systemctl • lsof • ps • top • Others? And, on your network? • Network Intrusion Detection systems (Bro) • Network flows (if possible without causing router slowdowns • Network monitoring tools – LibreNMS, flow tools (NfSen, etc.), check_mk, your favorite tool(s) • snmp properly configured and used What about ssh, passwords & keys? Configuring and using ssh incorrectly will guarantee a security compromise… The wrong way: – Using simple passwords for users – Allowing root to login with a password – In reality – allowing any login with a password The right way: – Disable all password access – Disable root access with password – Some disable root access completely Using ssh our way, so far For class we have compromised. Our way: – Allow user login with improved passwords – Allow root login with improved passwords Understanding password strength, see next slide…* *https://xkcd.com/936/ No passwords are better Proving identities with ssh keys CLIENT SERVER User's private key (+public) ~/.ssh/id_rsa User's public key ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys Server's private key (+public) Server's public key /etc/ssh/ssh_host_rsa_key ~/.ssh/known_hosts /etc/ssh_host_rsa_key.pub Improved password for workshop Method 1 (moderately strong, but not necessary) – 8 characters or more – Not a word in any language – A mix of numbers, upper and lower case – Include some punctuation characters Method 2 (stronger) – Use four words of 6 characters, or more – Use unrelated words Examples (do not use these!) 1. Tr0ub4dor&3 2. CorrectHorseBatteryStaple New NIST password guidelines https://www.passwordping.com/surprising-new- password-guidelines-nist/ • Some are surprised… • It’s about time! New US Government guidelines: 1. Remove periodic password change requirements 2. Drop the algorithmic complexity song and dance 3. Require screening of new passwords against lists of commonly used or compromised passwords ssh public and private keys Goal state: – Create a general user with sudo privileges – No password access allowed to hosts – Generate or use a private key and place your public key on your hosts. – Configure ssh on your hosts to only allow access via keys (no passwords) – Either turn off all access to root via ssh, or only allow root access via public / private ssh keys. Remember our earlier slide… “17,500+ ssh login attempts against the root account in 35 hours” References • Sample netstat usage: – https://www.thegeekstuff.com/2010/03/netstat-command- examples/ • perfSONAR firewall setup: – http://docs.perfsonar.net/manage_security.html • Using firewalld on CentOS: – https://linode.com/docs/security/firewalls/introduction-to- firewalld-on-centos/ • Firewalld documentation: – http://www.firewalld.org/documentation/ • Esnet ScienceDMZ overview: – https://fasterdata.es.net/science-dmz/science-dmz-security/ Questions?.