Remote Research Basics Quick Reference Guide Remote Research Basics Quick Reference Guide An introduction to networks and data movement for researchers with little to no technical knowledge. This guide includes simple tests and tools that you can use when working from home. All content is publicly available under the Creative Commons Attribution License https://creativecommons.org/licenses/by/4.0/ About AARNet AARNet is a not-for-profit organisation owned by Australian universities and research institutions. We provide high speed network for universities to connect researchers. We also provide cloud storage, known as CloudStor, and other network services. More information: www.aarnet.edu.au Networks and connections NBN vs ADSL The main difference for the consumer between the NBN and ADSL is speed. There are various options for speed, depending on requirements. This video helps explain: https://www.youtube.com/watch?time_continue=31&v=L3QJLEbUj94 Learn more about what is available via the NBN: https://www.nbnco.com.au/learn/speed https://www.nbnco.com.au/learn/speed/understanding-speed-and-data NREN The Australian National Research Network (NREN) operated by AARNet provides advanced research network infrastructure. It is fast, with connections from 10 Gbps to 40 and 100 Gbps. Learn more: https://www.aarnet.edu.au/network-and-services/advanced-network-services Don’t forget eduroam! eduroam is a secure global roaming wireless network for the research and education sector. In Australia eduroam is operated by AARNet. Lean more: https://www.eduroam.edu.au/ aarnet.edu.au 1 Remote Research Basics Quick Reference Guide How fast is fast? Speed test You can test the speed of your connection by using an app or a website. Test the speed of your connection: http://www.speedtest.net/ Factors affecting network speeds Take a look at these two websites that address some of the factors that affect network speeds: Time of day - https://www.accc.gov.au/consumers/internet-landline-services/broadband-speeds (go to section ‘Network limitations that affect speed’) Physical interference - https://www.performancenetworks.co.uk/blog/post/5-factors-can-affect-wifi-signal/ How much data? This is a comparative scale of bytes. Remember that stored data is measured in bytes (with a capital ‘B’). Source: http://testyotta.blogspot.com/2015/10/yotta.html Bits or bytes? https://www.youtube.com/watch?v=zGrSI2jSbVE aarnet.edu.au 2 Remote Research Basics Quick Reference Guide File transfer time calculation Time can be saved if the network speeds are tested ahead of trying to transfer data, network traffic levels are known, and file sizes have been calculated. How do you calculate file transfer times? Use: https://techinternets.com/copy_calc Here’s a handy table for you to use and share. Note: Science DMZ data rates: the table shows the bandwidth required to transfer data sizes for allotted time frames. The base data is sourced from ESnet and recalculated. Ping test Use these instructions: https://www.wikihow.com/Ping-an-IP-Address More information: https://en.wikipedia.org/wiki/Ping_(networking_utility) Traceroute Use these instructions: https://kb.intermedia.net/article/682 Like the ping test, you are looking for a few different things. This tutorial shows you how to read a traceroute: https://www.inmotionhosting.com/support/website/ssh/read-traceroute/ Data movement - some key concepts Bandwidth Bandwidth or throughput (bps) is the size of the “pipe” for data to flow in (the successful data transfer rate in bits per second) - https://www.comparitech.com/net-admin/throughput-vs-bandwidth/ Latency Latency is the amount of time (or delay) measured in milliseconds (ms) in sending data between two points. Speed Speed is a combination of bandwidth and latency (the amount of data that can be received every second) Remember! Packet loss is the failure of a packet of data to reach its destination. A hop is a path between networking devices (on a network). The hop count is the number of devices (routers) on a network between start/end points. aarnet.edu.au 3 Remote Research Basics Quick Reference Guide Some helpful tips Speed is constrained by the slowest part of the link, which may be the connection of your laptop to the network, or the campus LAN, or even the firewall at the campus boundary. Latency, distance, and packet loss are factors that can affect speed. The difference between moving lots of smaller files and one large file -- the difference is in the time taken in the file system (on the source system and again on the destination system) to locate each individual file and initiate a move process or a write process. Latency is not really relevant unless you are dependent on a quick response from the distant system (e.g. when Zooming), in which case it is unaffected by the speed of the link, but is determined by the distance and the finite speed of light or current: e.g. 250ms from Perth to New York (ping cuny.edu). TCP at latency has serious trouble peaking over ~500Mbit/s if even the most modest background noise of packet loss is present. See Esnet website: https://fasterdata.es.net/network-tuning/tcp-issues-explained/packet-loss/ Tools CloudStor CloudStor is AARNet’s file sync, share and storage service. CloudStor hosts a variety of tools that can help you with a range of data movement scenarios: FileSender Sync and ownCloud clients WebDAV rClone Rocket the data pump tool Collections plugin Cloudstor S3 buckets More information: https://support.aarnet.edu.au/hc/en-us/categories/200217608-CloudStor FileSender Email is good for sending messages and small files, but is no good for large files. FileSender is a file sharing service available via CloudStor that has the following benefits: No restriction on file type. Up to to 100 files max size of 2PB per upload. Provides a URL to your files that you can share with non-CloudStor users. You can send “Vouchers” to non-CloudStor users so they can share files with you. You can put an expiry date on the files that you send. Provides notification emails to you and your recipients about your files and transfers. Provides optional end-to-end password protected encryption (using AES-CBC SHA-256). More information: https://support.aarnet.edu.au/hc/en-us/sections/115000260773-CloudStor-FileSender aarnet.edu.au 4 Remote Research Basics Quick Reference Guide Sync and OwnCloud There are lots of simple ways you can manage your active data so that it is easy to find and safely stored. You can set up an automatic way to move your data to the cloud so that your data is safe and secure. This is called ‘syncing’. Using the ownCloud Sync Client you can keep your computer and your CloudStor folders up to date automatically. The sync client is available for Windows, Mac, OSX, Linux, iOS and Android. When accessing CloudStor for the first time you will be given the option of downloading an ownCloud sync app for your operating system. Syncing files enables you to create a local ownCloud folder on your PC/mac/Linux/mobile and set up sync rules to copy files to and from AARNet CloudStor. You can also enter your unique sync password here. Follow these instructions to install the client. Download the sync client here: https://cloudstor.aarnet.edu.au/client-download/ Install and open the ownCloud Desktop Sync Client. You will be prompted for the server address. Enter https://cloudstor.aarnet.edu.au/plus and click > “Next” Enter your institutional email address as your username and the sync password you created (not the password for your institutional login) and then click > “Next”. Ensure “Sync Everything” is selected and then click “Connect…” and CloudStor will start synchronising your data. Click on Cloudstor icon to expand the folders you have access to from the web client. To access a group folder, click on Shared folder to expand. Click on the folder you need to access. WebDAV (Web Distributed Authoring and Versioning) WebDAV is an extension to HTTP, the protocol that web-browsers and webservers use to communicate with each other. The WebDAV protocol enables a webserver to behave like a fileserver too, supporting collaborative authoring of web content. WebDAV extends the set of standard HTTP methods and headers to provide the ability to create a file or folder, edit a file in place, copy or move or delete a file, etc. More information: https://support.aarnet.edu.au/hc/en-us/articles/115007168507-Can-I-use-the-command-line-or-WebDav- https://support.aarnet.edu.au/hc/en-us/articles/360000741156-Accessing-public-links-over-WebDAV rClone Rclone is an open source multi-platform project which enables upload, download and synchronisation. More information: https://support.aarnet.edu.au/hc/en-us/articles/115007168507-Can-I-use-the-command-line-or-WebDav- Rocket CloudStor Rocket is a fast parallel upload tool to help researchers upload datasets from scientific instrumentation to CloudStor. Rocket has the following benefits: Standalone Windows application Upload only, different to the CloudStor sync client that mirrors data Allows tuning/tweaking to help upload your data as quickly as possible. More information: https://support.aarnet.edu.au/hc/en-us/sections/115000264294-CloudStor-Rocket Collections Collections is a data packaging tool. It makes it easy to collate, describe, package and share groups of files and associated metadata. With this tool you an create collections of CloudStor-based files, through mandatory and optional metadata elements. Each collection is packaged into an archive, complete with copies of the files, manifest and metadata. You can then export metadata human and machine- readable formats. More information: https://support.aarnet.edu.au/hc/en-us/sections/115000264274-CloudStor-Collections S3 gateways aarnet.edu.au 5 Remote Research Basics Quick Reference Guide Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services (AWS) that provides object storage through a web service interface.