Remote Research Basics Quick Reference Guide Remote Research Basics Quick Reference Guide

An introduction to networks and data movement for researchers with little to no technical knowledge. This guide includes simple tests and tools that you can use when working from home.

All content is publicly available under the Creative Commons Attribution License https://creativecommons.org/licenses/by/4.0/

About AARNet

AARNet is a not-for-profit organisation owned by Australian universities and research institutions. We provide high speed network for universities to connect researchers. We also provide cloud storage, known as CloudStor, and other network services.

More information: www.aarnet.edu.au

Networks and connections

NBN vs ADSL The main difference for the consumer between the NBN and ADSL is speed. There are various options for speed, depending on requirements.

This video helps explain: https://www.youtube.com/watch?time_continue=31&v=L3QJLEbUj94

Learn more about what is available via the NBN:

https://www.nbnco.com.au/learn/speed https://www.nbnco.com.au/learn/speed/understanding-speed-and-data

NREN The Australian National Research Network (NREN) operated by AARNet provides advanced research network infrastructure. It is fast, with connections from 10 Gbps to 40 and 100 Gbps.

Learn more: https://www.aarnet.edu.au/network-and-services/advanced-network-services

Don’t forget ! eduroam is a secure global roaming wireless network for the research and education sector. In Australia eduroam is operated by AARNet.

Lean more: https://www.eduroam.edu.au/

aarnet.edu.au 1 Remote Research Basics Quick Reference Guide How fast is fast?

Speed test You can test the speed of your connection by using an app or a website.

Test the speed of your connection: http://www.speedtest.net/

Factors affecting network speeds Take a look at these two websites that address some of the factors that affect network speeds:

Time of day - https://www.accc.gov.au/consumers/internet-landline-services/broadband-speeds (go to section ‘Network limitations that affect speed’) Physical interference - https://www.performancenetworks.co.uk/blog/post/5-factors-can-affect-wifi-signal/

How much data?

This is a comparative scale of bytes. Remember that stored data is measured in bytes (with a capital ‘B’).

Source: http://testyotta.blogspot.com/2015/10/yotta.html

Bits or bytes? https://www.youtube.com/watch?v=zGrSI2jSbVE

aarnet.edu.au 2 Remote Research Basics Quick Reference Guide File transfer time calculation

Time can be saved if the network speeds are tested ahead of trying to transfer data, network traffic levels are known, and file sizes have been calculated.

How do you calculate file transfer times? Use: https://techinternets.com/copy_calc

Here’s a handy table for you to use and share.

Note: Science DMZ data rates: the table shows the bandwidth required to transfer data sizes for allotted time frames. The base data is sourced from ESnet and recalculated.

Ping test Use these instructions: https://www.wikihow.com/Ping-an-IP-Address More information: https://en.wikipedia.org/wiki/Ping_(networking_utility)

Traceroute Use these instructions: https://kb.intermedia.net/article/682 Like the ping test, you are looking for a few different things. This tutorial shows you how to read a traceroute: https://www.inmotionhosting.com/support/website/ssh/read-traceroute/

Data movement - some key concepts

Bandwidth Bandwidth or throughput (bps) is the size of the “pipe” for data to flow in (the successful data transfer rate in bits per second) - https://www.comparitech.com/net-admin/throughput-vs-bandwidth/

Latency Latency is the amount of time (or delay) measured in milliseconds (ms) in sending data between two points.

Speed Speed is a combination of bandwidth and latency (the amount of data that can be received every second)

Remember! Packet loss is the failure of a packet of data to reach its destination. A hop is a path between networking devices (on a network). The hop count is the number of devices (routers) on a network between start/end points.

aarnet.edu.au 3 Remote Research Basics Quick Reference Guide Some helpful tips Speed is constrained by the slowest part of the link, which may be the connection of your laptop to the network, or the campus LAN, or even the firewall at the campus boundary. Latency, distance, and packet loss are factors that can affect speed. The difference between moving lots of smaller files and one large file -- the difference is in the time taken in the file system (on the source system and again on the destination system) to locate each individual file and initiate a move process or a write process. Latency is not really relevant unless you are dependent on a quick response from the distant system (e.g. when Zooming), in which case it is unaffected by the speed of the link, but is determined by the distance and the finite speed of light or current: e.g. 250ms from to New York (ping cuny.edu). TCP at latency has serious trouble peaking over ~500Mbit/s if even the most modest background noise of packet loss is present. See Esnet website: https://fasterdata.es.net/network-tuning/tcp-issues-explained/packet-loss/

Tools

CloudStor CloudStor is AARNet’s file sync, share and storage service. CloudStor hosts a variety of tools that can help you with a range of data movement scenarios:

FileSender Sync and ownCloud clients WebDAV rClone Rocket the data pump tool Collections plugin Cloudstor S3 buckets More information: https://support.aarnet.edu.au/hc/en-us/categories/200217608-CloudStor

FileSender Email is good for sending messages and small files, but is no good for large files.

FileSender is a file sharing service available via CloudStor that has the following benefits:

No restriction on file type. Up to to 100 files max size of 2PB per upload. Provides a URL to your files that you can share with non-CloudStor users. You can send “Vouchers” to non-CloudStor users so they can share files with you. You can put an expiry date on the files that you send. Provides notification emails to you and your recipients about your files and transfers. Provides optional end-to-end password protected encryption (using AES-CBC SHA-256). More information: https://support.aarnet.edu.au/hc/en-us/sections/115000260773-CloudStor-FileSender

aarnet.edu.au 4 Remote Research Basics Quick Reference Guide Sync and OwnCloud There are lots of simple ways you can manage your active data so that it is easy to find and safely stored. You can set up an automatic way to move your data to the cloud so that your data is safe and secure. This is called ‘syncing’. Using the ownCloud Sync Client you can keep your computer and your CloudStor folders up to date automatically. The sync client is available for Windows, Mac, OSX, Linux, iOS and Android. When accessing CloudStor for the first time you will be given the option of downloading an ownCloud sync app for your operating system. Syncing files enables you to create a local ownCloud folder on your PC/mac/Linux/mobile and set up sync rules to copy files to and from AARNet CloudStor. You can also enter your unique sync password here. Follow these instructions to install the client. Download the sync client here: https://cloudstor.aarnet.edu.au/client-download/ Install and open the ownCloud Desktop Sync Client. You will be prompted for the server address. Enter https://cloudstor.aarnet.edu.au/plus and click > “Next” Enter your institutional email address as your username and the sync password you created (not the password for your institutional login) and then click > “Next”. Ensure “Sync Everything” is selected and then click “Connect…” and CloudStor will start synchronising your data. Click on Cloudstor icon to expand the folders you have access to from the web client. To access a group folder, click on Shared folder to expand. Click on the folder you need to access.

WebDAV (Web Distributed Authoring and Versioning) WebDAV is an extension to HTTP, the protocol that web-browsers and webservers use to communicate with each other. The WebDAV protocol enables a webserver to behave like a fileserver too, supporting collaborative authoring of web content.

WebDAV extends the set of standard HTTP methods and headers to provide the ability to create a file or folder, edit a file in place, copy or move or delete a file, etc.

More information: https://support.aarnet.edu.au/hc/en-us/articles/115007168507-Can-I-use-the-command-line-or-WebDav- https://support.aarnet.edu.au/hc/en-us/articles/360000741156-Accessing-public-links-over-WebDAV rClone Rclone is an open source multi-platform project which enables upload, download and synchronisation.

More information: https://support.aarnet.edu.au/hc/en-us/articles/115007168507-Can-I-use-the-command-line-or-WebDav-

Rocket CloudStor Rocket is a fast parallel upload tool to help researchers upload datasets from scientific instrumentation to CloudStor.

Rocket has the following benefits: Standalone Windows application Upload only, different to the CloudStor sync client that mirrors data Allows tuning/tweaking to help upload your data as quickly as possible. More information: https://support.aarnet.edu.au/hc/en-us/sections/115000264294-CloudStor-Rocket

Collections Collections is a data packaging tool. It makes it easy to collate, describe, package and share groups of files and associated metadata.

With this tool you an create collections of CloudStor-based files, through mandatory and optional metadata elements. Each collection is packaged into an archive, complete with copies of the files, manifest and metadata. You can then export metadata human and machine- readable formats.

More information: https://support.aarnet.edu.au/hc/en-us/sections/115000264274-CloudStor-Collections

S3 gateways aarnet.edu.au 5 Remote Research Basics Quick Reference Guide S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services (AWS) that provides object storage through a web service interface.

The S3-compatible API connectivity option for CloudStor provides an S3-compliant interface for researchers to use with their S3- compatible storage applications, gateways, and other platforms.

More information: https://www.aarnet.edu.au/network-and-services/connectivity-services/amazon-web-services/

SWAN SWAN - the Service for Web-based Analysis - was developed by CERN, the European Organization for Nuclear Research, one of the world’s largest and most respected centres for scientific research and an AARNet partner in the development of CloudStor. SWAN provides Jupyter Notebooks to allow researchers to perform interactive data analysis in CloudStor using a web browser and without the need to install any software. Jupyter Notebooks can be launched through SWAN to process research data using cloud computing services operated by AARNet.

More information: https://news.aarnet.edu.au/data-analysis-in-cloudstor-swan-is-ready-for-researchers/

What is CloudStor SWAN? https://support.aarnet.edu.au/hc/en-us/articles/360000575395-What-is-CloudStor-SWAN-

Jupyter Notebooks Jupyter Notebooks is a free, open-source, interactive web tool which researchers use so they can combine software code, computational output, explanatory text and multimedia resources in a single document.

More information: https://jupyter.org/

Resources:

Types of networks:

https://en.wikipedia.org/wiki/Internet_access https://en.wikipedia.org/wiki/Internet_in_Australia https://en.wikipedia.org/wiki/Asymmetric_digital_subscriber_line https://en.wikipedia.org/wiki/G.992.5 https://en.wikipedia.org/wiki/National_research_and_education_network

More about the NBN:

https://www.nbnco.com.au/content/dam/nbnco2/documents/nbn-business-fact-sheets/nbn-business-fact-sheet-tc4.pdf https://www.accc.gov.au/consumers/internet-landline-services/broadband-speeds https://www.whistleout.com.au/Broadband/Guides/what-nbn-speed-do-you-need https://www.nbnco.com.au/residential/learn/speed https://en.wikipedia.org/wiki/National_Broadband_Network

aarnet.edu.au 6 Remote Research Basics Quick Reference Guide Notes and questions from panellists and participants

From Paul: The speed test result you get might not be representative of your modem/ to your ISP/internet, but can be affected by your device connection to your router/modem!

My home router shows two wireless networks (home-net, home-net5) - these are two different wireless frequencies (2.4Ghz & 5Ghz respectively). Some home wi-fi systems automatically only advertise one network and put you on the best, but some (like mine) you have to join manually.

For reference, 5Ghz is a newer, faster radio, but has less range/coverage.

I have a 100M download plan, and find I get ~50Mbps on my 2.4Ghz, but can get up to 95Mbps on my 5Ghz wifi or on a wired connection to the router (but less range!)

From Carina: Have a big data movement challenge you haven’t been able to solve? Contact us: [email protected]

From Ben: Latency very relevant to real-time music but there are ways to approach it, more info here if anyone is interested - ‘performing music on the internet’

https://le.unimelb.edu.au/video-and-media/additional-media-production-services/performing-music-over-the-internet

aarnet.edu.au 7