Broadband Quality of Service Experience in Bhutan May 2011 Thimphu This work was carried out with the aid of a grant from the International Development Research Centre, Canada and the Department for International Development, UK. What gets measured gets improved. But who can measure BB quality? • The regulator – All networks can be monitored/measured – Can order operators to change performance – May or may not make data available to consumers • The operators – Only their own network • Users/consumers – Only their own ISP – But useful only if comparison possible Different tools for consumers to measure BB quality • Several existing methods – SpeedNet, etc. – Emphasis on download and upload speeds • New method: “AT -tester” software – Methodology and software developed by IIT-Madras – Additional metrics other than throughput – Software freely available at www.bradbandasia.info – Testing results automatically uploaded to same web site; viewable by all What is measured: metrics explained • Upload and download speeds – Measured in kilo (or mega or giga) bits per second. Basic measure of speed • Round Trip Time (RTT) – Time taken for a packet to reach a destination and return • Jitter – Average Variation of RTT • Packet Loss – Number of packets (as %) which do not reach the destination Why different metrics? Throughput Delay Service Down Up RTT Jitter Loss Browse (text) ++ - + -- Browse (media) +++ - + + + Download file +++ - - - - Upload File - +++ --- Transactions + + ++ + + Streaming media +++ - + ++ ++ VOIP + + +++ +++ +++ Games ++ + +++ ++ ++ 3 domains are tested More on methodology… • Multiple packages tested – Leased lines, ADSL, Mobile internet • 3 domains tested – Within ISP, national, international (USA) • At multiple times of the day – 6 times, spread out through the day – Include Peak and off-peak • Each time, multiple readings – Outliers eliminated • Multiple locations in the country – Thimphu, Paro – Inside commercial premises (due to availability) Results from April 2011 1. ADSL “over-delivers” within ISP domain… Download in kbps- from ISP - ADSL packages 2000 1500 download in kbps kbps inin downloaddownload 1000 512 500 256 0 Time 0800 1100 1500 1800 2000 2300 Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Paro …. and in international domain Download in kbps- International -ADSL packages 2000 1500 Dowanload in kbps kbps inin Dowanload Dowanload 1000 512 500 256 0 Time 0800 1100 1500 1800 2000 2300 Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Paro 2. Mobile BB underperforms in ISP domain…. Download in kbps- ISP – 3G 2000 1500 download in kbps kbps inin downloaddownload 1000 500 550 0 Time 0800 1100 1500 1800 2000 2300 Bhutan Telecom 3G (550kbps to 1Mbps) Thimphu …and in international domain Download in kbps- International - 3G 2000 1500 Dowanload in kbps kbps inin Dowanload Dowanload 1000 500 550 0 Time 0800 1100 1500 1800 2000 2300 Bhutan Telecom 3G (550kbps to 1Mbps) Thimphu …and makes current 3G BB less suitable for streaming/VOIP but Jitter when pinged to an International server 400 Jitter (ms) Jitter 300 200 100 0 Time 0800 1100 1500 1800 2000 2300 Druknet Leased line (512 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Paro Bhutan Telecom 3G (550kbps to 1Mbps) Thimphu 3. For all packages, actual performance vs. stated shows huge variations within ISP Domain… Ratio of actual to stated speed – ISP server 1000 750 500 Ratio of actual to stated (%) (%) stated stated to to actualactualofof RatioRatio 250 100% 0 Time 0800 1100 1500 1800 2000 2300 Druknet Leased line (512 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Paro Bhutan Telecom 3G (550kbps to 1Mbps) Thimphu note: 3G speed taken as 550 kbps …. and in international domain Ratio of actual to stated speed - International server 1000 750 to stated (%) stated to actualactual 500 Ratio of Ratio 250 100% 0 Time 0800 1100 1500 1800 2000 2300 Druknet Leased line (512 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Paro Bhutan Telecom 3G (550kbps to 1Mbps) Thimphu note: 3G speed taken as 550 kbps 4. Performance drops going from ISP domain to National domain (another ISP) as seen by download speeds….. Download in kbps- ISP (All packages) Download in kbps- National (All packages) 2000 2000 1500 1500 download inkbps download Download inkbps Download 1000 1000 500 500 0 0 Time Time 0800 1100 1500 1800 2000 2300 0800 1100 1500 1800 2000 2300 Druknet Leased line (512 kbps) Thimphu Druknet Leased line (512 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Paro Druknet (prepaid) ADSL (256 kbps) Paro Bhutan Telecom 3G (550kbps to 1Mbps) Thimphu Bhutan Telecom 3G (550kbps to 1Mbps) Thimphu …and by the increase in packet loss Packet Loss when pinged to an ISP Packet Loss when pinged to a National 50 server 120 server 100 40 80 30 60 20 40 Packet loss (%) (%) lossloss Packet Packet Packet loss (%) loss Packet 10 20 0 0 Time Time 0800 1100 1500 1800 2000 2300 0800 1100 1500 1800 2000 2300 Druknet Leased line (512 kbps) Thimphu Druknet Leased line (512 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Paro Druknet (prepaid) ADSL (256 kbps) Paro 5. Int’l Round Trip Time (RTT) too high compared regulated standards in India, Singapore… RTT when pinged to an International server RTT (ms) RTT 1000 500 (Indian terrestrial) 350 (Singapore) 300 0 Time 0800 1100 1500 1800 2000 2300 Druknet Leased line (512 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Paro Bhutan Telecom 3G (550kbps to 1Mbps) Thimphu 6. Low value for money from many technologies for users most of the time (international domain) Download from International - kbps per USD 200 (kbps) per USD per (kbps) 100 Dowanload Dowanload 0 Time 0800 1100 1500 1800 2000 2300 Druknet Leased line (512 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Thimphu Druknet (prepaid) ADSL (256 kbps) Paro Compared to other countries also, many BT packages give low value for money. But not all Download from International - kbps per Value for Money - Reaching International servers USD 200 200 160 160 120 120 80 80 DD ow ow nload nload speedspeed (kbps)(kbps) perper dollardollar 40 40 0 0 Time DowanloadUSD per (kbps) 0800 1100 1500 1800 2000 2300 800 110 150 180 200 230 Druknet Leased line (512 kbps) Thimphu Tashi InfoComm Leased line (256 kbps) Thimphu BSNL (256 kbps) Bangalore, IN BSNL (1 Mbps) Bangalore, IN Airtel, (256 kbps) Delhi, IN Druknet (prepaid) ADSL (256 kbps) Thimphu Airtel (1 Mbps) Delhi, IN Bell (6 Mbps) Ottawa, CA Rogers (10 Mbps) Ottawa, CA Druknet (prepaid) ADSL (256 kbps) Paro Verizon (3 Mbps) Buffalo, US Comcast (6 Mbps) Denver, US What can Bhutan do? • Consumers – Download AT-Tester from www.bradbandasia.info (already configured for BT) – Run the tests (automatic, no interventions required) – Make purchase decisions based on this • BICMA – Publicize the availability of software and the results (to empower consumer) – Use software as a tool for diagnosis • Operators – Over time develop differentiated products for consumers • All stakeholders need to recognize this is early stages – light touch regulation – more collaboration (vs. command & control) – consumer centric thinking Thank You. [email protected] Test locations Druknet ADSL (256 kbps), Tashi Comm leased line (256 kbps) and Bhutan telecom 3G were tested at: Athang ICTech, Yarkay Plaza, Chang Lam, Thimphu Druknet Leased line (512 kbps) was tested at: JICA Bhutan Office, Doybum Lam/ Memorial Chorten, Thimphu , Bhutan. P .O.Box- 217 Druknet ADSL (256 kbps) was tested at Tsheryoen's Cafe, Paro Town(Opp. Telecom office), Paro , Bhutan Tested Packages coversion Price in USD Price in Nu rate Druknet Leased line (512 kbps) Thimphu 3166.90 141166.67 44.58 Tashi InfoComm Leased line (256 kbps) Thimphu 134.60 6000.00 44.58 Druknet (prepaid) ADSL (256 kbps) Thimphu 8.95 399.00 44.58 Druknet (prepaid) ADSL (256 kbps) Paro 8.95 399.00 44.58 Bhutan Telecom 3G (550kbps to 1Mbps) Thimphu 0.04 1.65 44.58.