Extensive Endpoint Service Availability Testing (EESAT) 2002 1 Background Cellular network quality in Malaysia is measured with EESAT with the purpose of benchmarking the Quality of Service (QoS) for cellular networks. The test was conducted in the Klang Valley from 30 August 2002 through 1 October 2002, involving five cellular networks: a. Celcom (019) b. DiGi (016) c. Maxis (012) d. TimeCel (017) e. TMTouch (013) EESAT could provide consumers with a common platform to gauge cellular networks performance. This is part of the Malaysian Communications and Multimedia Commission’s ongoing effort to keep consumers better informed. Furthermore, this would encourage cellular network operators to improve their networks which could lead to a better level of service quality. 2 Methodology Measurements were conducted with benchmarking tools that could generate calls automatically and uniformly for every cellular network. EESAT was conducted during weekdays between 8:00am and 7:00pm. In principle, EESAT involved sequence of test calls, defined in two methods of testing: a. Drive test b. Static test The drive test involves sequences of test calls made in a vehicle along selected routes which covers main roads, public access and hot spot areas. There are nine clusters identified for the drive test in the Klang Valley: Cluster Name Area Metro KL Kuala Lumpur City Centre including Bangsar, Damansara Heights and Pudu North KL Gombak, Selayang, Sentul, Segambut, Kepong, Batu Caves, Sg. Buloh and Rawang East KL Keramat, Setapak, Ampang, Hulu Kelang and 2 Pandan South KL Cheras, Sungai Besi, Kajang, Bandar Baru Bangi, Serdang and Seri Kembangan North PJ Petaling Jaya (North of Federal Highway), Subang and Damansara South PJ Petaling Jaya (South of Federal Highway), Subang Jaya, USJ, Puchong and Kinrara Shah Alam Shah Alam area from Padang Jawa to Shah Alam Stadium and NKVE to KESAS Highway Klang Klang Town, Port Klang, Westport, Meru, Pandamaran and Kapar MSC Putrajaya, Cyberjaya, KLIA and Nilai Town Drive test is performed to assess network performance from a moving subscribers’ point of view. On average, 200 test calls were made for each cellular network for every cluster. All test calls were intra-network calls to its own test number. Static test involves sequences of test calls made in a stationary position at selected locations which comprising public access, business and commercial areas. Ten locations were identified for static test in the Klang Valley: Locations Central Market Kuala Lumpur City Center (KLCC) The Store, Klang Town MidValley Megamall Kuala Lumpur International Airport (KLIA) USJ Taipan, Subang Jaya Bangunan MPPJ, Petaling Jaya Dataran Putra, Putrajaya The Street Mall, Cyberjaya Kompleks PKNS, Shah Alam Static measurement test is intended to investigate and observe the network reliability and availability in terms of a subscriber’s perception at the selected public access area. On average, 200 calls were made for each cellular network for each location. The calls were segregated to 40 test calls to own test number 3 and 40 test calls to each of the other cellular network test numbers. Thus, the static test would also involve inter-network call handling. 3 EESAT Results It is important to note that results of both the drive and static tests delineate cellular networks performance based on the selected routes and locations at the specific period of time only. 3.1 Service Coverage Performance Service coverage performance is defined as signal strength received by subscribers’ mobile phones while they are having conversation or are in a dedicated mode. Technically, coverage performance is measured with the parameter Received Signal Level (RxLev) with units in dBm. The availability of service coverage for drive test is defined that a network must achieve a minimum of -98dBm of signal strength, and -82dBm of signal strength for static test. Performance is based on the percentage of calls that have the minimum signal strength required. The performances of all networks for drive test are as shown below: RxLev > -98dBm - Klang Valley 100.00 99.95 99.90 99.82 99.77 99.80 99.70 99.70 99.60 99.50 99.40 99.32 Percentage (%) Percentage 99.30 99.20 99.10 99.00 MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 4 Drive Test Celcom Digi Maxis TimeCel TMTouch Signal strength minimum 99.82% 99.77% 99.95% 99.32% 99.70% of -98dBm The performances of all networks for static test are as shown below: RxLev > -82dBm - Klang Valley All 120.00 99.99 100.00 100.00 99.99 100.00 100.00 80.00 60.00 40.00 Percentage (%) 20.00 0.00 M AXIS_012 TM TOUCH_013 DIGI_016 TIM ECEL_017 CELCOM _019 Static Test Celcom Digi Maxis TimeCel TMTouch Signal strength minimum 100% 100% 99.99% 99.99% 100% of -82dBm 3.2 Signal Quality Performance Signal quality performance is defined as quality of voice transmission while subscribers are using their mobile phones. Excellent signal quality performance would guarantee no missing words or “ping pong” sound. Technically, this parameter is a measurement of Bit Error Rate (BER) of the signal received by subscribers. Bit Error Rate is then converted to a value ranging from 0 to 7. The parameter used is Received Signal Quality (RxQual). The result would give information of the radio quality performance as below: 5 RxQual Radio Quality Categorization 0-3 Excellent 4-5 Good 6-7 Poor Performance is based on the percentage of calls that have the minimum signal quality of 5. Table below shows the performances of all networks for drive test: RxQual <=5 - Klang V alle y 99.50 99.04 98.90 99.00 98.50 98.31 98.22 98.00 97.74 Percentage (%) Percentage 97.50 97.00 MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 Drive Test Celcom Digi Maxis TimeCel TMTouch Signal quality (RxQual) = 98.31% 98.22% 97.74% 98.90% 99.04% 5 and above 6 The table below shows the performances of all networks for static test: RxQual <=5 - Klang Valley All 99.63 99.65 99.62 99.60 99.58 99.54 99.55 99.50 99.50 Percentage (%) 99.45 99.40 M AXIS_012 TM TOUCH_013 DIGI_016 TIM ECEL_017 CELCOM _019 Static Test Celcom Digi Maxis TimeCel TMTouch Signal quality (RxQual) = 99.54% 99.50% 99.62% 99.58% 99.63% 5 and above 3.3 Speech Quality Speech quality measurement in EESAT is conducted to emulate speech quality as perceived by the human ear. The parameter used in this test is the Mean Opinion Score (MOS). MOS is a subjective speech quality measurement, which usually involves convening listening panel to assess speech quality, as recommended in ITU-T Recommendation P.800. However, in this EESAT exercise, MOS is obtained through the objective speech measurement method called Perceptual Speech Quality Measure (PSQM), based on ITU-T Recommendation P.861. The scale is defined from 1 to 5. Definition of MOS scale would be as below: 7 MOS scale Speech quality 1 Bad 2 Poor 3 Fair 4 Good 5 Excellent In this EESAT, MOS is measured in the drive test only. Performance is defined on the percentage of calls with the MOS value of 3 and above. Below are results of speech quality performance for all cellular networks: MOS >= 3 - Klang Valley 99.00 98.56 98.45 98.50 98.00 97.50 96.91 96.86 97.00 96.73 Percentage (%) Percentage 96.50 96.00 95.50 MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 Drive Test Celcom Digi Maxis TimeCel TMTouch MOS = 3 and above 96.86% 96.73% 98.45% 96.91% 98.56% 3.4 Dropped Call Rate Call drop is defined as a call that is disconnected prematurely during the 60- second holding period. A call drop would possibly be caused by weak signal strength, interference, congestion and other possible reasons. Table below 8 indicates percentage of calls that was disconnected prematurely during the drive test for each cellular network: Dropped Call rate - Klang Valley 1.80 1.60 1.60 1.40 1.20 1.00 1.00 0.80 0.60 0.48 0.52 Percentage (%) Percentage 0.40 0.21 0.20 0.00 MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 Drive Test Celcom Digi Maxis TimeCel TMTouch Dropped Call Rate 0.21% 0.48% 1.60% 0.52% 1.00% The table below indicates percentage of calls that was disconnected prematurely during the static test for each cellular network: Dropped Call Rate - Klang Valley All 0.18 0.15 0.16 0.14 0.12 0.10 0.08 0.06 0.05 0.05 0.05 Percentage (%) 0.04 0.02 0.00 0.00 M AXIS_012 TM TOUCH_013 DIGI_016 TIM ECEL_017 CELCOM _019 9 Static Test Celcom Digi Maxis TimeCel TMTouch Dropped Call Rate 0.00% 0.05% 0.15% 0.05% 0.05% 3.5 Congestion Rate Congestion is defined as failure to establish connection between caller and receiver after a call attempt has been made even when service coverage is good. This is due to the fact that resources in the network are insufficient to provide services for all subscribers in the area: Table below shows the congestion rate of all cellular networks for drive test: Blocked Call Rate - Klang Valley 25.00 20.00 Percentage (%) 15.00 10.00 5.00 NA 0.38 0.35 0.40 0.27 0.00 MAXIS_012 TMTOUCH_013 DIGI_016 TIMECEL_017 CELCOM_019 Drive Test Celcom Digi Maxis TimeCel TMTouch Congestion Rate 0.27% *NA 0.38% 0.40% 0.35% * DiG’'s result did not indicate its actual performance due to test number problem during the drive test 10 Table below shows the congestion rate of all cellular networks for static test: Block e d Call Rate - Klang V alle y All 0.90 0.80 0.80 0.70 0.60 0.50 0.40 0.30 Percentage (%) 0.20 0.15 0.10 0.00 0.00 0.00 0.00 M AXIS_012 TM TOUCH_013 DIGI_016 TIM ECEL_017 CELCOM _019 Static Test Celcom Digi Maxis TimeCel TMTouch Congestion Rate 0.15% 0.00% 0.00% 0.00% 0.80% 3.3 Call Setup Time Call Setup Time is the total time measured from sending in the called party number to the time the other party is ringing.