Pre-Qualifying a Wireless Campus Network Using Ixveriwave

CASE STUDY

Los Angeles County Museum of Art: Pre-qualifying a Wireless Campus Network Using IxVeriWave

Executive Summary Company The Los Angeles County Museum of Art (LACMA) was planning to deploy a campus- With more than 100,000 wide wireless network to support its mobile computing and communications initiatives works of art on display including multi-media tours developed for devices such as smart phones and PDAs. and close to one million The new wireless LAN (WLAN) would provide museum visitors with access to visitors per year, LACMA expanded content about art works on display, exhibitions, and making their way through is the largest encyclopedic the museum. The system being designed was to overlay and begin restructuring the museum west of Chicago. museum’s existing corporate network to also support the mobility of its workforce both Its far-reaching collections on and off campus. reflect the many cultural communities and heritages Before deploying the new network, LACMA decided to compare the performance of two of Southern California. competing WLAN solutions to enable an informed purchasing decision. LACMA used the IxVeriWave solution to conduct a comprehensive test that recreated the network and its users. Results helped LACMA choose the best solution for its needs and to optimize its network design and usage models to the actual network capabilities. Key Issues Through the use of IxVeriWave testing, LACMA was able to turn up its large-scale Selecting the best access corporate wireless network with complete confidence in its future performance. point (AP) for use in a WiFi network that would serve hundreds of museum Background visitors as well as museum employees. LACMA was established in 1910 as part of the Los Angeles Museum of History, Science and Art. It became an independent entity in 1961, moving into its current building complex in 1965. Results

With more than 100,000 works of art on display and close to one million visitors per The IxVeriWave solution year, LACMA is the largest encyclopedic museum west of Chicago. Its far-reaching equipped LACMA and collections reflect the many cultural communities and heritages of Southern California. its solution partners to evaluate various aspects In 2004, LACMA embarked on a multi-year “Transformation Project” for its campus with the goal of transforming the museum inside and out. Goals included creating new of AP performance and dynamic, light-filled spaces for viewing art, new ways to enjoy the surrounding park, network design and make and to navigate the treasures of LACMA’s encyclopedic collections. informed decisions to optimize performance. A key component of this plan was to introduce 21st Century concepts of experiencing art by enhancing the interaction between viewers and exhibits. This innovative approach would allow museum visitors to view art exhibits while carrying WiFi-enabled “Guest PDAs” pre-loaded with multimedia tours featuring various works of art that could be referenced and played within the galleries. This new in-gallery application was designed to extend the museum experience by saving the visitor’s preferences or favorites to LACMA’s web site, where they could be accessed via individual account and presented in a virtual space with additional links to expanded content.

26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012 CASE STUDY

The Challenge • Using RADIUS and Active Directory-based authentication for access to LACMA’s corporate How would LACMA know that the new wireless network network could support hundreds of planned users per gallery while • Providing PEAP MS-CHAP v2 secure access to delivering the desired Quality of Experience (QoE) to all? admissions ticket scanners, employee laptops, and guest PDAs How could the museum be confident of its choice in a vendor to equip the wireless network? Testing Goals

LACMA’s CIO faced the challenge of building a WiFi Objectives of the pre-deployment testing conducted by network that would serve hundreds of museum visitors LACMA, UpTime and Network Test included: as well as employees utilizing up to 300 access points (APs). The vision of enhanced visitor interaction would • Comparing vendor solutions in deciding which be supplemented by an open, mobile, no-walls work equipment to use environment enabling employees, docents, and volunteers to work efficiently virtually anywhere on campus. • Establishing network design and operating regimes that could be consistently and reliably supported by the LACMA realized that basing vendor selection solely upon chosen vendor competing vendor claims would not suffice. In conjunction with its system integrator, UpTime Information Technology, • Comparing stated design goals to actual capabilities of LACMA contracted Network Test, an independent the infrastructure benchmarking and network design consultancy to conduct a comparative test between two competing vendor systems • Testing network performance with representative to help in selecting the best solution for the museum’s traffic loads and at scale network needs.

In an innovative move, LACMA, UpTime, and Network Test To achieve these goals, four sets of tests were defined: decided to create a test that would subject a real network to the actual traffic loads and mixes that were expected upon deployment. The only way to conduct such a test was 1. WiMix ™ Load Test blending all planned data and voice to use the IxVeriWave solution to verify the simultaneous applications originating from 25 independent clients support, co-existence, and service differentiation of all communicating with each AP. required applications on the LACMA network:

• WiFi-enabled ticket scanners for employees that 2. Client Capacity Test using the WiMix load with various automate the admissions process to the exhibit halls numbers of clients to determine the maximum number of concurrent clients supported by each AP. • WiFi-enabled laptops, VoIP-enabled “Thin Client” PCs, and museum-provided mobile devices providing employees access to the corporate network 3. Guest PDA Goodput Test using the IxVeriWave TCP Goodput load test utilizing only one application type • WiFi-enabled mobile VoIP handsets used by employees (the “guest PDA”) and creating high-packet rates from independent clients communicating with • WiFi-enabled laptops brought in by museum visitors and used in conjunction with wireless hotspots to each AP. access the Internet 4. Employee Web Traffic Goodput Test using the • Active RFID tags that would support location-based IxVeriWave TCP Goodput load test utilizing only tracking for the museum’s collections one application type (the employee Web traffic) and creating high bandwidth consumption from In addition, the test was designed to verify the security requirements from the network: independent clients communicating with each AP.

• Segregating guest PDAs, smart phones, and laptops from access to the corporate network

26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012 CASE STUDY

WiMix

The IxVeriWave WiMix test suite accurately and reliably replicates real-world traffic models representative of vertical industry usage. Along with pre-packaged traffic mixes for healthcare, education, office buildings, government, law enforcement, and hot spot environments, WiMix offers users the ability to recreate the precise traffic mix expected in any given network environment.

WiMix loads the System Under Test (SUT) with this precise traffic mix and reports end-user QoE, one of the toughest challenges facing wireless network planners as the mix of mobile applications continues to grow and diversify. In LACMA’s case, the planned traffic mix was defined by the museum’s Information Systems Management and UpTimeIT.

The desired design goal in terms of the number of users and traffic mix supported are depicted in Table 1. LACMA’s original design goal was for each AP to support up to 79 user devices, but this goal was not met by any of the participating vendors. Testing continued using 25 user devices per access point, and those results led LACMA to increase the number of APs deployed to accommodate the 25-device-per-AP model.

Data Rate Total bandwidth Total bandwidth Number % of bandwidth User allocated consumption consumption per Application of devices allocated per Devices per device per AP for all APs (test-bed Traffic per A application traffic (Mbps) devices (Mbps) topology)

Guest PDA 60 0.200 12.000 72.000 HTTP/HTTPS 100 (Nokia N800) Employee 3 1.000 3.000 18.000 Oracle database 40 laptops HTTP/HTTPS 30 Outlook 30 Scanners Barcode (Symbol 3 0.128 0.384 2.304 100 scanning MCS70) VolP Clients 3 0.450 0.450 2.700 G.711 codec type 100 Guest laptops 10 0.256 2.560 15.360 HTTP/HTTPS 70 POP3, SMTP, 30 IMAP

Table 1 – LACMA Design Goal for number of users supported and traffic mix

Total number of user devices: 79 Total bandwidth consumed per AP: 18.394 Mbps Total bandwidth consumed for 6 APs (test-bed topology): 110.364 Mbps

26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012 CASE STUDY

WiMix

Parameters:

• Guest PDAs create short, 100-byte HTTP transactions

• Employee laptops combine HTTP, Oracle SQL queries over HTTP, and email traffic (MS-Exchange, IMAP, POP, and SMTP)

• Barcode scanners use a short, 20-byte transaction over a proprietary protocol

• VoIP handsets create bi-directional packetized voice traffic using standard SIP, SIP/SDP, and RTP

• Guest laptops combine HTTP and email traffic over IMAP, POP, and SMTP

In addition, the SUT needs to allow for 802.1X authentication and encryption for all user devices, with the exception of the VoIP handsets and guest laptops.

Test Bed Topology

The test bed included the following components:

• 1 x WLAN controller (supplied by participating vendors)

• 6 x 80211 a/b/g APs (supplied by participating vendors)

• 1 x IxVeriWave WT 90 Traffic Generator/Analyzer equipped with 6 x 802.11 a/b/g test interfaces and 1 x Gigabit Ethernet test interface

• 1 x HP ProCurve 3500yl L2/L3 GE/PoE switch

• 1 x Windows 2003 Server providing Active Directory, DHCP, and RADIUS service

• Cables, attenuators, and line splitters as needed

Figure 1 shows the physical configuration of the test bed.

26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012 CASE STUDY

6 Access Points

IxVeriwave Test 90 Traffic generator/analyzer

WLAN Controller

802.1X Authentication Server 802.11a/b/g Connections

Gigabit Ethernet Connection

LACMA’s network design called for hundreds of access points to be mounted in various art galleries, hallways, pubic areas, and office spaces. These APs would be connected to dozens of WLAN network controllers. The test- bed consisted of only six APs connected to a single WLAN controller. Although the test bed was much smaller in scale than the production network, LACMA’s IT management deemed the test bed setup to be sufficiently rigorous to model actual network deployment conditions.

As RF interference is always a concern in WLAN performance benchmarking and can lead to non-repeatable and non-reproducible results. To address this issue, and also to minimize the impact of variability in the RF environment, all APs were directly cabled to the IxVeriWave test interfaces using coaxial cables and SMA connectors.

Test Results

Although LACMA’s design goal was for each AP to support up to 79 user devices, it was quickly determined that this goal could not be met by any of the participating vendors for the actual traffic mix defined. Additional testing revealed that Vendor A’s system was able to support a maximum of 30 user devices per access point, and Vendor B’s system was able to support a maximum of 25. LACMA decided to increase the number of deployed APs to accommodate the 25-user-per-AP model.

Consequently, testing with both vendors continued with 25 user devices per AP. For Vendor A, additional testing with 30 user devices per AP was also conducted. While the number of supported devices had to be reduced, the total bandwidth consumed per AP remained at the design goal of 18.394 Mbps. The relative ratios of each user device and traffic type also remained unchanged as defined in Table 1.

26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012 CASE STUDY

Test 1 Results: WiMix Load Test

The WiMix test models actual conditions on production networks, with multiple clients using multiple SSIDs and each client generating traffic from multiple applications with combinations of short and long frames. This type of realistic traffic mix adds considerable overhead and contention on the medium, causing forwarding rates to be lower than simple “drag race” tests in which one flow or a small number of flows on a single SSID are used to measure performance.

Figure 2 depicts the results of the WiMix Load Test. Both vendor solutions were able to achieve an aggregate forwarding rate of roughly 16 Mbps per AP, or 96-98 Mbps for the 6 AP setup.

250 Theoretical Maximum = 223.938 200

150 Vendor A 113.69 Design Goal = 110.364 Vendor B 100 96.37 98.30

NA Aggregate rate across six AP’s (MBPS) Aggregate rate across six AP’s 0 25 User-devices per AP 30 User-devices per AP

Figure 2 – WiMix Load Test Aggregate Results

1200

1018.30 995.82 1000 918.00

800 Vendor A 600 26601 Agourakbps Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012Vendor B 460.613 400 401.37 307.97 200

NA 0 Per User-device TCP Per User-device UDP Per User-device TCP Per User-device UDP Goodput (kbps); Forwarding-Rate (kbps); Goodput (kbps); Forwarding-Rate (kbps); 25 User-devices per AP 25 User-devices per AP 30 User-devices per AP 30 User-devices per AP 250 Theoretical Maximum = 223.938 200

150 Vendor A 113.69 Design Goal = 110.364 Vendor B 100 96.37 98.30

NA Aggregate rate across six AP’s (MBPS) Aggregate rate across six AP’s 0 25 User-devices per AP 30 User-devices per AP

CASE STUDY

In addition to examining the aggregate results, the QoE was evaluated as it pertains to the various traffic types. These results are depicted in Figure 3.

1200

1018.30 995.82 1000 918.00

800 Vendor A 600 kbps Vendor B 460.613 400 401.37 307.97 200

Figure 3 – WiMix Load Test QoE by Traffic Type

Of particular interest was the quality of voice calls. To measure voice quality, the R-Value measurement was used. R-Value differs from subjective scoring methods such as MOS (Mean Opinion Score) in that it derives a quality measurement using empirical observations of delay and loss. In practice, there is a strong correlation between MOS and R-Value scoring. Both vendors delivered excellent audio quality, and both configured their systems to use the IEEE 802.11e/WMM (Wireless Multimedia Extensions) to prioritize VoIP traffic and ensure consistent delay and jitter.

The voice quality results can be seen in Figure 4.

26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012 CASE STUDY

100

8585.8282 Theoretical Maximum = 93 900 84.8484.84 85.585.56 6 Practical Maximum = 86 “Toll Quality” = 80+ 800 70 60 alue V

R- 50

40 Vendor A 30 Vendor B 20 10 0 RR-VValuel per VVlVolPlP hdhandset,t R-Value per VoV lP handset,h 25 User-Devices per AP 30 User-Devices per AP

Figure 4 – WiMix Load Test – Voice Quality

Test 2 Results: Client Capacity with WiMix Load Test

While LACMA expressed primary interest in forwarding rate an audio quality, it also wanted to know how many clients each AP could support given its traffic mix. To determine the maximum client limit, the WiMix tests were run using different numbers of clients. Tests determined that, with the actual traffic mix defined, Vendor A’s system topped out at 30 user devices while Vendor B’s system topped out at 25.

While Vendor A’s system moved traffic substantially faster with 30 clients than with 25, it also delivered slightly lower audio quality (see Figure 4). This may have been an indication that the system was fully loaded, and that 30 clients per AP was the maximum load, not a recommended load for regular operation in production.

While it was possible to operate both systems with far higher client counts per AP when only a single type of application was supported, as shown by the results of tests 3 and 4, the tradeoffs involved with higher client counts made it problematic to use a single capacity data point as the basis for determining the overall AP count. Multiple devices adding SSID numbers onto each AP would add considerable overhead at the link layer, while devices’ use of variable-size frames at the application and transport layers increased contention.

26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012 CASE STUDY

Test 3 Results: Guest PDA Goodput

To evaluate the end-user experience a visitor to the museum would experience, the TCP goodput of mobile devices was measured. As defined in RFC-2647, goodput measures the rate at which data is transferred, minus any data lost or retransmitted. Since most applications use TCP, which has a retransmission mechanism for dealing with lost datagrams, goodput is a useful way of measuring the rate at which user applications actually receive data.

An analysis of traffic captured from a guest PDA showed a mean transaction size of about 97 bytes (both upstream and downstream). This relatively small data size stresses the frame-per-second processing limits of each system under test.

These results are shown in Figure 5.

13.8913.89 14 13.7513.75 12.7912.79 13.0513.05 12.5512.55

s (Mbps) 12.0812.08 12

10 Vendor A

8 Vendor B 6 4 2

Aggregate TCP Goodput across six AP’ 0 100G GuesG t 255G Guest 400G0 GuestG t PDAs per AP PDAs per AP PDAs per AP

Figure 5 – Guest PDA Goodput

From a design standpoint, it was possible to allocate some APs solely for use by guest PDAs (or any single application type), and thus scale client count per AP. But in light of LACMA’s desire to implement universal WLAN connectivity throughout the museum, this approach would prove costly and complex by requiring additional APs to be deployed to support other applications.

26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012 CASE STUDY

Test 4 Results: Employee Web Traffic Goodput

To evaluate the end-user experience a LACMA employee would experience while using a laptop on the wireless corporate network, an employee web traffic goodput test was conducted. Multiple studies of web object size distribution put the mean object size at somewhere between 8 and 13 Kbytes, and thus larger than a single 802.11 or Ethernet frame can carry. Accordingly, web traffic was broken into multiple frames, each using a maximum payload size of roughly 1460 bytes, the value used in tests.

The web tests were a sort of reverse image of guest PDA tests. The PDA tests used small frames to stress the frame-per-second limits of the two systems, while the web tests measured each system’s byte-per-second capabilities. Taken together, these two tests give an accurate appraisal of each device’s forwarding capabilities.

The results, presented in Figure 6, pointed to Vendor A’s system being capable of moving web traffic faster than Vendor B’s system for all three levels of employee laptops.

160 142.98142.98 144.551444.55 139.361339.36 140 133.43131 3.43 131.63131311.63 130.3213130.0 32 s (Mbps) 120

100 Vendor A

80 Vendor B 60 40 20

Aggregate TCP Goodput across six AP’ 0 10 EpmployeeEpmpl 25 EElEpmployeepmpl 40 EEpmployeel Laptops per AP Laptops per AP Laptops per AP

Figure 6 – Employee Web Traffic Goodput

Conclusion

Precise testing using IxVeriWave systems combined with a highly repeatable test methodology enabled LACMA to make the right vendor choice prior to a substantial capital investment. It also allowed the museum to properly design its network and establish operational criteria, improving the end-user experience and reducing support costs.

26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 9156021-01 Rev. A, July 2012