Case Study Quality that Reflects Experience:DXOMARK Benchmarks Displays Using Real-World Conditions and Scientific Data from ProMetric® Imaging Colorimeters

About DXOMARK DXOMARK provides independent device quality OVERVIEW benchmarks based on scientific assessment. Headquartered in Paris, France, the company op- Company: DXOMARK is a French engineering services erates state-of-the-art labs with documented test company whose testing, software, and benchmarking services help smartphone, speaker, and digital camera makers create protocols to evaluate the objective and perceptual the best possible products. Its website — dxomark.com — is performance of DSLR and DSC cameras and their the world’s largest online source of digital image, audio, and lenses, smartphone cameras, chipsets & ISPs, display quality ratings, trusted by consumers and industry battery life, and displays under real-world condi- leaders to produce objective, accurate analyses of the perfor- tions; DXOMARK also evaluates audio quality of mance of new products. speakers and smartphones. The results of hun- Challenge: Evaluate the visual quality and performance of dreds of tests are aggregated as numerical values, display devices according to how users experience them in which are sub-scores for key quality attributes the real world. and provide an overall quality index, the device’s DXOMARK score. Referenced widely by the global Opportunity: On a short timeline, the company needed to consumer electronics community, a DXOMARK develop a smartphone display test bench to identify, inte- score is used by manufacturers and news media grate, and deploy best-in-class display measurement tools for comprehensive and objective visual performance evaluation to communicate a device’s quality ranking. of attributes such as luminance, color, contrast, uniformity, flicker, gloss, and touchscreen response. Since launching its website in 2008, DXOMARK has rapidly established a reputation among con- Solution Components: Radiant Vision Systems’ ProMetric® sumers and manufacturers as a trusted source of I Imaging Colorimeter, FPD Conoscope Lens, and TrueTest™ reviews based on scientific data. The company Software; Konica Minolta’s CS-2000 Spectroradiometer, CM- 25d Spectrophotometer, and CA-410 Display Color Analyzer. began by testing DSLRs and camera equipment, later adding smartphone camera systems. With Solution Benefits:Radiant’s display test systems offer multi- touchscreen displays the pivotal interface for ple advantages to meet DXOMARK’s needs: smartphone operation, DXOMARK extended its testing to include display performance. Opened • Tristimulus filters that measure light and color according in 2020, DXOMARK’s display test lab reflects the to the human eye’s response • High-resolution imaging that captures data for each pixel company’s dedication to device benchmarking in today’s high-resolution device screens using scientific testing, applying the latest display • Electronic focus and aperture control to automate lens test equipment and methods to evaluate quality settings according to measurement processes according to the user experience. • Flexibility to use a single system for multiple measure- ment needs Display Quality as a Factor of User Experience Results: The DXOMARK Display Bench and test lab was Device manufacturers and display panel makers launched on schedule, achieving measurement with optimal commonly perform visual inspection and display efficiency thanks to the flexibility of the Radiant and Konica calibration as part of their quality control opera- Minolta systems. Using these tools and user-focused test tions. Typically, displays are tested in dark labs, protocols, DXOMARK captures scientific data and delivers with a measurement system aligned at a specific objective display quality benchmarks referenced by manufac- distance and angle relative to the display. These turers and consumers alike. settings limit the effects of variables like ambient at DXOMARK. “DXOMARK’s approach combines light, view angle, and focus, isolating the display perceptual analysis and objective measurements, all to characterize its brightness, color, and other vi- being performed in conditions consistent with actual sual qualities under optimal conditions. Of course, use cases. Our objective was to change the spirit this is not the way a user visualizes their smart- of existing standards by doing measurement where phone or any device display. nobody else wanted to go.”

DXOMARK takes a different approach to mea- Fundamental to DXOMARK’s test processes are sci- surement. “We measure aspects like brightness, entific measurements and repeatable test protocols. contrast, color, flicker—all sorts of parameters, Since user experience of a device is inherently sub- which are physical attributes of the display,” ex- jective—and subjective analysis alone cannot pro- plains Nicolas Touchard, Vice President of Mar- vide a benchmark—the combination of perceptual keting at DXOMARK. “But we make assessments analysis and scientific measurements are required. under real-life conditions. That is something quite Both analyses should be performed in conditions different from the test conditions as defined in the that reproduce the user’s environment for the most industry standards as used by manufacturers.” relevant measurements. Additionally, measurement Users view their devices under a range of ambi- systems, test setups, and protocols should be ent light conditions—at night, in sunlight, indoors, thoroughly defined and always performed the same outdoors—and at variable distances, angles, and way so that measurements are reliable regardless of visibility conditions. Manufacturer testing mea- who performs testing and when testing occurs. As sures specifications of the display itself, while Touchard explains, “If the instruments, the way the DXOMARK provides an objective understanding measurements are done, the setup, the protocols of display quality under real-world conditions. are not defined, you can’t have consistent data.”

Scientific data includes photometric values mea- sured from the display under various conditions, which must be acquired in a controlled environment, limiting human intervention to standardize the test process across devices and ensure objectivity.

DXOMARK’s test processes employ scientific me- trology tools, visual inspection systems and proto- cols, software, and robotics. Devices like spectrom- eters and imaging colorimeters are measurement systems that acquire photometric data such as a display’s luminance value (Lv), chromaticity coordi- nates (u ,v on the CIE 1976 chromaticity diagram), brightness and color uniformity, and contrast. Ro- botics automate´ ´ test equipment and devices, in- cluding alignment, relative angle, and positioning, to match real-world test scenarios. A centralized software platform controls all aspects of the test process remotely, such as sending test patterns to the display via WiFi, operating measurement equipment and robotics, and adjusting variables like “All the standards right now are based on mea- ambient light. surements made without ambient lighting, but that’s not a consumer use case,” explains DXOMARK’s test protocols define the procedure François Grandclerc, Display Quality Director and conditions relevant to measuring display at- tributes (performance parameters to be tested) dure. Several tools would also need to be selected according to each device use case (how a device to accomplish the objectives for testing according would be used in the real world). “When we do to the defined attributes and use cases. our scoring, we test across many use cases like web-browsing, night reading, in-car navigation, DXOMARK evaluated a range of display test in- gaming, watching videos. All of these use cases struments for the display test lab, including imag- encompass what people will do with their smart- ing colorimeters, specialized lenses, gloss meters, phone,” explains Touchard. “In each of these sce- flicker measurement devices, spectroradiometers, narios, we check attributes and do measurements high-speed cameras, and even robotics to test of physical characteristics across these attributes, touchscreen response. Bringing together all of the and we do this considering the ambient environ- pieces for automated display evaluation also meant ment—the environment of the user.” integrating measurement equipment, robotics, and software into a controlled fixture. The DXOMARK Not all display attributes must be tested for every lab features a custom-engineered test chamber— possible use case. DXOMARK’s test protocols en- the Display Bench—that automates DXOMARK’s sure the most relevant attributes are evaluated for display testing under repeatable conditions. The each scenario. For a use case like web browsing, Display Bench includes robotic mounts for remote for example, most display attributes (readability, alignment of smartphones and measurement equip- color, motion, touch, artifacts) should be tested as ment and an LED panel to mimic ambient lighting part of the user’s web browsing experience. For conditions from darkness to daylight. in-car navigation, however, fewer display attri- butes (readability, artifacts) require testing as the For its automated visual inspection system, user is not typically interacting with the display. DXOMARK chose the ProMetric® I-series Imaging Colorimeter and FPD Conoscope Lens from Radiant Components of a Scientific Display Test Lab Vision Systems. Within the Display Bench, the To meet the demand for display quality bench- ProMetric I Imaging Colorimeter measures lumi- marking, the DXOMARK team set out to develop a nance, chromaticity, and uniformity of the display new test lab dedicated to evaluating display per- in each test scenario, as ambient lighting and other formance. DXOMARK hired several specialists to conditions are varied for each display attribute and outline their display quality benchmarking proce- use case. Each measurement uses multiple test

A matrix of display quality attributes, testing conditions, and use cases that illustrate the complexity of DXOMARK display tests based on real-world conditions. Image courtesy of DXOMARK. Inside DXOMARK’s Display Bench, featuring a Radiant ProMetric® I Imaging Colorimeter, multi-axis robotic mounting equipment, and LED array to emulate lighting in use-case scenarios. Image courtesy of DXOMARK. images, such as a white image for luminance and uniformity, colored images for chromaticity mea- surement, and a checkerboard contrast image for contrast and modulation transfer function (MTF).

Radiant’s equipment relies on imaging, enabling the ProMetric system to capture the entire dis- play at once and measure multiple attributes in a matter of seconds. Test images are pushed to the display in sync with camera and software oper- ations, which are specific to each measurement The ProMetric I Imaging Colorimeter synchronizes test scenario, providing fully automated testing without operations with images on the display for comprehensive downtime. evaluation in seconds.

Used both inside and inside the Display Bench, Radiant’s FPD Conoscope Lens is paired with the same ProMetric Imaging Colorimeter to capture a polar plot of the display as viewed to ±70° in a single image. This measurement characterizes luminance, chromaticity, and contrast changes as the display is viewed by a user from different angles. A typical use case would be when mul- tiple users are viewing photos or movies on the A Radiant Vision Systems FPD Conoscope Lens measures same device, and each user’s vantage point varies the view angle performance of a display (left), providing a based on their position relative to the display. polar plot of luminance as seen across angles to ±70° (right). The DXOMARK display test lab also incorporates Radiant’s ProMetric imaging systems also come in equipment from Radiant’s sister company, Konica a range of high-resolution sensor options to accom- Minolta Sensing in Europe. In the Display Bench, a modate the increasing resolutions of today’s smart Konica Minolta CS-2000 Spectroradiometer mea- device displays, and even measure multiple displays sures brightness and color fidelity of the device at once. over changing ambient-light conditions and to provide calibration data for the Radiant ProMetric “For a smartphone now, you can have a 6-inch Imaging Colorimeter. A Konica Minolta CM-25d display with around 4K resolution. So, we needed Spectrophotometer is also used to evaluate gloss a tool that shows us a luminance or color map with based on light reflected from the display surface. enough pixels to measure one device—at least,” states Grandclerc. “But in our approach, we mea- Benefits of Radiant Vision Systems Equipment sure four devices at the same time, and we need The capabilities of Radiant’s test and measure- sufficient information measured across all devices. ment equipment align well with DXOMARK’s goals We needed the highest resolution available on the for user-focused display testing in several ways. market, and it was Radiant.”

Radiant’s imaging colorimeters not only provide Radiant equipment also gives DXOMARK flexibility objective, scientific data according to colorimetric to meet a range of test criteria. When testing view values, but are designed with tristimulus optical angle performance, the ProMetric Imaging Colorim- filter systems that measure light as it is perceived eter’s standard lens can be swapped with the FPD by the human eye. During measurement, sub- Conoscope Lens to acquire angular measurements, sequent exposures of the display are captured providing “the ability to switch from one to the other through each of the camera’s filters, which are without buying another tool,” comments Grandclerc. weighted to human spectral sensitivity curves (color-matching functions standardized by CIE), ensuring the truest measure of light and color quality according to human perception.

Radiant’s ProMetric I Imaging Colorimeters measure color with an innate spectral response that matches standard CIE color-matching functions, quantifying display color as perceived by a user. Equipment is only one part of a complete solution, however. Global support from both Radiant Vision Systems in the U.S.A. and Konica Minolta Sensing in Europe were equally integral to DXOMARK’s test setup and operations. “The quick and profession- al support from the team in the United States and here in Europe is something that is very important,” states Grandclerc. “The Radiant camera is a key tool for us, so when there is an issue, we need quick support.” Although development of the display test lab extended into 2020, amid the COVID-19 pan- demic, team members from both Radiant and Kon- ica Minolta came together to ensure DXOMARK’s Radiant ProMetric imaging systems can be paired goals were met. “We even did some calibrations via with a range of electronically controlled and spe- video-conferencing,” notes Grandclerc. cialized lens options (like the FPD Conoscope) to accomplish a range of automated testing. Beyond a Benchmarking Score Looking to the future, DXOMARK plans extend its Another benefit of Radiant’s equipment is elec- display quality testing, adding additional equipment tronic control. The focus and aperture settings of and protocols to evaluate displays across new use the ProMetric imaging system’s lenses are re- cases. However, the company’s focus is not limit- motely adjustable from software. Integrated with ed to applying scientific measurement to produce automated controls, specific lens settings can be benchmark scoring for consumers. applied for different device positions and align- ments set by robotics to ensure repeatable testing As a leader in measurement across the entire device without human intervention. “We can develop a component ecosystem, DXOMARK aims to give smart algorithm to detect where the best focus manufacturers and consumers alike a view of dis- is—same for the aperture,” states Grandclerc. play performance that is easy-to-understand, rele- “We wanted something that we could control elec- vant, and furthers device innovation. DXOMARK’s tronically without putting an operator inside of the test data not only allows apples-to-apples compar- Bench to check everything—to avoid any mis- ison of quality across devices but provides manu- takes, to be sure we are repeatable. Repeatability facturers and display panel makers with scientific is key in our business.” metrics that help drive development of devices with improved performance. This knowledge is key for Similarly, each ProMetric imaging system is con- any manufacturer whose goal is to give customers trolled by software to capture images and apply the highest quality user experiences, beyond display measurements in sync with automated processes. performance specifications measured in a dark lab. Radiant’s TrueTest™ Automated Visual Inspection Software is the control platform for each Radiant “Our primary purpose is to make accurate, repeat- device and features programmable API and SDK able measurements and perceptual analyses that to enable integration into fully automated test correlate with human perception, and which (at the benches and testing processes. Commenting on end of the day) enable us to give a score—which is the importance of software integration, Grandclerc very consumer oriented,” explains Touchard. “Then recalls, “It was maybe one of my first questions to our goal is to deliver very detailed reports of the Radiant—what can you do with the SDK to drive characteristics of the device we’re testing—serv- the camera?” With this capability, DXOMARK was ing our customers and providing very accurate and able to incorporate the Radiant hardware and in-depth analysis so that they can understand how software functionality into its turnkey test system. they are positioned relative to competition and how they can make improvements.” With scientific data from Radiant’s ProMetric GLOBAL OFFICE LOCATIONS Imaging Colorimeters, Konica Minolta systems, and other leading display test and measurement Radiant maintains direct sales, engineering, and support offices equipment, DXOMARK guides the consumer and personnel throughout North America, China, and Korea. Radiant is also sold and supported in other areas of the world by electronics industry toward a better understanding our sister offices in the Konica Minolta Sensing Business. of device quality based on the ultimate standard— the user experience. By qualifying the perfor- AMERICAS mance of displays under real-world conditions, Global HQ Radiant Vision Systems DXOMARK helps set the industry’s focus on 18640 NE 67th Ct. user-centric design. Redmond, WA 98052 USA +1 425 844-0152 [email protected] Learn more about DXOMARK: www.dxomark.com Regional Offices Cupertino, California Novi, Michigan

ASIA “DXOMARK develops and uses China HQ Shanghai, China Regional Offices Shenzhen, China test methods to mimic smartphone Suzhou, China

users under different conditions, Japan Tokyo, Japan and the Radiant camera provides Korea Seongnam, South Korea Taiwan Area Zhubei, Taiwan very accurate, repeatable, extensive Asia-Pacific Singapore measurements to test all photometric parameters that count for visual EUROPE Europe HQ Nieuwegein, Netherlands display quality.” Regional Offices Diegem, Belgium - Nicolas Touchard, Vice President of Marketing, DXOMARK Paris, France Munich, Germany Milan, Italy Wrocław, Poland Vastra Frolunda, Sweden Dietikon, Switzerland Istanbul, Turkey Warrington, United Kingdom

Contact your local Radiant office: RadiantVisionSystems.com

Copyright ©2021 Radiant Vision Systems LLC. All rights reserved. Specifications are subject to change without notice. Radiant, Radiant Vision Systems, and ProMetric, are registered A Konica Minolta Company trademarks of Radiant Vision Systems LLC. 2021/03/15