Result Analysis: Statistical Distributions: LRE Algorithm and Binomial Proportion Cis

Result Confidence Assessment and Trial Planning Maciej Muehleisen Ericsson (ERI) [email protected]

29th of March, 5GCroCo Lunchtime Web-Seminar 1 (Hosted by 5G-PPP) 2

5G Cross Border Control

Innovation Action H2020-ICT-18-2018 Contract 825050 Cooperative, Connected and Autonomous Mobility (CCAM) a 5G-PPP Phase III Project Before we Start…

• This presentation is being recorded and recording will be shared • Slides will be shared • MATLAB scripts for presented methods are listed at the end Outline

• About Me • Background: 5GCroCo Deliverable D4.2 • Thoughts about Terms: PoC vs. Demo vs. Test vs. Trial • Use Case: Anticipated Cooperative Collision Avoidance (ACCA) • Trial Execution • How to determine if KPI requirements are achieved? • How to assure “identical” experiments • Result analysis • Plausibility Checks • Student-t Confidence Intervals (CIs) using Batch Means Method • Statistical Distributions: Limited Relative Error (LRE) Algorithm and Binomial Proportion CIs • Summary and Conclusion About Me

Key research interest: Modelling, design, evaluation, and certification of highly reliable / safety critical communication systems • 2008 – 2012 ComNets - RWTH Aachen University • Leading Communication Networks I Exercise • 4G IMT-Advanced Evaluation • Open Wireless Network Simulator developer • PhD research on “VoIP Performance of LTE Networks: VoLTE versus OTT” (2015) • 2012 – 2016 ComNets - Hamburg University of Technology (TUHH) • Lecturer Communication Networks I • Group leader “Mobile & Vehicular Communication” (focus on aviation, maritime) • Sometimes acting group leader for “Sensor Networks and IoT” & “Future Internet and Network Planning ” • Since 2017 Ericsson Research Germany • Research Area “Networks” – Master Researcher - Industry Verticals Coordination (focus on automotive) • Coordination of tech. work in external associations (5GAA, AECC, ETSI-ITS) and projects (5GCroCo, 5GMOBIX, 5G-ROUTES, ART-04 SHOW) • Deputy Technical Coordinator 5GCroCo 5GCroCo Deliverable D4.2

• For first version (v1.0) of 5GCroCo result Deliverable D4.2, many experiments did not go as expected • Many trials started later due to COVID and no time to repeat • Often too few samples collected and/or measurement equipment failed (e.g. clock drift) • v2.0 end of March; v3.0 June • Second trial round starting in summer and lasting until end of December Link Thoughts about Terms

Proof of Concept (PoC) & demo are usually the same, but sometimes the following applies: • Demo: physically show the actual service/product; shortcuts can exist, e.g. Ethernet instead of 4G/5G • PoC: can be same as demo; in ICT context it often requires that the actual communication system is used to prove it is capable to deliver the service In 5GCroCo we strictly distinguish between tests and trials: • Test: one or more components, or even all components, are evaluated if they behave as expected; can be by inspection or quantitative (measurement); typically done before setting up demo, PoC or trial • Trial: quantitative evaluation of the system by measuring previously defined KPIs in defined scenarios • KPIs are usually on application/service level • Further measurements can be collected to better understand/explain measured KPIs (“PIs”) • Quantitative test results (see bullet above) can also be used for better understanding of measured KPIs Use Case: Anticipated Cooperative Collision Avoidance (ACCA)

• User Story 1: “stationary vehicle” broke down and sends a “Hazard Report” to the backend • User Story 2: no “Hazard Report” but backend Link to video: analyzes Cooperative Awareness Messages (CAMs) to detect the “stationary vehicle” and https://www.youtube.com/watch?v=jehWj4sq9Zc send “Hazard Notifications” • User Story 3: Backend analyzes CAMs to detect traffic jam and send “Hazard Notifications” • Round 2 (second half 2021): Detection by vehicle sensors Trial Execution: How to Determine if KPI Requirements are Achieved? • Application Level Reliability: ≥99% • Too late ➔ lost (1 s delay allowed) Hazard ACCA Backend Notification Hazard Report JSON over MQTT (Renault uses for PSA ETSI-ITS DENM over UDP) Hazard Notification DENM over UDP for Renault Trial Execution: How to Determine if KPI Requirements are Achieved? • Application Level Reliability: ≥99% • Too late ➔ lost (1 s delay allowed) ACCA Backend • Counting Hazard Report intended (Renault uses receivers, e.g. ETSI-ITS DENM • FailureCount += over UDP) 3 if Hazard Report lost Trial Execution: How to Determine if KPI Requirements are Achieved? • Application Level Reliability: ≥99% • Too late ➔ lost (1 s delay allowed) Hazard ACCA Backend Notification • Counting Hazard Report JSON over MQTT intended (Renault uses for PSA receivers, e.g. ETSI-ITS DENM over UDP) Hazard • FailureCount += Notification 3 if Hazard DENM Report lost over UDP • FailureCount += for Renault 1 for every lost Hazard Notification Trial Execution: How to Determine if KPI Requirements are Achieved?

• Application Level Reliability: ≥99%

• How many “Hazard Reports” do we need to confidently determine the KPI?

• How long will it take?

• Can we speed it up? Trial Execution: How to Determine if KPI Requirements are Achieved?

• Application Level ●Answers: Reliability: ≥99%

• How many “Hazard ●100 – 1000 “events” as a rule of Reports” do we need thumb: to confidently 1 / 100 = 99% ➔ determine the KPI? (100 to 1000) / (10000 to 100000) = 99%

• How long will it take?

• Can we speed it up? Trial Execution: How to Determine if KPI Requirements are Achieved?

• Application Level ●Answers: Reliability: ≥99%

• How many “Hazard ●100 – 1000 “events” as a rule of Reports” do we need thumb: to confidently 1 / 100 = 99% ➔ determine the KPI? (100 to 1000) / (10000 to 100000) = 99%

• How long will it ●10000 to 100000 “hazards” if one take? vehicle receives

• Can we speed it up? Trial Execution: How to Determine if KPI Requirements are Achieved?

• Application Level ●Answers: Reliability: ≥99%

• How many “Hazard ●100 – 1000 “events” as a rule of Reports” do we need thumb: to confidently 1 / 100 = 99% ➔ determine the KPI? (100 to 1000) / (10000 to 100000) = 99%

• How long will it ●10000 to 100000 “hazards” if one take? vehicle receives

• Can we speed it up? ●Yes, but watch out for pitfalls Trial Execution: How to Determine if KPI Requirements are Achieved?

Tipps and precautions when “tricking time”: Influence of power saving (DRX) • The exponential distribution models “random occurrence” well • It prevents unintended correlations from periodicities, e.g.: • Transmission Time Interval (TTI) slot boundaries • Time Division Duplex (TDD) frame durations • Gradually decrease the mean interarrival time to check if results remain the same • Control Plane time outs • Network overload How to Assure “Identical” Experiments

• In simulation everything can be the same except from random number initialization • In real-world trials you can just try your best, esp. on public roads • Static test in perfect radio conditions before drive testing • Ping and Iperf tests for preparation • Repeat the experiments with changing parameters (e.g. 4G vs. 5G) directly one after another • Keep antenna placement identical • Adjust trial duration and path to the „largest time- scale effect“ (usually radio channel quality from distance) • Think about what impacts the KPIs and create according scenarios (see 5GCroCo Deliverable D4.1 Section 6.3 and D4.2 „Influence on KPIs“ sections for each use case) Result Analysis: Plausibility Checks

• Check for known maxima and ? minima: • Maximum throughput according to iperf/nuttcp and/or known Spectral Efficiency • Minimum latency / round trip time according to Medium Access Control (MAC) protocol and RAN config Result Analysis: Plausibility Checks

• First check time series (raw samples) before doing statistical analysis; consider publishing them • Check timestamps of all nodes for monotonous increase Result Analysis: Plausibility Checks

# Confidence • # Samples CI / Do not consider erroneous samples User Story Test Case # Samples Mean [ms] Interval CI [ms] Considered Mean in analysis but explain why/how you (CI) censored Batches PSA➔PSA 444 230 638.0 5 ± 12.6 2.0 % PSA➔RSA 130 115 633.1 5 ± 6.9 1.1 % 1 RSA➔RSA 167 155 625.7 5 ± 4.3 0.7 % RSA➔PSA 204 95 608.9 5 ± 13.8 2.3 % TMS➔PSA 894 655 474.6 10 ± 5.7 1.2 % 3 TMS➔RSA 243 195 41.8 10 ± 1.7 4.1 % D4.2 v1.0 Table 3-26: Application Level Latency Result Analysis: Plausibility Checks

# Confidence • # Samples CI / Do not consider erroneous samples User Story Test Case # Samples Mean [ms] Interval CI [ms] Considered Mean in analysis but explain why/how you (CI) censored Batches PSA➔PSA 444 230 638.0 5 ± 12.6 2.0 % • Or just let go of it and repeat the PSA➔RSA 130 115 633.1 5 ± 6.9 1.1 % 1 trial RSA➔RSA 167 155 625.7 5 ± 4.3 0.7 % RSA➔PSA 204 95 608.9 5 ± 13.8 2.3 % • „Anything that can go wrong, will go TMS➔PSA 894 655 474.6 10 ± 5.7 1.2 % 3 wrong“, Edward A. Murphy, TMS➔RSA 243 195 41.8 10 ± 1.7 4.1 % Aerospace Engineer D4.2 v1.0 Table 3-26: Application Level Latency

User Story Test Case # # Samples Mean # CI CI / Max. LRE ➔ Determine required time for Samples Considered [ms] Confidence [ms] Mean Confidence trials and take it time 3 Interval (CI) with Rel. • Batches Error Below Try 1: get familiar with each 5% [ms] / other and the equipment Percentile • Try 2: Good results, but some PSA1➔PSA1 4445 Due to several problems, summarized in Section 4.3 together with 1 PSA2➔PSA2 444 solutions that are being applied, these results cannot be processed failed experiments or filtered to allow a sensible analysis • Try 3: Done! D4.2 v1.0 Table 3-20: Application Level Latency Based on Hazard Reports/Notifications for PSA Result Analysis

Infinite set of all possible KPI measurements for this scenario (population)

ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddd….

E.g. delay “d” Result Analysis

Infinite set of all possible Experiments “extract” samples KPI measurements for this out of this population scenario (population)

σ푁푏 푑 ddddddddddd Experiment “blue”: d d d … d ➔ 푑 = 푖=1 푏,푖 ddddddddddd b,1 b,2 b,3 b,Nb 푏 푁푏 ddddddddddd ddddddddddd How “representative” are the ddddddddddd ddddddddddd samples from experiment ddddddddddd “blue”? ddddddddddd ddddddddddd ddddddd….

E.g. delay “d” Result Analysis

Infinite set of all possible Experiments “extract” samples KPI measurements for this out of this population scenario (population)

E.g. delay “d” result is “similar” Result Analysis

Infinite set of all possible Experiments “extract” samples KPI measurements for this out of this population scenario (population)

E.g. delay “d” Result Analysis

Infinite set of all possible Experiments “extract” samples KPI measurements for this out of this population scenario (population)

σ푁푏 푑 ddddddddddd Experiment “blue”: d d d … d ➔ 푑 = 푖=1 푏,푖 ddddddddddd b,1 b,2 b,3 b,Nb 푏 푁푏 ddddddddddd ddddddddddd σ푁푟 푑 ddddddddddd 푖=1 푟,푖 Experiment “red”: dr,1 dr,2 dr,3 … dr,Nr ➔ 푑푟 = ddddddddddd 푁푟 ddddddddddd 푁푔 ddddddddddd σ푖=1 푑푔,푖 ddddddddddd Experiment “green”: dg,1 dg,2 dg,3 … dg,Ng ➔ 푑푔 = ddddddd…. 푁푔 ..... E.g. delay “d” Result Analysis

Infinite set of all possible Experiments “extract” samples KPI measurements for this out of this population scenario (population) Do one long run (e.g. several rounds) and split into result “batches”:

ddddddddddd db,1 db,2 db,3 … db,N dr,1 dr,2 dr,3 … dr,N dg,1 dg,2 dg,3 … dg,N ddddddddddd ddddddddddd ddddddddddd ddddddddddd Be sure batches are “identical” ddddddddddd ddddddddddd ➔ Not too many batches ddddddddddd ➔ One or more “rounds” should ddddddddddd form a batch ddddddd….

E.g. delay “d” Result Analysis: Example Population is exponentially distributed with mean We will do 20 identical experiments with 1000 samples each 25 ms (in reality, distribution OR is of course unknown) We do an experiment with 20000 samples and split it into 20 batches

ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddd….

Standard deviation of experiment/batch means can E.g. delay “d” indicate similarity/divergence of experiments Result Analysis: Example Population is exponentially distributed with mean We will do 20 identical experiments with 100 samples each 25 ms (in reality, distribution OR is of course unknown) We do an experiment with 2000 samples and split it into 20 batches

ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddd….

Standard deviation of experiment/batch means can E.g. delay “d” indicate similarity/divergence of experiments Result Analysis: Example Population is exponentially distributed with mean We will do 20 identical experiments with 1000 samples each 25 ms (in reality, distribution OR is of course unknown) We do an experiment with 20000 samples and split it into 20 batches

ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddd….

E.g. delay “d” Result Analysis: Example Population is exponentially distributed with mean We will do 100000 identical experiments with 750 | 1000 samples each 25 ms (in reality, distribution OR is of course unknown) We do an experiment with [750 | 1000] x 100000 samples and split it into 100000 batches

ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddddddd ddddddd….

The sample means are exponentially distributed E.g. delay “d” regardless of the distribution of the population Result Analysis: Student-t CIs Using Batch

Means Method The „true“ result is in here with 95 % probability • Let x1 … xk be some statistical property of a KPI from k independent and identical trials Upper ➔ The sample mean 풌 Lower CI σ풊=ퟏ 풙풊 풌 CI is Student-t distributed with k degrees of freedom

• In the following, we consider xi to be the mean values of a KPI for each trial i (or result batch from trial, see next slide) • It works with any statistical property like median (50- percentile), p-percentile, variance, …

• MATLAB: [a,b,c] = ttest(x), where x is the vector containing x1 … xk and c(1), c(2) are the lower and upper CI bounds around mean(x) • (mean(x) – c(1)) / mean(x) shows the relative size of the CI around the mean ➔ typical 5 % - 10 % can be tolerated (more on next slides) Result Analysis: Student-t CIs Using Batch Means Method

• In simulation you just Clock time gliches restart your simulator with different random number seeds • For trials you can do the same several times or • You do one long trial and separate results into equally sized „batches“ Result Analysis: Student-t CIs Using Batch Means Method

• In simulation you just restart your simulator with different random number seeds • For trials you can do the 1 … 10 same several times or • You do one long trial and separate results into equally sized „batches“ • Try 10 batches first, but: • Batches should be „identical“ • Should contain min 50 samples each Result Analysis: Student-t CIs Using Batch Means Method CI is 0.3 % of mean • In simulation you just ➔ Confident result restart your simulator with different random number seeds • For trials you can do the same several times or • You do one long trial and separate results into equally sized „batches“ • Try 10 batches first, but: • Batches should be „identical“ • Should contain min 50 samples each Result Analysis: Student-t CIs Using Batch Means Method

User Story Test Case # Samples # Samples Mean [ms] # Confidence CI [ms] CI / Mean • In 5GCroCo we Considered Interval (CI) Batches defined 10 % PSA1➔PSA1 17358 17330 38.4 10 ± 0.1 0.3 % PSA2➔PSA2 7232 7220 34.8 10 ± 0.1 0.3 % 1 to be RSA➔RSA 70 70 44.0 5 ± 8.4 19.2 % PSA➔RSA 45 45 164.6 5 ± 37.4 22.7 % acceptable size D4.2 v1.0 Table 3-21: Application Level Latency Based on Hazard Reports/Notifications for PSA

of CI CI is 22.7 % of mean • CIs may not ➔ Not confident result overlap to confidently state „A is better than B“ Result analysis: Statistical distributions: LRE Algorithm and Binomial Proportion CIs

• How likely is the latency above the requirement? • Binomial experiment (two possible outcomes): “above” or “below” threshold • Binomial Proportional CI: 푝ො(1−푝ො) • 퐶퐼 = ±푧 푛 • z = 1.96 for 95 % CI • 푝Ƹ fraction of desired outcomes (e.g. “below 75 ms”) • n number of independent sample • Consecutive packets are not “independent” (buffers, radio channel quality, …) • Still allows a first estimation • Minimum required number of samples (solve for n) • If this CI is too large, also LRE error (following slides) will be too large • 5 % - 10 % can be agreed to be acceptable • E.g. for a 99-percentile it is 5 % - 10 % of 1 - 0.99, not of 0.99: CI ≤ 0.0005 for 5 % and CI ≤ 0.001 for 10 % Result analysis: Statistical distributions: LRE Algorithm and Binomial Proportion CIs

• How likely is the latency above the requirement? • Binomial experiment (two possible outcomes): “above” or “below” threshold • Binomial Proportional CI: 푝Ƹ(1−푝Ƹ) • 퐶퐼 = ±푧 푛 • z = 1.96 for 95 % CI • 푝Ƹ fraction of desired outcomes (e.g. “below 75 ms”) • n number of independent sample • Consecutive packets are not “independent” (buffers, radio channel quality, …) • Still allows a first estimation • Minimum required number of samples (solve for n) • If this CI is too large, also LRE error (following User Test Case # 50-Percentile (Median) 90-Percentile slides) will be too large Story Sam # Per- CI Rel. # Per- CI Rel. CI • 5 % - 10 % can be agreed to be acceptable -ples Sam- centile CI Sam- centil • E.g. for a 99-percentile it is 5 % - 10 % of ples [ms] ples e [ms] 1 - 0.99, not of 0.99: RSA➔RSA 70 35 37 ±12 % 24 % 63 61 ±7 % 70 % 1 CI ≤ 0.0005 for 5 % and CI ≤ 0.001 for 10 PSA➔RSA 45 22 105 ±15 % 30 % 40 398 ±9 % 90 % % D4.2 v1.0 Table 3-22: 50- and 90-Percentile of Application Level Latency for Trials with RSA Vehicles and Corresponding Binomial Proportion CIs Result analysis: Statistical distributions: LRE Algorithm and Binomial Proportion CIs

Minimum number of uncorrelated samples n for an error below 10 %: • 50-percentile: 384 • 90-percentile: 3460 • 99-percentile: 38000

푝Ƹ(1 − 푝Ƹ) 퐶퐼 = ±푧 푛

• Origin of the 100 – 1000 “rule of thumb” User Test Case # 50-Percentile (Median) 90-Percentile introduced earlier (Slide 11) Story Sam # Per- CI Rel. # Per- CI Rel. CI • Once several 1000 samples or more -ples Sam- centile CI Sam- centil ples [ms] ples e [ms] available ➔ Use LRE Algorithm RSA➔RSA 70 35 37 ±12 % 24 % 63 61 ±7 % 70 % 1 PSA➔RSA 45 22 105 ±15 % 30 % 40 398 ±9 % 90 % D4.2 v1.0 Table 3-22: 50- and 90-Percentile of Application Level Latency for Trials with RSA Vehicles and Corresponding Binomial Proportion CIs Result analysis: Statistical distributions: LRE Algorithm and Binomial Proportion CIs

• Limited Relative Error (LRE) Algorithm • Developed at RWTH Aachen by Prof. Bernhard Walke‘s predecessor Prof. Schreiber • Published in English, but, back then, in German journal „AEÜ - International Journal of Electronics and Communications “ • Relative: Is my confidence bad because I did the experiment too short / not often enough or because it is so „random“ ➔ normalize confidence measure to standard deviation • Idea: Extend the binomial „above or below“ experiment to also capture correlation through transition probabilities Result analysis: Statistical distributions: LRE Algorithm and Binomial Proportion CIs

• RWTH Aachen ComNets Institute: • LRE was part of simulators e.g. OpenWNS 17330 samples • TU Hamburg (TUHH) ComNets Institute: • Stand alone command line and Python code was extracted (considered packets) from that (Link) • Was then also integrate to OMNet++ simulator (Link) • Prerequisite: • Many more than 1000 samples • Find suitable minimum, maximum and bin size • >100 samples per bin • >100 transitions per bin • Going from „1 ms“ to „5 ms“ at 1 ms bin size causes a „transition“ at 2, 3, 4, and 5 ms bin • LRE result: • Bins you can trust • Bins you cannot trust, because: • Prerequisites not met • Error too high (default max. error 5 % corresponding to 95 % commonly used for CIs) Result analysis: Statistical distributions: LRE Algorithm and Binomial Proportion CIs

This is not LRE <7.5 ms ≥7.5 ms (but maybe Binomial Proportion)

LRE Demo: 5G MEC Ping results from 5GCroCo 5G Corridor (Luxembourg) Result analysis: Statistical distributions: LRE Algorithm and Binomial Proportion CIs

This is LRE <7.5 ms ≥7.5 ms

LRE Demo: 5G MEC Ping results from 5GCroCo 5G Corridor (Luxembourg) Summary and Conclusion

• Prepare and test as much as you can „Anything that can go • With Ethernet / DSL / virtual interfaces / … • With public mobile radio networks wrong, will go wrong“, • In labs Edward A. Murphy, • … • One day before or earlier: assure you can connect the Aerospace Engineer modem/router to the intended trial network (and do not use it

until the actual trials) Vilfredo Pareto • Still be prepared that things will go wrong 80% of the result is • Check the KPI requirements and determine obtained with 20% • The most challenging KPI wrt obtaining tangible (confident) of the total effort results • An estimate of required samples • Key factors influenceing the KPI There’s another 80% • Time frames and geographical dependencies of the key factors effort to finish ➔ Trial duration and path To know more: Maciej Muehleisen http://5gcroco.eu Follow us in twitter: @5GCroCo Ericsson (ERI) Connect in LinkedIn [email protected] Subscribe to our Newsletter Contact us: [email protected] MATLAB Batch Mean 95% CI function [m, ci, relCI] = getBatchMeanCI(data, batches) l = floor(length(data) / batches); d = data(1:(l * batches)); dBatch = vec2mat(d, batches); m = mean(d); [a, b, c] = ttest(mean(dBatch)); ci = m - c(1); relCI = ci/m; MATLAB Binomial Proportion 95% CI

% n is the total number of samples and np the ones with desired outcome function [ci, relCI, p] = getBiCI(np, n) p = np / n; z = 1.96; ci = z * sqrt((p * (1 - p))/n); if p < 0.5; relCI = ci / p; else; relCI = ci / (1 - p); end; C++ / Python Implementation of LRE

TUHH ComNets Git Repository