When Does a Bit Matter? Techniques for Verifying the Correctness of Assembly Languages and Floating-Point Programs Samuel D. Pollard Computer and Information Science 28 May 2021 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL 1 Introduction 2 Binary Analysis 3 A Statistical Analysis of Error in MPI Reduction Operations 4 Scalable Error Analysis for Floating-Point Program Optimization 5 Conclusion and Future Research Directions 2 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Framing My Thesis I I enjoy working with either no abstraction or lots of abstraction • Assembly • Java , • Matlab/ I I noticed a couple, common abstractions which when they failed were hard to fix • Instruction Set Architectures (ISAs) • Floating Point (FP) 3 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Some Intuitive Definitions I High-level: using abstractions; not concerned with underlying implementation of a program I Low-level: the opposite Key Challenge Abstractions give insight into the nature of a program. 4 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Dissertation Question How can we apply high-level reasoning techniques about computer programs to low-level implementations? Specifically, 1 How can we write specifications of instruction set architectures (ISAs) that enable static analysis for program verification? 2 How can we formalize and quantify the error from floating-point arithmetic in high-performance numerical programs? 5 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL 1 Introduction 2 Binary Analysis 3 A Statistical Analysis of Error in MPI Reduction Operations 4 Scalable Error Analysis for Floating-Point Program Optimization 5 Conclusion and Future Research Directions 6 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Quameleon: A Lifter and Intermediate Language for Binary Analysis Based on previously published work in collaboration with Philip Johnson-Freyd, Jon Aytac, Tristan Duckworth, Michael J. Carson, Geoffrey C. Hulette, and Christopher B. Harrison [6] 7 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Motivation I Need to analyze binaries on old, obscure ISAs • ISAs not supported by existing tools • No machine-readable specification • Bad old days: No IEEE 754 floats, no 8-bit bytes I Other tools gain lots of efficiency from expressive ISAs and feature-rich Intermediate Languages (ILs) I We instead require an adaptable IL Fun example: cLEMENCy ISA invented for DEFCON had 9-bit bytes, 27-bit words, middle-endian [9] 8 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Architectural Overview ISA Concrete Execution Engine Specification DSL Custom Symbolic Execution Engines Quameleon M6800 Intermediate Weakest Precondition Language Other ISAs LLVM/KLEE Angr toolchain (Symbolic Execution, etc.) Optimizations for Analysis Abstract Interpretation 9 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Architectural Overview ISA Concrete Execution Engine Specification DSL Custom Symbolic Execution Engines Quameleon M6800 Intermediate Weakest Precondition Language Other ISAs LLVM/KLEE Angr toolchain (Symbolic Execution, etc.) Optimizations for Analysis Abstract Interpretation 9 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Design Goals of the Quameleon Intermediate Language (QIL) I Sound analysis of binaries I Lift binaries into a simple IL amenable to multiple analysis backends I Close to LLVM IR in spirit I Size of QIL (∼ 60 instructions) means easy to manipulate, harder to write I Balance this with Haskell as a macro-assembler for QIL 10 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Quameleon Intermediate Language (QIL) I Static Single Assignment (SSA) I Program consists of a list of blocks, single entry, multiple-exit I Data are stored in bit vectors of parametrizable width I Can read/write to locations like RAM, registers I Keep track of I/O as sequence of reads/writes 11 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Haskell Embedded Domain Specific Language (DSL) ISA Concrete Execution Engine Specification DSL Custom Symbolic Execution Engines Quameleon M6800 Intermediate Weakest Precondition Language Other ISAs LLVM/KLEE Angr toolchain (Symbolic Execution, etc.) Optimizations for Analysis Abstract Interpretation 12 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Sample M6800 A <− 0xE A <− A & [0x40] We want to match the manual precisely 13 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL . and Its Corresponding Semantics ANDrl -> do ra <- getRegVal r op <- loc8ToVal l -- Loc. of 8 bits in RAM rv <- andBit ra op z <- isZero rv writeReg r rv writeCC Zero z -- CC = Condition Code branch next 14 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Back-ends ISA Concrete Execution Engine Specification DSL Custom Symbolic Execution Engines Quameleon M6800 Intermediate Weakest Precondition Language Other ISAs LLVM/KLEE Angr toolchain (Symbolic Execution, etc.) Optimizations for Analysis Abstract Interpretation 15 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Current Back-ends 1 Emulator 2 Bridge to angr • angr is a symbolic execution engine primarily for cybersecurity • Treat QIL as an ISA that angr can execute 16 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Optimizations ISA Concrete Execution Engine Specification DSL Custom Symbolic Execution Engines Quameleon M6800 Intermediate Weakest Precondition Language Other ISAs LLVM/KLEE Angr toolchain (Symbolic Execution, etc.) Optimizations for Analysis Abstract Interpretation 17 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL QIL-QIL Optimizations The goal is to facilitate analysis Constant folding I Reduce I Branch to known value code I Dead code elimination size I Inlining with simple heuristics e.g., inline everywhere Simplify I Defunctionalization CFG 18 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Dissertation Question How can we apply high-level reasoning techniques about computer programs to low-level implementations? Specifically, 1 How can we write specifications of instruction set architectures (ISAs) that enable static analysis for program verification? 2 How can we formalize and quantify the error from floating-point arithmetic in high-performance numerical programs? 19 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL 1 Introduction 2 Binary Analysis 3 A Statistical Analysis of Error in MPI Reduction Operations 4 Scalable Error Analysis for Floating-Point Program Optimization 5 Conclusion and Future Research Directions 20 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL A Statistical Analysis of Error in MPI Reduction Operations Based off previously published work with Boyana Norris [7]. 21 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL A Brief Introduction to Floating-Point Arithmetic The rest of this talk focuses on floating-point (FP) arithmetic and floating-point operations (FLOPs) Ariane V, the $500 million overflow 22 / 52 But it’s what we’re stuck with Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL We Don’t Trust Floating Point I Doesn’t map perfectly to real numbers I Can’t even represent 1/10 exactly I Complex behavior of error and exceptions [1] 23 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL We Don’t Trust Floating Point I Doesn’t map perfectly to real numbers I Can’t even represent 1/10 exactly I Complex behavior of error and exceptions But it’s what we’re stuck with [1] 23 / 52 Does this bit matter? Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Floating-Point Arithmetic Is Not Associative I Let ⊕ be floating-point addition I 0:1 ⊕ (0:2 ⊕ 0:3) = 0x1.3333333333334p-1 I (0:1 ⊕ 0:2) ⊕ 0:3 = 0x1.3333333333333p-1 I Worse error when the magnitudes are different 24 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Floating-Point Arithmetic Is Not Associative Does this bit matter? I Let ⊕ be floating-point addition I 0:1 ⊕ (0:2 ⊕ 0:3) = 0x1.3333333333334p-1 I (0:1 ⊕ 0:2) ⊕ 0:3 = 0x1.3333333333333p-1 I Worse error when the magnitudes are different 24 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Absolute vs. Relative Error Let x^ be an approximation for x. Then relative error is x^ − x x and absolute error is jx^ − xj I Think of absolute error as financial calculations; off by at most 1/10 cent (one mill) I Think of relative error as significant digits 25 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Bound on Relative Error I Let · be one of f+; −; ÷; ×} and be its corresponding floating-point operation. Then x · y = (x y)(1 + e) where jej ≤ . (1) −53 I For double-precision = 2 I This holds only for x y 6= 0 and normal (not subnormal) 26 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL Message Passing Interface (MPI) I An API for communication between computers I de facto standard for high-performance computing (HPC) I Both “too high-level and too low-level” [8] 27 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL MPI Reduce I Assume an array A of size n 0 I Reduce A to a single value 0 3 6 • e.g. MPI_SUM time I Distribute A across 0 1 2 3 4 5 6 7 8 MPI ranks (each pk ) I Unspecified but usually deterministic A A . A p0 p1 . p8 reduction order on the same topology 28 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion How many ways are there to do this reduce? I Depends on how we define acceptable reduction strategy I We list four families 1 Canonical Left-Associative (Canon) 2 Fixed Order, Random Association (FORA) 3 Random Order, Random Association (RORA) 4 Random Order, Left-Associative (ROLA) 29 / 52 Intro Binary Analysis Reduce Error FP Error Analysis Conclusion HPCL 1.