Integration and Component-Based Software Testing Learning

Learning objectives

•! Understand the purpose of integration testing –! Distinguish typical integration faults from faults that Integration and Component-based should be eliminated in unit testing –! Understand the nature of integration faults and how Software Testing to prevent as well as detect them •! Understand strategies for ordering construction and testing –! Approaches to incremental assembly and testing to reduce effort and control risk •! Understand special challenges and approaches for testing component-based systems

What is integration testing? Integration versus Unit Testing

Module test Integration test System test •! Unit (module) testing is a necessary foundation Specification: Module Interface specs, module Requirements –! Unit level has maximum controllability and visibility interface breakdown specification –! Integration testing can never compensate for Visible structure: Coding details Modular structure — none — inadequate unit testing (software architecture) •! Integration testing may serve as a process check –! If module faults are revealed in integration testing, Scaffolding Some Often extensive Some they signal inadequate unit testing required: –! If integration faults occur in interfaces between Looking for faults Modules Interactions, System correctly implemented modules, the errors can be in: compatibility functionality traced to module breakdown and interface specifications

(c) 2007 Mauro Pezzè & Michal Young Ch 21, slide 3 (c) 2007 Mauro Pezzè & Michal Young Ch 21, slide 4 Integration Faults Example: A Memory Leak •! Inconsistent interpretation of parameters or values –! Example: Mixed units (meters/yards) in Martian Lander Apache web server, version 2.0.48 •! Violations of value domains, capacity, or size limits Response to normal page request on secure (https) port –! Example: Buffer overflow •! Side effects on parameters or resources static void ssl io filter disable(ap filter t *f) –! Example: Conflict on (unspecified) temporary file •! Omitted or misunderstood functionality { bio filter in ctx t *inctx = f->ctx; –! Example: Inconsistent interpretation of web hits No obvious error, but •! Nonfunctional properties inctx->ssl = NULL; Apache leaked memory –! Example: Unanticipated performance issues inctx->filter ctx->pssl =slowly NULL; (in normal use) or •! Dynamic mismatches } quickly (if exploited for a –! Example: Incompatible polymorphic method calls DOS attack)

Example: A Memory Leak Example: A Memory Leak

Apache web server, version 2.0.48 Apache web server, version 2.0.48 Response to normal page request on secure (https) port Response to normal page request on secure (https) port static void ssl io filter disable(ap filter t *f) static void ssl io filter disable(ap filter t *f) { bio filter in ctx t *inctx = f->ctx; { bio filter in ctx t *inctx = f->ctx; SSL_free(inctx -> ssl); SSL_free(inctx -> ssl); The missing code is for a Almost impossible to find inctx->ssl = NULL; structure defined and inctx->ssl = NULL; with unit testing. inctx->filter ctx->pssl =created NULL; elsewhere, inctx->filter ctx->pssl =(Inspection NULL; and some } accessed through an } dynamic techniques could opaque pointer. have found it.)

•! Yes, I implemented •! Yes, I implemented •! I didn’t think at all !module A", but I !module A", but I about the strategy didn’t test it didn’t test it for testing. I didn’t thoroughly yet. It thoroughly yet. It design !module A" for will be tested along will be tested along testability and I with !module A" when with !module A" when didn’t think about that’s ready. that’s ready. the best order to build and test modules !A" and !B".

Integration Plan + Test Plan Big Bang Integration Test

An extreme and desperate approach: •! Integration test Test only after integrating all modules

... plan drives and is driven by the ... +!Does not require scaffolding project “build •!The only excuse, and a bad one Build Plan Test Plan plan” -! Minimum observability, diagnosability, efficacy, –! A key feature of the ... system architecture feedback System Architecture and project plan -! High cost of repair •! Recall: Cost of repairing a fault rises as a function of time between error and repair

•! Structural orientation: Top Modules constructed, integrated and tested based on a hierarchical project structure –! Top-down, Bottom-up, Sandwich, Backbone •! Functional orientation: stub A stub B stub C Modules integrated according to application characteristics or features Working from the top level (in terms of “use” –! Threads, Critical module or “include” relation) toward the bottom. No drivers required if program tested from top-level interface (e.g. GUI, CLI, web app, etc.)

Top down .. Top down ...

Top Top

A stub B stub C A B C

Write stubs of called or As modules replace used modules at each stubs, more stub X stub Y step in construction stub X stub Y functionality is testable

Top

Starting at the leaves of the A Driver B C “uses” hierarchy, we never need stubs ... until the program is complete, and all X Y functionality can be X tested

Bottom Up .. Bottom Up ...

Driver

Driver A Driver ... but we must ... an intermediate construct drivers for module replaces a each module (as in driver, and needs its unit testing) ... own driver ...

X Y X Y

Driver Driver Driver Driver Driver

A B A B C

... so we may have several working X Y X Y subsystems ...

Bottom Up (complete) Sandwich .

Top Top (parts)

A B C Stub C

... that are eventually Working from the extremes (top and X Y integrated into a Y single system. bottom) toward center, we may use fewer drivers and stubs

Top (more) Top

A C A C

A “thread” is a portion of several Sandwich integration modules that together provide a is flexible and user-visible program feature. X Y adaptable, but X complex to plan

Thread ... Thread ...

Top Top

A B C A B C

Integrating one As in sandwich thread, then another, integration testing, we X Y etc., we maximize X Y can minimize stubs visibility for the user and drivers, but the integration plan may be complex (c) 2007 Mauro Pezzè & Michal Young Ch 21, slide 28 (c) 2007 Mauro Pezzè & Michal Young Ch 21, slide 29 Critical Modules Choosing a Strategy

•! Strategy: Start with riskiest modules •! Functional strategies require more planning –! Risk assessment is necessary first step –! Structural strategies (bottom up, top down, –! May include technical risks (is X feasible?), process sandwich) are simpler risks (is schedule for X realistic?), other risks –! But thread and critical modules testing provide better process visibility, especially in complex •! May resemble thread or sandwich process in systems tactics for flexible build order •! Possible to combine –! E.g., constructing parts of one module to test –! Top-down, bottom-up, or sandwich are reasonable functionality in another for relatively small components and subsystems •! Key point is risk-oriented process –! Combinations of thread and critical modules –! Integration testing as a risk-reduction activity, integration testing are often preferred for larger subsystems designed to deliver any bad news as early as possible

Working Definition of Component Components — Related Concepts

•! Reusable unit of deployment and composition •! Framework –! Deployed and integrated multiple times •!Skeleton or micro-architecture of an application –! Integrated by different teams (usually) •!May be packaged and reused as a component, with “hooks” or “slots” in the interface contract •!Component producer is distinct from component user •! Design patterns •! Characterized by an interface or contract •!Logical design fragments •!Describes access points, parameters, and all functional and •!Frameworks often implement patterns, but patterns are not non-functional behavior and conditions for using the frameworks. Frameworks are concrete, patterns are component abstract •!No other access (e.g., source code) is usually available •! Component-based system •! Often larger grain than objects or packages •!A system composed primarily by assembling components, –! Example: A complete database system may be a often “Commercial off-the-shelf” (COTS) components component •!Usually includes application-specific “glue code”

•! Application programming interface (API) is •! The component builder’s challenge: distinct from implementation –! Impossible to know all the ways a component may be –! Example: DOM interface for XML is distinct from used many possible implementations, from different –! Difficult to recognize and specify all potentially sources important properties and dependencies •! Interface includes everything that must be known to use the component •! The component user’s challenge: –! More than just method signatures, exceptions, etc –! No visibility “inside” the component –! May include non-functional characteristics like –! Often difficult to judge suitability for a particular performance, capacity, security use and context –! May include dependence on other components

Testing a Component: Producer View Testing a Component: User View

•! First: Thorough unit and subsystem testing •! Not primarily to find faults in the component –! Includes thorough functional testing based on •! Major question: Is the component suitable for application program interface (API) this application? –! Rule of thumb: Reusable component requires at least –! Primary risk is not fitting the application context: twice the effort in design, implementation, and •!Unanticipated dependence or interactions with environment testing as a subsystem constructed for a single use •!Performance or capacity limits (often more) •!Missing functionality, misunderstood API •! Second: Thorough acceptance testing –! Risk high when using component for first time –! Based on scenarios of expected use •! Reducing risk: Trial integration early –! Includes stress and capacity testing –! Often worthwhile to build driver to test model •!Find and document the limits of applicability scenarios, long before actual integration

Application •! Integration testing focuses on interactions –! Must be built on foundation of thorough unit testing –! Integration faults often traceable to incomplete or Adaptor misunderstood interface specifications •!Prefer prevention to detection, and make detection easier by imposing design constraints Component •! Strategies tied to project build order –! Order construction, integration, and testing to reduce cost or risk •! Applications often access components through •! Reusable components require special care an adaptor, which can also be used by a test –! For component builder, and for component user driver