A FISH called WANDA, 2013
A FISH called WANDA
WANDA: A Measurement Tool for Forensic Document Examiners Measurement Science and Standards in Forensic Handwri ng Analysis, NIST Campus, Conference & Webcast 4./5. June 2013
Katrin Franke, Department of Computer Science and Media Technology, Gjøvik University College h p://www.nislab.no
Forensics Lab 1 A FISH called WANDA, 2013 Katrin Franke, PhD, Professor
§ Professor of Computer Science, 2010 PhD in Ar ficial Intelligence, 2005 MSc in Electrical Engineering, 1994 § Industrial Research and Development (19+ years) Financial Services and Law Enforcement Agencies § Courses, Tutorials and post-graduate Training: Law Enforcement, BSc, MSc, PhD
§ Chair IAPR/TC6 – Computa onal Forensics, 2008-2012 § IAPR Young Inves gator Award, 2009 Interna onal Associa on of Pa ern Recogni on
Forensics Lab 2 kyfranke.com A FISH called WANDA, 2013 Current Affilia on
§ Norwegian Information Security Laboratory (NISlab)
Department of Computer Science and Media Technology, Gjøvik University College, P.O. Box 191, N-2802 Gjøvik, Norway.
§
http://www.nislab.no Forensics Lab 3 KyFranke - ICDAR 2007 - Tutorial - Computational Forensics 3 A FISH called WANDA, 2013 Disclaimer
§ The following slides have been published previously, i.e.
§ Franke, K., Schomaker, L., Vuurpijl, L., Giesler, S. (2003). FISH-New: A common ground for computer-based forensic writer identification (Abstract). Forensic Science International, Volume 136(S1-S432) p. 84, Proc. 3rd European Academy of Forensic Science Triennial Meeting, Istanbul, Turkey.
§ Franke, K. (2004). Digital image processing and pattern recognition in the forensic analysis of handwriting (Abstract). In Proc. 6th International Congress of the Gesellschaft für Forensischen Schriftenuntersuchung (GFS), Heidelberg, Germany.
§ Franke, K., Rose, S. (2004). Ink-deposition model: The relation of writing and ink deposition processes. In Proc. 9th International Workshop on Frontiers in Handwriting Recognition (IWFHR), Tokyo, Japan, pp. 173-178.
Forensics Lab 4 EAFS 2003-09-23 FISH-new: A Common Ground for Writer Identification
Katrin Franke, Fraunhofer IPK, Berlin, Germany
Lambert Schomaker, Rijksuniversiteit Groningen, The Netherlands
Louis Vuurpijl, Nijmegen University, The Netherlands
Stefan Giesler, Freiburg, Germany
Slide 1 EAFS 2003-09-23 Demands in Forensic Science
• Objective measurement and classification, • Robustness and Reproducibility of the results, • Secure against Falsifications.
Initiation, Extension and Adaptation of Computer-based investigation methods
00000001000 01100001000 Method: Combining specific Knowledge from • Forensic handwriting examination, • Computer vision, and • Pattern recognition.
Slide 3 EAFS 2003-09-23
Computer-based Forensic handwriting examination
00000001000 Systems operating in forensic labs: 01100001000 • SCRIPT (NIFO/TNO, Netherlands) and • FISH (Bundeskriminalamt, Germany)
Forensic InformationSystem Handwriting Since 1988 FISH is operating in forensic labs, handwriting is: FISH Database*: • Classified by shape characteristics, 77.000 Investigation cases, • Compared with database, 17.500 Handwritten products, • Presented according recognized similarities, 32.000 Persons, • Digitally stored, and 78.000 Identifications of persons, • Managed. 86.000 Documents.
Slide 4 * (31th December 1997) EAFS 2003-09-23
Strategic Meeting: * November 28th-29th, 2001 Status quo in November 2001
Initiator: Bundeskriminalamt, Germany, • Common Standards are lacking Guests: Netherlands • Systems become outdated Forensic Institute, • Improvements can be expected, The Netherlands, If state-of-the-art methods in US Secret Service, US, pattern recognition are used FBI, US.
Fraunhofer IPK, GfaI, Germany * Project: Vergelijk - Nijmegen University, Comparison of the Script and the FISH system Rijksuniversiteit Groningen, Contracting Entity: Netherlands Forensic Institute - (2000) The Netherlands, Performed at: NICI - Nijmegen University, L. Schomaker and L. Vuurpijl
Slide 5 EAFS 2003-09-23
1997 Timeline: Fraunhofer IPK project by BKA/BMI - “ESD” Modular system for the elimination of noisy signals in documents A FISH called 2000 WANDA NICI, Nijmegen University project by NFI - “Vergelijk” Comparison of the Script and the FISH system
2001 August - Initial discussions with BKA for new FISH developments November, 28th-29th - Strategic meeting at BKA
2002 October, 1st - Official project start
2003 May, 19th – Project –result presentation at BKA
Until 2006 Continuous advancement and extension of WANDA modules
Slide 6 EAFS 2003-09-23
WANDA Team 2003
Artificial Intelligence Institute, Rijksuniversiteit Groningen
international Unipen Foundation
Slide 7 Project start: 16th September 2002 EAFS 2003-09-23
WANDA Team 2003
• Off-line and on-line handwriting analysis • Forensic signature analysis
• Motor-control theory • Pattern recognition and Image processing Artificial Intelligence Institute, Rijksuniversiteit Groningen • System engineering
• Fundamental and Applied Research
international Unipen Foundation • Industrial Development
Slide 8 Project start: 16th September 2002 EAFS 2003-09-23 Fraunhofer IPK Altug Metin Christian Taubenheim Christian Veenhuis Project Participants Katrin Franke Bundeskriminalamt Mario Köppen Werner Kuckuck Martin Peng Axel Kerhoff Nino Palavandishvili Manfred Philipp Ottmar Bünnemeyer Hartmut Gieschen Steffen Rose Tomas Kühn LKA Berlin Wolfgang Penk Gerhard Grube Xiufen Liu Reinhard Zschach
iUF / RUG Geertie Zwarts Isabelle Guyon Johan Everts Lambert Schomaker Maarten Jacobs Katrin Franke
Lambert Schomaker NICI Louis Vuurpijl Hubert Voogd Stefan Giesler Louis Vuurpijl Maartijn Beenders Slide 9 Merijn van Erp
SWE Stefan Giesler EAFS 2003-09-23 Overall idea
Standardization of the data-format, Modularization and Extensibility in system concept, Objectification in measurements and Replicability of results.
Usage of stable standards: Programming language Java on clients and general server Data interchange language XML (Extensible Markup Language).
XML everywhere: • System configuration and plug-in specification, • Data transfer protocol, • Data modeling and annotation, and • Journaling of data processing.
Slide 10 EAFS 2003-09-23 Framework concept
• User interacts via Java client
• Client is the front-end for accessing & processing information
• Information is distributed over and hosted by trusted servers
• Via their clients, users request services provided by the servers
Slide 11 EAFS 2003-09-23
Slide 12 WANDA System EAFS 2003-09-23 Plug-In Concept I
Slide 13 EAFS 2003-09-23 Plug-In Concept II
Costumer tailored system Investigation Bureaus and/or Forensic Laboratories Signature Analysis and/or Writer Identification System Local Network and/or Remote Access
Integration / Connection to systems at work
System updates and Extension to latest developments without Framework adaptation - plug-and-play
Standardization of Investigation Methods
Research Platform
Slide 14 EAFS 2003-09-23 Client desktop with Plug-In modules
Case Browser Central Document View
Slide 15 EAFS 2003-09-23 WANDA Scan and WANDA Tablet Plug-In
Slide 16 EAFS 2003-09-23 Combined processing of on- and offline data
“Simple” reconstruction of the writing sequence Direct tracing of the handwritten stroke, e.g. for measurements of loop sizes, stroke width or ink deposit Assignment of on-line and off-line handwriting characteristics Off-line specimens synthesis
K. Franke, L. Schomaker, W. Penk 2003.
Slide 17 EAFS 2003-09-23 WANDA Clean Plug-In
K. Franke & M. Köppen 1999.
Slide 18 EAFS 2003-09-23 WANDA Annotation Plug-Ins Concept for the Description of Handwriting Characteristics.
Script Writer Content Material
Slide 19 EAFS 2003-09-23 Script
Usage of established categories
Usage of unambiguous level - low - medium - high
Slide 20 EAFS 2003-09-23 Content - Verbatim
Amount of Handwriting Bridges to Linguistic Plain text for further usage
Slide 21 TOM BROKAW NBC TV 30 ROCKEFELLER PLAZA NEW YORK NY.10112 EAFS 2003-09-23 Writer
Consideration of Characteristics that may provide Clues on handwritings individuality.
Others -> Cases Management
Slide 22 EAFS 2003-09-23 Material
Writing instrument
Pen tip
Ink Paper Writing pad
Currently only verbal description !
Slide 23 EAFS 2003-09-23
XML-based data storage?!
http://www.fosterfreeman.co.uk ESDA Spectral Slide 25 EAFS 2003-09-23 WANDA Measurement Plug-In
WMP van Erp, Vuurpijl, Franke, Schomaker 2003
Slide 26 EAFS 2003-09-23 Concepts of the WMP Measurement Fish Script WMP
Ascender/Descender heights √ √ √ Corpus heights √ √ √ Letter widths √ √ √ Slant √ √ √ Upper/lower loops √ √ Line distance √ √ The WMP measures Word width √ all interactive FISH features and Isolate characters √ √ (implicit) employs the WAM Follow characters √ √ (implicit) Allograph matching √
Slide 27 EAFS 2003-09-23 How to use WMP
Interactive selection Automatic measurement
Online help for novice Hotkeys for experts
Slide 28 EAFS 2003-09-23 Diversity in letter shapes….
Within and between writer variation: Exemplars of the character ‘t’
Slide 29 EAFS 2003-09-23 …… and structure through allographs
Finding structure in diversity by use of “n-ary” hierarchical clustering Vuurpijl & Schomaker 1997
Considered handwritings of 6600 writers from Europe and the US
Slide 30 EAFS 2003-09-23 WAM: Wanda Allograph Matcher
The document examiner copy-draws the trajectory of a specific character
Use pattern recognition to match this to the database of prototypical allographs
Index (annotate) the document as containing an example of this particular allograph
Allowing for automated writer search
And uniform labeling of documents
Slide 32 EAFS 2003-09-23 How to use the WAM - Step 1
Slide 33 EAFS 2003-09-23 How to use the WAM - Step 2
Slide 34 EAFS 2003-09-23 Automatic extracted features - 2003
Schomaker, Bulacu, van Erp 2003
Slide 35 EAFS 2003-09-23 Edge-direction features
Bulacu, Schomaker, Vuurpijl 2003 Derived features
Slide 36 EAFS 2003-09-23 Results I
For a query sample, the set W will contain one matching sample of the same writer and 500 distractor samples by 250 other writers.
Schomaker, Bulacu, van Erp 2003
Slide 37 EAFS 2003-09-23 Results II
Same documents of 100 writers were processed with the two systems working in forensic labs and our currently available version
TOP1 (TOP10) System A: 34% (90%) System B: 65% (90%)
Latest results: 79% (95%) Bulacu & Schomaker 2003
Slide 38 ReferencesEAFS 2003-09-23 2004: WANDA: A common ground for forensic handwriting examination and writer identification K Franke, L Schomaker, C Veenhuis, L Vuurpijl, M van Erp, I Guyon ENFHEX news-Bulletin of the European Network of Forensic Handwriting Experts
2004: The WANDAML markup language for digital document annotation K Franke, I Guyon, L Schomaker, L Vuurpijl, Frontiers in Handwriting Recognition, 2004. IWFHR-9 2004. Ninth ...
2003, WANDA: A generic framework applied in forensic handwriting analysis and writer identification K Franke, L Schomaker, C Veenhuis, C Taubenheim, I Guyon, L Vuurpijl, M van … Proc. 3rd International Conference on Hybrid Intelligent Systems, 927-938
2003, The WANDA measurement tool for forensic document examination M Erp, LG Vuurpijl, K Franke, LRB Schomaker, Nijmegen: Nijmegen Institute for Cognition and Information
2004: Automatic writer identification using connected-component contours and edge-based features of uppercase western script L Schomaker, M Bulacu, Pattern Analysis and Machine Intelligence, IEEE Transactions on 26 (6), 787-798
2007: Text-independent writer identification and verification using textural and allographic features M Bulacu, L Schomaker, Pattern Analysis and Machine Intelligence, IEEE Transactions on 29 (4), 701-717
2005: Robotic writing trace synthesis and its application in the study of signature line quality. K Franke, LRB Schomaker, Journal of Forensic Document Examination 16
2005: The influence of physical and biomechanical processes on the ink trace: Methodological foundations for the forensic analysis of signatures SlideK Franke, 39 Rijksuniversiteit Groningen
2010: Computational forensics: An overview K Franke, SN Srihari, Computational Forensics, 1-10 2004-10-27 Ink-Deposition Model: the relation of writing and ink deposition processes
Katrin Franke, Steffen Rose
Fraunhofer Institute for Production Systems and Design Technologies SlideDepartment 1 of Pattern Recognition Berlin, Germany 2004-10-27 Challenges in Forensic Applications
u Working with residual ink traces V u Reduced writer-specific information, e.g. clues about writing behavior are highly degraded
X u Numerous physical and biomechanical effects on the ink trace
X
u Optical Inspection of the inner ink trace X X characteristics, so-called stroke morphology
Slide 2 2004-10-27 Focus on the influence of pen used
Solid ink type (Mechanical) Pencil
Fluid ink type Viscous ink type Fountain, Fine-line, Ballpoint pen Roller pen etc.
Slide 3
Franke, Bünnemeyer & Sy IWFHR’ 2002 2004-10-27 Major research questions
u What are the interactions between evoked writing instrument and writing behaviors?
u Can ink-deposition characteristics be exploited in order to recover individual characteristics of a handwritten signature?
u How can disturbing material influences be neutralized during a computer-based analysis procedure?
Slide 4 2004-10-27 Relation Writing Behaviors to Ink-Trace Characteristics
Demanded studies:
Extraction of ink traces Normalization of ink deposit Validating of ink deposit stability Modeling of relative ink intensity
Slide 5 Robotic Trace Synthesis 2004-10-27 Flat bed machine (GFV-SW) with CNC Computer Numerical Control. Processes NC-program code.
Position accuracy of 1 µm
Max. movement velocity of 12.8 m/min (214 mm/s)
Diverse pens can be plugged in the tool mounting and employed for writing. Slide 6
Electronic pen-tabled is used for Cross-validations. 2004-10-27
Writing Robot in Action
Start movie
Slide 7 Signatures 2004-10-27 written by the Robot
%! G17 G90! G0 Z-70! F0.6 ! G1 X0.566 Y8.316 Z-80.000 ! G1 X0.338 Y8.039 Z-80.000 ! F1.2 ! G1 X0.125 Y7.934 Z-80.000 ! F1.6 ! G1 X0.000 Y8.173 Z-81.000 ! F1.8 ! G1 X0.027 Y8.890 Z-81.000 ! F1.9 ! G1 X0.250 Y10.151 Z-81.000 ! G1 X3.174 Y18.506 Z-81.000 ! G1 X4.051 Y20.026 Z-81.000 ! F1.8 ! . . .! G1 X47.949 Y6.301 Z-75.000 ! F1.3 ! G1 X49.104 Y6.730 Z-75.000 ! F1.0 ! G1 X49.974 Y7.028 Z-75.000 ! SlideG0 8 Z0! G0 X0 Y0! M30!
Franke & Schomaker, Journal of Forensic Document Examination (in Press) 2004-10-27 Our Approach: Analysis by Synthesis
u Synthesize simple ink traces
à Employ different pen
à Vary “writing behavior”
u Analyze produced ink traces
à Consider general ink intensity distribution
à Inspect changes
Slide 9 2004-10-27 Synthesis of ballistic trajectories and ink traces
Slide 10 2004-10-27 Physical ink deposition processes
Solid ink type - (Mechanical) Pencil
File effect Paper fibers sticking out of the fiber-mat are colorized.
Viscous ink type - Ballpoint pen
Unrolling and squeezing effect deposits the viscous paste onto the paper surface.
Fluid ink type - Fountain, Fine-line, Roller pen etc.
Capillary effect Paper fibers are soaked with water-based ink.
Slide 11 Franke & Rose, IWFHR‘2004 2004-10-27 Variations in ink-intensity distribution
Slide 12 2004-10-27 Regressions of intensity profile plots
Ink Deposition Model Relationship of applied pen tip force and relative ink intensity Slide 13 2004-10-27 Ink Deposition Models
Gray-ragtime considers particular Ink Deposition Model + average ink-intensity range
Slide 14 2004-10-27 IDM Applications Next Steps
u Signature Processing Control Ink distribution and/or stroke width analysis
u Normalization of ink traces Relative ink distribution being pen/ink independent
u Soft-Synthesis of off-line specimen from on-line data (brushing function)
u Assignment to temporal characteristics
Slide 15 2004-10-27 Digital Densitron
Intensität 200 160 120 ?
80
40 Bildbreite Intensity profile of the whole image 0 0 20 40 60 80 100
100 Bildbreite Intensity profile of an image row Intensität 200 80 160 600
120 40
80 20 500 40
0 400 10 Number of intensity levels 300 Computation of the gray value histogram
30 200
50 100
0 Bildhöhe 0 127 255
Slide 16
Franke & Grube, JFDE‘1998 Franke, Mannheimer Hefte‘2004 2004-10-27 ISU.Samples - Densitron I
Slide 17 2004-10-27 Generated Signatures Specimens
u Genuine signatures of 10 different writers using an electronic ink pen and an electronic writing tablet 10 samples per writer = 100 synthesized signatures
u One signature sample using 24 different ball-point pens 10 samples per pen = 240 synthesized signatures
Questioned to be answered:
1. How accurate are synthesized on-line signatures ? 2. How stable are stroke morphology characteristics ?
Ball-point Pen Tip
Slide 18 2004-10-27
Studied stroke morphology characteristics
b
a u Relative Ink Distribution c d u Stroke Phenomena
a) Striations b) Ink drops, ink depots c ) Ink depositing gaps, d) Feathering
Slide 19
Source: Widmer, 1996 Interactive Labeling2004-10-27 of Stroke Phenomena using WANDA
Phenomena Client- and Server-Plug-In, which extend the functionality of the WANDA Workbench
Slide 20 2004-10-27 Super-imposed Labeled Phenomena: 1 - Ink drop
Slide 21 2004-10-27 Super-imposed Labeled Phenomena: 2 - Ink deposit
Slide 22 2004-10-27 Super-imposed Labeled Phenomena: 3 - Feathering
Slide 23 2004-10-27 Super-imposed Labeled Phenomena: 4 - Striations
Slide 24 2004-10-27 Super-imposed Labeled Phenomena: 5 - Ink gap
Slide 25 2004-10-27 Methode evaluation I - Stroke phenomena - Complete Reduced match_Result = Phenomena set Phenomena set correct_pheno_labels / ( Fusion of feathering, total_pheno_lables * 100 striation and ink gap ) Intra-group Allowed Tolerances: Comparison 10° Rotation ( for reach ball-point pen 96.0 % 96.7 % 10 Pixel Translation separately - 10 samples ) for 300 dpi scan Inter-group Tolerance due to Comparison 91.8% 94.7 % Human Labeling ( cross-validation over all 5 Pixel Translation 240 samples ) Validation with
Slide 26 Ideal reference 94.6 % 96.5 % pattern 2004-10-27 Method evaluation II - Ink Distribution -
26 Ballpoint pen, 10 Samples each
Slide 27 2004-10-27 Lessons Learned
u The Influence of Physical and Biomechanical Processes on the Ink Trace - Methodological foundations for the forensic analysis of signatures.
u University of Groningen, Artificial Intelligence Institute, Faculty of Mathematics and Natural Sciences, The Netherlands, 2005.
u Online available via http://www.kyfranke.com
u 255 pages, 118 figures, 31 tables, 321 references
Slide 28 A FISH called WANDA, 2013 Current Affilia on
§ Norwegian Information Security Laboratory (NISlab)
Department of Computer Science and Media Technology, Gjøvik University College, P.O. Box 191, N-2802 Gjøvik, Norway.
§
http://www.nislab.no Forensics Lab 5 KyFranke - ICDAR 2007 - Tutorial - Computational Forensics 5 A FISH called WANDA, 2013 NISlab – Working Areas • Biometrics – User Authen ca on • Forensics – BTA Protocol – Forensic Readiness – Incidence Response – Inves ga on/Analysis
• Security Management • Security Technology – Risk-based Design – So ware Security – Security Economics – System Administra on – System/Adversary Modeling – Network and Cri cal – Human Factors, Policies Infrastructure Protec on
Testimon (lat. evidence) Computational & Digital Forensics: Forensics LabFraud Detection, Analysis and Prevention 6 A FISH called WANDA, 2013
Computa onal Forensics:
Adding Efficiency and Intelligence to BIG DATA Inves ga on
Forensics Lab 7 A FISH called WANDA, 2013
Thank you for your considera on of comments!
Ge ng in touch WWW: kyfranke.com Email: [email protected] Skype/gTalk: kyfranke Phone: +47 61 135 254
Forensics Lab 8