DOCSLIB.ORG

E-Discovery Processing Primer

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
E-Discovery Processing Primer 1 Processing in E-Discovery, a Primer Craig Ball1 © 2019 All Rights Reserved Table of Contents Table of Contents .......................................................................................................................... 2 Introduction .................................................................................................................................. 4 Why process ESI in e-discovery? Isn’t it “review ready?” ......................................................... 5 Files ............................................................................................................................................... 6 A Bit About and a Byte Out of Files ............................................................................................ 6 Digital Encoding ......................................................................................................................... 7 Decimal and Binary: Base 10 and Base Two ........................................................................... 7 Bits ......................................................................................................................................... 7 Bytes ...................................................................................................................................... 8 The Magic Decoder Ring called ASCII ..................................................................................... 9 Unicode ................................................................................................................................ 12 Mind the Gap! ...................................................................................................................... 13 Hex ....................................................................................................................................... 13 Base64 ................................................................................................................................. 14 Why Care about Encoding? .................................................................................................. 15 Hypothetical: TaggedyAnn ................................................................................................... 16 File Type Identification ............................................................................................................ 16 File Extensions ..................................................................................................................... 17 Binary File Signatures ........................................................................................................... 17 File Structure ....................................................................................................................... 18 Data Compression ................................................................................................................... 19 Identification on Ingestion .......................................................................................................... 21 1 Adjunct Professor, E-Discovery & Digital Evidence, University of Texas School of Law, Austin, Texas. The author acknowledges the contributions of Diana Ball as editor and of David Sitsky of Nuix for his insights into processing. 2 Media (MIME) Type Detection ................................................................................................ 22 Media Type Tree Structure .................................................................................................. 22 When All Else Fails: Octet Streams and Text Stripping ............................................................ 24 Data Extraction and Document Filters ........................................................................................ 24 Recursion and Embedded Object Extraction ............................................................................... 27 Simple Document................................................................................................................. 28 Structured Document .......................................................................................................... 28 Compound Document .......................................................................................................... 28 Simple Container ................................................................................................................. 29 Container with Text ............................................................................................................. 29 Family Tracking and Unitization: Keeping Up with the Parts ....................................................... 29 Exceptions Reporting: Keeping Track of Failures ......................................................................... 29 Lexical Preprocessing of Extracted Text ...................................................................................... 30 Normalization .......................................................................................................................... 30 Character Normalization ......................................................................................................... 30 Unicode Normalization ............................................................................................................ 30 Diacritical Normalization ......................................................................................................... 31 Case Normalization.................................................................................................................. 31 Impact of Normalization on Discovery Outcomes ................................................................... 32 Time Zone Normalization ............................................................................................................ 33 Parsing and Tokenization ............................................................................................................ 33 Building a Database and Concordance Index .............................................................................. 37 The Database ........................................................................................................................... 37 The Concordance Index ........................................................................................................... 38 Culling and Selecting the Dataset ................................................................................................ 39 Immaterial Item Suppression .................................................................................................. 39 De-NISTing ............................................................................................................................... 40 Cryptographic Hashing: ........................................................................................................... 40 Deduplication .......................................................................................................................... 45 Near-Deduplication .............................................................................................................. 45 Office 365 Deduplication ..................................................................................................... 49 3 Relativity E-Mail Deduplication ............................................................................................ 50 Other Processing Tasks ............................................................................................................... 52 Foreign Language Detection .................................................................................................... 52 Entropy Testing ....................................................................................................................... 52 Decryption ............................................................................................................................... 52 Bad Extension Flagging ............................................................................................................ 52 Color Detection ....................................................................................................................... 52 Hidden Content Flagging ......................................................................................................... 52 N-Gram and Shingle Generation .............................................................................................. 53 Optical Character Recognition (OCR) ....................................................................................... 53 Virus Scanning ......................................................................................................................... 53 Production Processing ................................................................................................................. 53 Illustration 1: E-Discovery Processing Model ............................................................................. 53 Illustration 2: Anatomy of a Word DOCX File .............................................................................. 55 Introduction Talk to lawyers about e-discovery processing and you’ll likely get a blank stare suggesting no clue what you’re talking about. Why would lawyers want anything to do with something so disagreeably technical? Indeed, processing is technical and strikes attorneys as something they need not know. That’s lamentable because processing
Load more

Recommended publications
  • Hieroglyphs for the Information Age: Images As a Replacement for Characters for Languages Not Written in the Latin-1 Alphabet Akira Hasegawa
    Rochester Institute of Technology RIT Scholar Works Theses Thesis/Dissertation Collections 5-1-1999 Hieroglyphs for the information age: Images as a replacement for characters for languages not written in the Latin-1 alphabet Akira Hasegawa Follow this and additional works at: http://scholarworks.rit.edu/theses Recommended Citation Hasegawa, Akira, "Hieroglyphs for the information age: Images as a replacement for characters for languages not written in the Latin-1 alphabet" (1999). Thesis. Rochester Institute of Technology. Accessed from This Thesis is brought to you for free and open access by the Thesis/Dissertation Collections at RIT Scholar Works. It has been accepted for inclusion in Theses by an authorized administrator of RIT Scholar Works. For more information, please contact [email protected]. Hieroglyphs for the Information Age: Images as a Replacement for Characters for Languages not Written in the Latin- 1 Alphabet by Akira Hasegawa A thesis project submitted in partial fulfillment of the requirements for the degree of Master of Science in the School of Printing Management and Sciences in the College of Imaging Arts and Sciences of the Rochester Institute ofTechnology May, 1999 Thesis Advisor: Professor Frank Romano School of Printing Management and Sciences Rochester Institute ofTechnology Rochester, New York Certificate ofApproval Master's Thesis This is to certify that the Master's Thesis of Akira Hasegawa With a major in Graphic Arts Publishing has been approved by the Thesis Committee as satisfactory for the thesis requirement for the Master ofScience degree at the convocation of May 1999 Thesis Committee: Frank Romano Thesis Advisor Marie Freckleton Gr:lduate Program Coordinator C.
    [Show full text]
  • Unicode Ate My Brain
    UNICODE ATE MY BRAIN John Cowan Reuters Health Information Copyright 2001-04 John Cowan under GNU GPL 1 Copyright • Copyright © 2001 John Cowan • Licensed under the GNU General Public License • ABSOLUTELY NO WARRANTIES; USE AT YOUR OWN RISK • Portions written by Tim Bray; used by permission • Title devised by Smarasderagd; used by permission • Black and white for readability Copyright 2001-04 John Cowan under GNU GPL 2 Abstract Unicode, the universal character set, is one of the foundation technologies of XML. However, it is not as widely understood as it should be, because of the unavoidable complexity of handling all of the world's writing systems, even in a fairly uniform way. This tutorial will provide the basics about using Unicode and XML to save lots of money and achieve world domination at the same time. Copyright 2001-04 John Cowan under GNU GPL 3 Roadmap • Brief introduction (4 slides) • Before Unicode (16 slides) • The Unicode Standard (25 slides) • Encodings (11 slides) • XML (10 slides) • The Programmer's View (27 slides) • Points to Remember (1 slide) Copyright 2001-04 John Cowan under GNU GPL 4 How Many Different Characters? a A à á â ã ä å ā ă ą a a a a a a a a a a a Copyright 2001-04 John Cowan under GNU GPL 5 How Computers Do Text • Characters in computer storage are represented by “small” numbers • The numbers use a small number of bits: from 6 (BCD) to 21 (Unicode) to 32 (wchar_t on some Unix boxes) • Design choices: – Which numbers encode which characters – How to pack the numbers into bytes Copyright 2001-04 John Cowan under GNU GPL 6 Where Does XML Come In? • XML is a textual data format • XML software is required to handle all commercially important characters in the world; a promise to “handle XML” implies a promise to be international • Applications can do what they want; monolingual applications can mostly ignore internationalization Copyright 2001-04 John Cowan under GNU GPL 7 $$$ £££ ¥¥¥ • Extra cost of building-in internationalization to a new computer application: about 20% (assuming XML and Unicode).
    [Show full text]
  • Character Set Migration Best Practices For
    Character Set Migration Best Practices $Q2UDFOH:KLWH3DSHU October 2002 Server Globalization Technology Oracle Corporation Introduction - Database Character Set Migration Migrating from one database character set to another requires proper strategy and tools. This paper outlines the best practices for database character set migration that has been utilized on behalf of hundreds of customers successfully. Following these methods will help determine what strategies are best suited for your environment and will help minimize risk and downtime. This paper also highlights migration to Unicode. Many customers today are finding Unicode to be essential to supporting their global businesses. Oracle provides consulting services for very large or complex environments to help minimize the downtime while maximizing the safe migration of business critical data. Why migrate? Database character set migration often occurs from a requirement to support new languages. As companies internationalize their operations and expand services to customers all around the world, they find the need to support data storage of more World languages than are available within their existing database character set. Historically, many legacy systems required support for only one or possibly a few languages; therefore, the original character set chosen had a limited repertoire of characters that could be supported. For example, in America a 7-bit character set called ASCII is satisfactory for supporting English data exclusively. While in Europe a variety of 8 bit European character sets can support specific subsets of European languages together with English. In Asia, multi byte character sets that could support a given Asian language and English were chosen. These were reasonable choices that fulfilled the initial requirements and provided the best combination of economy and performance.
    [Show full text]
  • Plain Text & Character Encoding
    Journal of eScience Librarianship Volume 10 Issue 3 Data Curation in Practice Article 12 2021-08-11 Plain Text & Character Encoding: A Primer for Data Curators Seth Erickson Pennsylvania State University Let us know how access to this document benefits ou.y Follow this and additional works at: https://escholarship.umassmed.edu/jeslib Part of the Scholarly Communication Commons, and the Scholarly Publishing Commons Repository Citation Erickson S. Plain Text & Character Encoding: A Primer for Data Curators. Journal of eScience Librarianship 2021;10(3): e1211. https://doi.org/10.7191/jeslib.2021.1211. Retrieved from https://escholarship.umassmed.edu/jeslib/vol10/iss3/12 Creative Commons License This work is licensed under a Creative Commons Attribution 4.0 License. This material is brought to you by eScholarship@UMMS. It has been accepted for inclusion in Journal of eScience Librarianship by an authorized administrator of eScholarship@UMMS. For more information, please contact [email protected]. ISSN 2161-3974 JeSLIB 2021; 10(3): e1211 https://doi.org/10.7191/jeslib.2021.1211 Full-Length Paper Plain Text & Character Encoding: A Primer for Data Curators Seth Erickson The Pennsylvania State University, University Park, PA, USA Abstract Plain text data consists of a sequence of encoded characters or “code points” from a given standard such as the Unicode Standard. Some of the most common file formats for digital data used in eScience (CSV, XML, and JSON, for example) are built atop plain text standards. Plain text representations of digital data are often preferred because plain text formats are relatively stable, and they facilitate reuse and interoperability.
    [Show full text]
  • Windows NLS Considerations Version 2.1
    Windows NLS Considerations version 2.1 Radoslav Rusinov [email protected] Windows NLS Considerations Contents 1. Introduction ............................................................................................................................................... 3 1.1. Windows and Code Pages .................................................................................................................... 3 1.2. CharacterSet ........................................................................................................................................ 3 1.3. Encoding Scheme ................................................................................................................................ 3 1.4. Fonts ................................................................................................................................................... 4 1.5. So Why Are There Different Charactersets? ........................................................................................ 4 1.6. What are the Difference Between 7 bit, 8 bit and Unicode Charactersets? ........................................... 4 2. NLS_LANG .............................................................................................................................................. 4 2.1. Setting the Character Set in NLS_LANG ............................................................................................ 4 2.2. Where is the Character Conversion Done? .........................................................................................
    [Show full text]
  • UTF-8 from Wikipedia, the Free Encyclopedia
    UTF-8 From Wikipedia, the free encyclopedia UTF-8 is a character encoding capable of encoding all possible characters, or code points, defined by Unicode and originally designed by Ken Thompson and Rob Pike.[1] The encoding is variable-length and uses 8-bit code units. It was designed for backward compatibility with ASCII and to avoid the complications of endianness and byte order marks in the alternative UTF-16 and UTF-32 encodings. The name is derived from Unicode (or Universal Coded Character Set) Transformation Format – 8- bit.[2] UTF-8 is the dominant character encoding for the World Wide Web, accounting for 89.1% of all Web pages in May 2017 (the most popular East Asian encodings, Shift JIS and GB 2312, have 0.9% and 0.7% respectively).[4][5][3] The Internet Mail Consortium (IMC) recommended that all e-mail programs be able to display and create mail using UTF-8,[6] and the W3C recommends UTF-8 as the default encoding in XML and HTML.[7] UTF-8 encodes each of the 1,112,064[8] valid code points in Unicode using one to four 8-bit bytes.[9] Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. The first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single octet with the same binary value as ASCII, so that valid ASCII text is valid UTF-8-encoded Unicode as well. Since ASCII bytes do not occur when encoding non-ASCII code points into UTF-8, UTF-8 is safe to use within most programming and document languages that interpret certain ASCII characters in a special way, such as '/' in filenames, '\' in escape sequences, and '%' in printf.
    [Show full text]
  • [MS-VUVP-Diff]: VT-UTF8 and VT100+ Protocols
    [MS-VUVP-Diff]: VT-UTF8 and VT100+ Protocols Intellectual Property Rights Notice for Open Specifications Documentation . Technical Documentation. Microsoft publishes Open Specifications documentation (“this documentation”) for protocols, file formats, data portability, computer languages, and standards as well as overviews of the interaction among each of these technologiessupport. Additionally, overview documents cover inter-protocol relationships and interactions. Copyrights. This documentation is covered by Microsoft copyrights. Regardless of any other terms that are contained in the terms of use for the Microsoft website that hosts this documentation, you maycan make copies of it in order to develop implementations of the technologies that are described in the Open Specifications this documentation and maycan distribute portions of it in your implementations usingthat use these technologies or in your documentation as necessary to properly document the implementation. You maycan also distribute in your implementation, with or without modification, any schema, IDL'sschemas, IDLs, or code samples that are included in the documentation. This permission also applies to any documents that are referenced in the Open Specifications. documentation. No Trade Secrets. Microsoft does not claim any trade secret rights in this documentation. Patents. Microsoft has patents that maymight cover your implementations of the technologies described in the Open Specifications. documentation. Neither this notice nor Microsoft's delivery of thethis documentation grants any licenses under those patents or any other Microsoft patents. However, a given Open Specification maySpecifications document might be covered by the Microsoft Open Specifications Promise or the Microsoft Community Promise. If you would prefer a written license, or if the technologies described in the Open Specificationsthis documentation are not covered by the Open Specifications Promise or Community Promise, as applicable, patent licenses are available by contacting [email protected].
    [Show full text]
  • Electronic Document Preparation Pocket Primer
    Electronic Document Preparation Pocket Primer Vít Novotný December 4, 2018 Creative Commons Attribution 3.0 Unported (cc by 3.0) Contents Introduction 1 1 Writing 3 1.1 Text Processing 4 1.1.1 Character Encoding 4 1.1.2 Text Input 12 1.1.3 Text Editors 13 1.1.4 Interactive Document Preparation Systems 13 1.1.5 Regular Expressions 14 1.2 Version Control 17 2 Markup 21 2.1 Meta Markup Languages 22 2.1.1 The General Markup Language 22 2.1.2 The Extensible Markup Language 23 2.2 Markup on the World Wide Web 28 2.2.1 The Hypertext Markup Language 28 2.2.2 The Extensible Hypertext Markup Language 29 2.2.3 The Semantic Web and Linked Data 31 2.3 Document Preparation Systems 32 2.3.1 Batch-oriented Systems 35 2.3.2 Interactive Systems 36 2.4 Lightweight Markup Languages 39 3 Design 41 3.1 Fonts 41 3.2 Structural Elements 42 3.2.1 Paragraphs and Stanzas 42 iv CONTENTS 3.2.2 Headings 45 3.2.3 Tables and Lists 46 3.2.4 Notes 46 3.2.5 Quotations 47 3.3 Page Layout 48 3.4 Color 48 3.4.1 Theory 48 3.4.2 Schemes 51 Bibliography 53 Acronyms 61 Index 65 Introduction With the advent of the digital age, typesetting has become available to virtually anyone equipped with a personal computer. Beautiful text documents can now be crafted using free and consumer-grade software, which often obviates the need for the involvement of a professional designer and typesetter.
    [Show full text]
  • Electronic Discovery Workbook
    Computer Metadata File Electronic Discovery Workbook 2016 University of Texas School of Law LAW335E: E-Discovery and Digital Evidence Fall 2016 Ver. 16.0818 © Craig Ball, All Rights Reserved Name: Course Workbook Reading Assignments NOTE: This table is for your convenience; but, the timing and scope of your responsibilities in this course are established by the latest Syllabus, not this table. Always go by the Syllabus!! The following Workbook exercises and readings should be completed prior to the start of class on the dates set out for same below. There will always be additional readings on Canvas. Monday, August 29 Read pp. 4-84; Complete Exercise 1 Monday, September 12 Read pp. 85-101; Complete Exercise 2 Monday, September 19 (No class meeting this date) Read pp. 102-175; Complete Exercises 3-8 Monday, September 26 Read pp. 176-234; Complete Exercises 9-12 Monday, October 3 Read pp. 235-283, Complete Exercise 13 Monday, October 10 Read pp. 284-327; Complete Exercise 14, Part 1, pp. 282 Monday, October 17 No Workbook selections Monday, October 24 Read pp. 328-351; Complete Exercise 15 Monday, October 31 Read pp. 352-385; Complete Exercises 16-17 Monday, November 7 Read pp. 386-402; Begin Exercise 18 1 Contents Goals for this Workbook ...................................................................................................... 4 Introduction to Electronic Discovery and Digital Evidence .................................................... 6 Introduction to Discovery in U.S. Civil Litigation ................................................................... 9 The “E-Discovery Rules” (1,16,26,34 & 45) of the Federal Rules of Civil Procedure With Committee Notes accompanying 2006 and 2015 Amendments .......................................... 14 TRCP Rule 196.4 Electronic or Magnetic Data (enacted 1999) ...........................................
    [Show full text]
  • Junk Characters in Bb Annotate for Several Non-English Languages
    Junk Characters in Bb Annotate for Several non-English Languages Date Published: Jul 31,2020 Category: Planned_First_Fix_Release:Learn_9_1_3900_0_Release,SaaS_v3800_15_0; Product:Grade_Center_Learn,Language_Packs_Learn; Version:Learn_9_1_Q4_2019,Learn_9_1_Q2_2019,SaaS Article No.: 000060296 Product: Blackboard Learn Release: 9.1;SaaS Service Pack(s): Learn 9.1 Q4 2019 (3800.0.0), Learn 9.1 Q2 2019 (3700.0.0), SaaS Description: Incorrect or non-textual font symbols such as §, © and ¶ appeared in the Blackboard Annotate User Interface when using several non-English Language Packs, including Arabic, Spanish, Korean, and Japanese. Steps to Replicate: Prerequisite: The Learn environment has converted to Blackboard Annotate. 1. Log into Blackboard Learn as System Administrator 2. Set the Language Pack to a non-English language, such as Arabic, Spanish, Korean, or Japanese 3. Log in as Instructor 4. Navigate to a Course with Assignments 5. Grade any assignment using Blackboard Annotate Expected Behavior: The user interface displays proper characters for the language chosen. Observed Behavior: Symbols such as §, © and ¶, or characters from other languages appear. Symptoms: Incorrect characters appear in the Blackboard Annotate User Interface. Cause: Characters consist of one or more binary bytes indicating a location in a 'codepage' for a specific character encoding, such as CP252 for Arabic. Information regarding the encoding used needs to be sent by the server to the browser for it to use the correct codepage. If an incorrect codepage is used to look up the characters to be displayed, unintelligble characters known as "Mojibake" will appear because the locations in one codepage will not will not necessarily contain the same characters as another.
    [Show full text]
  • Unicode Identifiers and Reflection
    Unicode Identifiers And Reflection D1953R0 Reply to: [email protected] ​ Audience: SG-7, SG-15 Abstract SG-16 members are looking at extending the basic character set to support Unicode Identifiers. SG-7 is designing tools to convert identifiers to string (as well as the reverse). Therefore it will be necessary to be able to reflect (and reifere) on identifiers containing characters outside of the basic character sets. We explore solutions Unicode Identifiers Extending the basic character set is an area of ongoing research, but the general direction is: ● Based on TR31 ​ ● Specified Normalization of identifiers at compile time (more likely NFC) - to ensure consistent behavior (and mangling) across translation units and implementations. ● Limited to (assumed) UTF-encoded files, because no one wants mojibake in their identifiers The general motivation is not to encourage Unicode characters in identifiers but to ensure a consistent, reliable behavior across platforms. However, the goal of that paper is not specified how Unicode identifiers should work but rather to open a discussion as to how they should be reflected upon. C++ Text Model Primer For people not familiar with the work of SG-16, here is briefly how C++ handle text ● Each token is converted from the “source character encoding” (which is determined by the compiler in an implementation-defined way - GCC and Clang assumes UTF-8 by default while MSVC uses UTF BOMs and user locale to determine the “source character encoding” - Both GCC and MSVC provides flags to let their user override that behavior) ● To the internal character encoding, which is not specified but implied to be a Unicode encoding ● String literals are further converted to the _execution encoding_ whose character set is a subset of the internal character set.
    [Show full text]
  • Windows Mbox Viewer User Manual 1.0.2.6 Table of Contents 1 Modification History
    Windows MBox Viewer User Manual 1.0.2.6 Table of Contents 1 Modification History.......................................................................................................................3 2 Feedback..........................................................................................................................................3 3 Overview.........................................................................................................................................4 4 Installation.......................................................................................................................................4 5 Running the MBox viewer..............................................................................................................4 5.1 Argument List Summary..............................................................................................................4 5.2 Setting Options from GUI............................................................................................................5 5.3 Basic Use Case.............................................................................................................................6 5.4 Mail Context Menu.......................................................................................................................7 5.5 Mail Archive Context menu.........................................................................................................8 5.6 Mail Attachments..........................................................................................................................9
    [Show full text]