DOCSLIB.ORG

Big Data Conversion Techniques Including Their Main Features and Characteristics 2017 Edition Main Title Main 20 17 Edition

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Big Data Conversion Techniques Including Their Main Features and Characteristics 2017 Edition Main Title Main 20 17 Edition Big data conversion techniques including their main features and characteristics 2017 edition Main title 20 17 edition 17 STATISTICAL WORKING PAPERSBOOKS Big data conversion techniques including their main features and characteristics 2017 edition Manuscript completed in July 2017 Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use that might be made of the following information. Luxembourg: Publications Office of the European Union, 2017 © European Union, 2017 Reuse is authorised provided the source is acknowledged. The reuse policy of European Commission documents is regulated by Decision 2011/833/EU (OJ L 330, 14.12.2011, p. 39). Copyright for photographs: © Shutterstock/Rawpixel.com For any use or reproduction of photos or other material that is not under the EU copyright, permission must be sought directly from the copyright holders. For more information, please consult: http://ec.europa.eu/eurostat/about/policies/copyright The information and views set out in this publication are those of the authors and do not necessarily reflect the official opinion of the European Union. Neither the European Union institutions and bodies nor any person acting on their behalf may be held responsible for the use which may be made of the information contained therein. PDF ISBN 978-92-79-70523-6 ISSN 2315-0807 doi:10.2785/461700 KS-TC-17-003-EN-N Abstract Abstract Big data have high potential for nowcasting and forecasting economic variables. However, they are often unstructured so that there is a need to transform them into a limited number of time series which efficiently summarise the relevant information for nowcasting or short term forecasting the economic indicators of interest. Data structuring and conversion is a difficult task, as the researcher is called to translate the unstructured data and summarise them into a format which is both meaningful and informative for the nowcasting exercise. In this paper we consider techniques to convert unstructured big data to structured time series suitable for nowcasting purposes. We also include several empirical examples which illustrate the potential of big data in economics. Finally, we provide a practical application based on textual data analysis, where we exploit a huge set of about 3 million news articles for the construction of an economic uncertainty indicator. Keywords: Big Data, Unstructured Data, Time Series Conversion, Data Features. Acknowledgement: This work has been carried out by George Kapetanios∗, Massimiliano Marcellino† and Fotis Papailias‡ for Eurostat under a contract with GOPA. The Eurostat project manager was Dario Buono∗∗. ∗ [email protected] † [email protected] ‡ [email protected] ∗∗[email protected] Big Data Conversion Techniques including their Main Features and Characteristics 3 Table of contents Table of contents Abstract ........................................................................................................................................................ 3 1 Introduction ................................................................................................................................. 8 2 Literature review ........................................................................................................................9 2.1 Data mapping ............................................................................................................................... 9 2.2 Recent research papers ...........................................................................................................10 2.2.1 Electronic payments data ...................................................................................................... 10 2.2.2 Mobile phone usage data ......................................................................................................11 2.2.3 Sensor data ........................................................................................................................... 12 2.2.4 Satellite images data ............................................................................................................. 13 2.2.5 Price data .............................................................................................................................. 14 2.2.6 Textual data .......................................................................................................................... 15 3 A general data conversion framework for unstructured numerical Big Data ..................................................................................................................16 3.1 Conceptual setting .....................................................................................................................16 3.2 Aggregation ................................................................................................................................. 17 3.3 Features extraction ....................................................................................................................18 3.4 Data mining ................................................................................................................................. 18 3.4.1 Random subsampling ...........................................................................................................21 4 Empirical examples ................................................................................................................26 4.1 Financial markets data .............................................................................................................26 4.2 Mobile phone usage data .........................................................................................................28 4.3 Sensor data ................................................................................................................................. 32 4.4 Online prices data ......................................................................................................................36 4.5 Online search data ....................................................................................................................38 4.6 Social media................................................................................................................................ 41 5 From Reuters News data to uncertainty indexes ......................................................... 44 5.1 Obtaining the data .....................................................................................................................44 5.2 Obtaining a list of links ..............................................................................................................45 5.3 Removing duplicates .................................................................................................................46 5.3.1 Removing broken links .......................................................................................................... 47 5.3.2 Removing duplicate URLs .....................................................................................................48 5.3.3 Removing duplicate headlines ..............................................................................................48 Big Data Conversion Techniques including their Main Features and Characteristics 4 Table of contents 5.4 Scraping new articles ................................................................................................................49 5.5 Article volume over time ...........................................................................................................50 5.6 Constructing the uncertainty index ........................................................................................51 5.6.1 Empirical estimator ................................................................................................................ 51 5.6.2 From word mentions to indexes ............................................................................................52 5.7 Indices .......................................................................................................................................... 52 6 Conclusions .............................................................................................................................. 53 References ................................................................................................................................................. 54 Appendix .................................................................................................................................................... 57 Big Data Conversion Techniques including their Main Features and Characteristics 5 List of tables List of tables Table 1: Comparing statistics of the original sample and the two random subsamples. 22 Table 2: Raw and cleaned data for a hypothetical security with ticker XXX. 27 Table 3: Example of mobile phone usage data. 29 Table 4: Sample data for GPS using mobile phones. 32 Table 5: Sample data of web scraped prices. 37 Table 6: Various correlated keywords to “gbpusd” as returned by Google Correlate. 40 Table 7: Sample Twitter data scraped from Twitter. 41 Table 8: Summary of de-duplication of articles. 46 Table 9: Dictionaries of keywords by index. 52 Big Data Conversion Techniques including their Main Features and Characteristics 6 List of figures List of figures Figure 1: General description of big data conversion ...................................................................
Load more

Recommended publications
  • Clarifying the Legacy Data Conversion Plan & Introducing The
    PharmaSUG 2017 - Paper SS11 Documenting Traceability for the FDA: Clarifying the Legacy Data Conversion Plan & Introducing the Study Data Traceability Guide David C. Izard, Chiltern International Ltd. Kristin C. Kelly, Merck & Co. Inc. Jane A. Lozano, Eli Lilly & Company ABSTRACT Traceability from data collection through to the presentation of analysis results has always been a concern of the US Food & Drug Administration (FDA). The introduction of electronic data as part of submission added additional steps to confirm provenance of information. Now the requirement to provide clinical and non-clinical data based on a set of FDA endorsed data standards adds exponentially to the challenge, especially if legacy format data structures were utilized when the study was originally executed and reported but data meeting FDA requirements must be present in your submission. The PhUSE organization, made up of volunteers across industry, has worked closely with the FDA to develop tools to support the organization, presentation and interpretation of clinical and non-clinical data to regulatory bodies. Examples include the Study & Analysis Data Reviewer's Guides and the Study Data Standardization Plan. These documents describe routine situations where FDA endorsed data standards are deployed at the time a study is initiated; additional support is needed when the provenance of the data is not as straightforward. The FDA's Study Data Technical Conformance Guide calls out for the need to provide a Legacy Data Conversion Plan & Report when legacy data is the source of deliverables based on FDA endorsed data standards, but it is not very clear as to when you must provide one.
    [Show full text]
  • Design and Implementation of ADPCM Based Audio Compression Using Verilog
    The International Journal Of Engineering And Science (IJES) || Volume || 3 || Issue || 5 || Pages || 50-55 || 2014 || ISSN (e): 2319 – 1813 ISSN (p): 2319 – 1805 Design and Implementation of ADPCM Based Audio Compression Using Verilog Hemanthkumar M P, Swetha H N Department of ECE, B.G.S. Institute of Technology, B.G.Nagar, Mandya-571448 Assistant Professor ,Department of ECE, B.G.S. Institute of Technology, B.G.Nagar, Mandya-571448 ---------------------------------------------------------ABSTRACT----------------------------------------------------------- Internet-based voice transmission, digital telephony, intercoms, telephone answering machines, and mass storage, we need to compress audio signals. ADPCM is one of the techniques to reduce the bandwidth in voice communication. More frequently, the smaller file sizes of compressed but lossy formats are used to store and transfer audio. Their small file sizes allow faster Internet transmission, as well as lower consumption of space on memory media. However, lossy formats trade off smaller file size against loss of audio quality, as all such compression algorithms compromise available signal detail. This paper discusses the implementation of ADPCM algorithm for audio compression of .wav file. It yields a compressed file ¼ of the size of the original file. The sound quality of the audio file is maintained reasonably after compression. KEYWORDS: ADPCM, audio compression. --------------------------------------------------------------------------------------------------------------------------------------
    [Show full text]
  • The Development of Algorithms for On-Demand Map Editing for Internet and Mobile Users with Gml and Svg
    THE DEVELOPMENT OF ALGORITHMS FOR ON-DEMAND MAP EDITING FOR INTERNET AND MOBILE USERS WITH GML AND SVG Miss. Ida K.L CHEUNG a, , Mr. Geoffrey Y.K. SHEA b a Department of Land Surveying & Geo-Informatics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, email: [email protected] b Department of Land Surveying & Geo-Informatics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, email: [email protected] Commission VI, PS WG IV/2 KEY WORDS: Spatial Information Sciences, GIS, Research, Internet, Interoperability ABSTRACT: The widespread availability of the World Wide Web has led to a rapid increase in the amount of data accessing, sharing and disseminating that gives the opportunities for delivering maps over the Internet as well as small mobile devices. In GIS industry, many vendors or companies have produced their own web map products which have their own version, data model and proprietary data formats without standardization. Such problem has long been an issue. Therefore, Geographic Markup Language (GML) was designed to provide solutions. GML is an XML grammar written in XML Schema for the modelling, transport, and storage of geographic information including both spatial and non-spatial properties of geographic features. GML is developed by Open GIS Consortium in order to promote spatial data interoperability and open standard. Since GML is still a newly developed standard, this promising research field provides a standardized method to integrate web-based mapping information in terms of data modelling, spatial data representation mechanism and graphic presentation. As GML is not for data display, SVG is an ideal vector graphic for displaying geographic data.
    [Show full text]
  • Merchandising Data Conversion Implementation Guide
    Oracle® Retail Merchandising Conversion Implementation Guide Release 19.0 F24057-02 May 2020 Oracle® Retail Merchandising Conversion Implementation Guide, Release 19.0 Copyright © 2020, Oracle and/or its affiliates. All rights reserved. Primary Author: Contributors: This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S.
    [Show full text]
  • File to Pdf Converter Download TXT to PDF
    file to pdf converter download TXT to PDF. Convert TXT to PDF with just a few clicks. Download the desktop version or try the web app online today. Drop file here or. From Device. Your files are only stored on our servers for 24 hours, after which they are permanently destroyed. How to convert TXT to PDF. 1 Choose a file. Upload a TXT file from your computer, or through a cloud storage service such as Google Drive or Dropbox. You can also simply drag-and-drop it into the box to upload to convert to pdf. Once select your file or drag and drop it into the box, your file will begin to process automatically. Our software will begin converting your TXT file into a PDF file in no time. 2 Download or send link by email. Once the process to convert TXT to PDF is complete you can access your PDF file by downloading it to your computer and viewing it in your browser. We can also help you out by emailing you a link to your PDF document, which will be valid for 24 hours! If you choose to receive your file by email, ensure that you access it in the 24 hour window. We wouldn't you to forget about your nicely converted file. Xml file converter to pdf. How to Convert XML to PDF? Click the “Choose Files” button to select your XML files. Click the “Convert to PDF” button to start the conversion. When the status change to “Done” click the “Download PDF” … Xml File To Pdf Converter.
    [Show full text]
  • Coupled Schema Transformation and Data Conversion for XML and SQL
    Coupled Schema Transformation and Data Conversion for XML and SQL Pablo Berdaguer, Alcino Cunha?, Hugo Pacheco, and Joost Visser? DI-CCTC, Universidade do Minho, Portugal [email protected] Abstract. A two-level data transformation consists of a type-level trans- formation of a data format coupled with value-level transformations of data instances corresponding to that format. We have implemented a sys- tem for performing two-level transformations on XML schemas and their corresponding documents, and on SQL schemas and the databases that they describe. The core of the system consists of a combinator library for composing type-changing rewrite rules that preserve structural infor- mation and referential constraints. We discuss the implementation of the system’s core library, and of its SQL and XML front-ends in the func- tional language Haskell. We show how the system can be used to tackle various two-level transformation scenarios, such as XML schema evolu- tion coupled with document migration, and hierarchical-relational data mappings that convert between XML documents and SQL databases. Key words: Haskell, Transformation, SQL, XML 1 Introduction Coupled software transformation involves the modification of multiple software artifacts such that they remain consistent with each other [12,8]. Two-level data transformation is a particular instance of coupled transformation, where the coupled artifacts are a data format on the one hand, and the data instances that conform to that format on the other hand [7]. In this paper we will focus on the transformation of data formats described in the XML Schema or in the SQL language, coupled with the conversion of the corresponding data captured in XML documents or stored in SQL databases.
    [Show full text]
  • Section 4.15 Page 1 Delaware Department of Services for Children, Youth and Their Families FACTS II, RFP #07 Identified and Addressed
    Delaware Department of Services for Children, Youth and Their Families FACTS II, RFP #07 4.1 FACTS II Requirements Summary 4.11 Interfaces Section 4 4.2 Functional Requirements 4.12 System Development Bidder’s 4.3 Technical Requirements 4.13 System Testing Products, Methodology, 4.4 Customer Relations Management Tools 4.14 System Training and Approach to 4.5 Project Initiation and Management 4.15 Conversion the Project 4.6 System Hardware 4.16 System Implementation 4.7 System Planning and Analysis 4.17 Post Implementation Support 4.8 Requirements Verification 4.18 Support Federal Review 4.9 System Design 4.19 Security 4.10 Reports DE_SACWIS-002o_4 4.15 Conversion RFP reference: 6.15 Conversion, Page 56 Deloitte brings an automated conversion solution that minimizes the risk associated with a large data conversion based on our experience in converting Deloitte’s Iterative conversion data for large-scale Child Services solutions. Our process rigorously tests approach focuses on minimizing manual conversion process conversion through our iterative conversion methodology. Deloitte’s data cleansing methodology improves data Deloitte understands that the successful quality implementation of FACTS II is dependent upon a Use of conversion score card to quality data conversion effort. Converting data from continuously evaluate data disparate legacy systems into one integrated solution quality and can be a daunting task. Despite this challenge, we have successfully completed numerous data conversion efforts with a scope similar to the FACTS II project. The conversion of historical child welfare data is also important to support the ongoing business operations and reporting such as National Child Abuse and Neglect Data System (NCANDS), Adoption and Foster Care Analysis and Reporting (AFCARS) and National Youth in Transition Database (NYTD) which require a historical view of data.
    [Show full text]
  • XSLT in Context
    XSLT in Context This chapter is designed to put XSLT in context. It's about the purpose of XSLT and the task it was designed to perform. It's about what kind of language it is, and how it came to be that way; and it's about how XSLT fits in with all the other technologies that you are likely to use in a typical web-based application. I won't be saying much in this chapter about what an XSLT stylesheet actually looks like or how it works: that will come later, in Chapters 2 and 3. I shall begin by describing the task that XSLT is designed to perform – transformation – and why there is the need to transform XML documents. I'll then present a trivial example of a transformation in order to explain what this means in practice. The chapter then moves on to discuss the relationship of XSLT to other standards in the growing XML family, to put its function into context and explain how it complements the other standards. I'll describe what kind of language XSLT is, and delve a little into the history of how it came to be like that. If you're impatient you may want to skip the history and get on with using the language, but sooner or later you will ask "why on earth did they design it like that?" and at that stage I hope you will go back and read about the process by which XSLT came into being. Finally, I'll have a few things to say about the different ways of using XSLT within the overall architecture of an application, in which there will inevitably be many other technologies and components each playing their own part.
    [Show full text]
  • Dev Refs\Data Conversion Utilities
    Printed:11/19/98 4:16 PM By:SCEA Filename:final44Dataconv.doc Last saved by:SONY On:11/19/98 3:28 PM Comments: Master Copy, TechRef Release 4.3. Data ConversionUtilities , TechRef Release 4.3. © 1998 Sony Computer Entertainment Inc. Publication date: November 1998 Sony Computer Entertainment America 919 E. Hillsdale Blvd., 2nd floor Foster City, CA 94404 Sony Computer Entertainment Europe Waverley House 7-12 Noel Street London W1V 4HH, England The Data Conversion Utilities manual is supplied pursuant to and subject to the terms of the Sony Computer Entertainment PlayStation® License and Development Tools Agreements, the Licensed Publisher Agreement and/or the Licensed Developer Agreement. The Data Conversion Utilities manual is intended for distribution to and use by only Sony Computer Entertainment licensed Developers and Publishers in accordance with the PlayStation® License and Development Tools Agreements, the Licensed Publisher Agreement and/or the Licensed Developer Agreement. Unauthorized reproduction, distribution, lending, rental or disclosure to any third party, in whole or in part, of this book is expressly prohibited by law and by the terms of the Sony Computer Entertainment PlayStation® License and Development Tools Agreements, the Licensed Publisher Agreement and/or the Licensed Developer Agreement. Ownership of the physical property of the book is retained by and reserved by Sony Computer Entertainment. Alteration to or deletion, in whole or in part, of the book, its presentation, or its contents is prohibited. The information in the Data Conversion Utilities manual is subject to change without notice. The content of this book is Confidential Information of Sony Computer Entertainment.
    [Show full text]
  • Software Version: 1.02 Prerequisites
    HistoStitcher Installation and Usage Manual HistoStitcher is a tool for the reconstruction of stitched whole mount histology sections (WMHS) by selecting common fiducials on the edges of adjacent histology quadrants. HistoStitcher offers real-time rendering of histology images by using a multi-resolution pyramid for each histology quadrant. This software is intended for research purposes only, not for diagnostic or clinical use. Software Version: 1.02 Prerequisites: Please make sure that your computer meets the following requirements: - Windows 7 64-bit operating system or later Installation: The following components are included in the install package. Leave the checkboxes filled unless you previously installed those packages. 1) The HistoStitcher Software 2) The Data Conversions tool 3) The Matlab runtime compiler 7.14 4) The AMD-APP-SDK v2.7 The user must agree with the License agreement provided with the installer in order to install the application. After accepting this agreement, follow the instructions and specify the destination folder where you would like HistoStitcher installed. To complete the installation, restart the computer. We would like to acknowledge the use of some external libraries in this project: · Qt 4.8.4 · Python 2.7.3 · Matlab Runtime compiler 7.14 · Glew 1.7.0 · freeGlut 2.8.0 · Openjpeg 1.5.0 · OpenSlide 3.2.6 · JPeg-turbo 1.2.0 · FreeImage 3.15.3 · Zeromq 2.2.0 · Lua 5.1 · Db & db_stl 5.3 (Berkeley database library ) · Boost 1.49 · MPL Data conversion HistoStitcher can only load images that have been converted into db format. These images are tile- based multi-resolution JPEG-encoded images.
    [Show full text]
  • Formatting Display of Spatial Metadata Based on Xml
    FORMATTING DISPLAY OF SPATIAL METADATA BASED ON XML Xiaodong ZHOU a,*, Chuncheng YANGa, Nina MENGb aXi'an Institute of Surveying and Mapping, Xi’an 710054,China bCollege of Geology Engineering and Geomatics, Chang’an University, Xi’an 710054,China KEY WORDS: XML; XSLT; GIS; Spatial Metadata; Data Transformation; Formatting Transformation ABSTRACT: XML is the abbreviation of Extensible Markup Language, and is technology creating structural data. When XML appeared, it was widely used in some fields, for example: e-commerce, data exchange, multimedia transmission, etc. Spatial Metadata is description about geographic spatial data and related information resources, and it helps and promotes us to effectively locate, evaluate, compare, obtain and use geographic related data when it describes and explains content, quality, condition, position and other features about geographic spatial data. XML expressing metadata, not only is exact and can provide effective methods for metadata communication between heterogeneous systems, but also can strongly support for metadata transformations and formatting display. On the basis of mutual information definition and understanding structure, with XSLT we could define relevant XSL files to let processor to do relevant transforming operation and output HTML ( other required format ), and browse transforming results by IE browser. In the paper, we expressed spatial metadata with XML and used XSLT technology, then realized spatial metadata transformations by defining relevant XSL files and visualization by IE browser. 1. INTRODUCTION technologies, research how to present complex spatial metadata content quickly, effectively, vividly to the users, so that users XML stands for Extensible Markup Language, which is a can easily position, evaluation, comparison, access and use of technology of creating the structured data.
    [Show full text]
  • Data Conversion
    Data Conversion XML Conversion Overview Since its founding in 1986, Quality Associates, Inc. (QAI) has remained faithful to its guarantee to construct unparalleled electronic document conversion services based on a rich legacy of superior customization and exceptional quality. QAI is a leading US-based organization that currently operates one of the most advanced facilities of its kind, with extensive capabilities to capture data from text from virtually any computer medium and format, as well as from paper and microfilm, and reformat it to fit a client’s emergent needs. QAI provides a world-class mix of services and software to a broad range of industries such as, federal, state and local government agencies, publishing, aerospace and transportation, defense, manufacturing, telecommunications, technology, professional societies and services, and institutions. QAI’s expertise is widely acknowledged. Understanding our clients’ requirements On-time delivery We work with our clients to understand exactly what they QAI’s Production Control System (PCS) tracks the status and need, assuring that we meet all requirements as we develop location of every product (or record) and allows clients to the appropriate plan ultimately moving into full production. monitor all online through secure access on the web. We do not miss deadlines! Quality you can rely on We employ disciplined project management techniques Flexibility & multiple outputs assuring that all steps in our process are focused on A process designed to be easily modifiable as requirements delivering on time and to specification. change and to deliver client files exactly the way they need them. Risk minimization Scalability Our approach is based on a tried and proven process that We have processed more than a billion pages to date and are has been perfected over thousands of conversion projects continually expanding our infrastructure to handle the rapidly and more than a billion pages converted.
    [Show full text]