DOCSLIB.ORG

XML Processing and Website Scraping in Java How to Use Jsoup and Xmlbeam in Practice

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
XML Processing and Website Scraping in Java How to Use Jsoup and Xmlbeam in Practice XML processing and website scraping in Java How to use JSoup and XMLBeam in practice Gábor László Hajba This book is for sale at http://leanpub.com/javaxml This version was published on 2019-12-29 This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and many iterations to get reader feedback, pivot until you have the right book and build traction once you do. © 2014 - 2019 Gábor László Hajba Tweet This Book! Please help Gábor László Hajba by spreading the word about this book on Twitter! The suggested hashtag for this book is #WebsiteScrapingWithJava. Find out what other people are saying about the book by clicking on this link to search for this hashtag on Twitter: #WebsiteScrapingWithJava Contents Preface .............................. 1 What took me the most time? ............. 1 Acknowledgement .................... 2 XML Processing and the Google App Engine ..... 3 Why GAE? ......................... 3 Getting the data ...................... 4 XML to HTML ...................... 4 XML to PDF ........................ 6 XML to RTF ........................ 7 XML to “.*X” ........................ 8 Exporting the files in GAE ............... 8 XML Processing Advanced ................. 11 XML processing when memory matters ........ 12 Website scraping with JSoup and XMLBeam ..... 13 Runtime comparison advanced .............. 14 Upgrade to Java 8 ....................... 15 Custom printing for HTML with JSoup ......... 16 Printing XMLBeam projections .............. 17 Preface This is a book about using XML and HTML processing tools with the Java platform. I do not want to explain every tool you can find on the internet. It would be overwhelming and bad for the time-line of this book. So I just look at those tools I encountered in my daily work. Sometimes I toyed with those tools outside my daily work and researched something new or tried out some new features. One of those new features war the performance-tuning of the Website Scraper application mentioned in Chapter 3, where I thought about the performance of the application (you’ll see running with the whole data-set was a pain) and I changed some measurement parts and optimized the tools for a better comparison. And at the end I did parallelize the whole process with Java 8 and streams. The tools I will mention and explain a bit are XMLBeam and JSoup. The first one is an awesome XML processing engine, the second one is a website scraper tool. However you could use XMLBeam to do the same as JSoup – but the query language is a bit bothersome if you do not know XPath. What took me the most time? As I was writing this book the most time went on to have a stable base of application what I can use for performance 1 Preface 2 measurement and displaying the results. I had many ideas in mind and as I developed the pieces I got on more and better solutions and things I wanted to show you so I went on, refactored a part of the code. This meant that sometimes I had to redo runs with different configurations to have better results. But I do not regret that I did these errands – if I want them call so. I learned new things and I have many ideas where to go on. Perhaps this will result in another book or more blog articles. Acknowledgement I have to thank Sven Ewald, the creator of XMLBeam¹, for reviewing my book’s chapters about XMlBeam. Beside this he found time to answer my question and provide me samples with answers. Because LeanPub currently cannot display the whole book’s Table of Contents in the sample, I use this workaround to show you what you’ll get if you buy this book. I know, it is a bit awkward (you get some empty pages in your sample PDF) but so you can see what comes in the other version. ¹http://xmlbeam.org XML Processing and the Google App Engine In this chapter I’ll introduce you to XML processing and the Google App Engine (GAE). Why GAE? This is a good question. Mostly because I’ve worked with the GAE and I encountered some problems with it and the XML processing. So I thought I could share my problems and solutions with you. Perhaps you are interested in it or even it helps you to solve some problems. Writing about development to a GAE environment is always kind of “fun” because you have your solutions – and at the end you get a punch in your face from the GAE: some classes you want to use are not permitted. Then you start looking for a solution inside of the feasible area. This was the case when we (a co-worker and I) had the task to render an XML (provided from somewhere somehow – it is not important in the current context) in various formats: PDF and RTF. And as a bonus (because rendering those documents was not the easiest thing) I implemented a web-based display too to see if we get the right data. Visualizing XML as HTML is always the easiest thing. For me at least. 3 XML Processing and the Google App Engine 4 Getting the data The data came through a SOAP interface in an XML-bundle. I will not go into detail how to access the SOAP interface because it was not the easiest thing, and I’ve written an article in my blog about it some time ago. And SOAP is dead, REST is in, and currently HATEOAS is the new path you should walk when you work with remote structured data. However XML is a good structured data format which you can use in many ways. As you’ll see later, we needed an XML parser to get the data extracted from the transmitted XML. For this I created a quick and easy XML extractor which took the XML and extracted the required data with some XPath expressions into objects. It was not the best solution but it was least time-consuming. And it was a good practice for me to work with XML. XML to HTML As I mentioned this was no requirement but I wanted to see results as soon as possible so I added an HTML display of the XML input. Converting XML to HTML is easy: you only need to do an XSL Transformation (XSLT) and then you are done. The result you get is an HTML file (or XML or text – depending on your configuration). But this is for GAE a no-go because you are not allowed to create files dynamically from your application. Nevertheless you can end up with a solution to display your XML data represented as an HTML page: you only have to XML Processing and the Google App Engine 5 add the stylesheet to your data and most of the browsers will display it correctly. How to add the stylesheet? You have to add a tag containing the stylesheet to your XML- Data. For example: <?xml-stylesheet type="text/xsl" href="stylesheets/detailHtml.xsl\ "?> to transform the XML to HTML with XSLT (the detail- Html.xsl contains the transformation information). If you get your data from an interface (for example from a SOAP service) you have to be a bit tricky to get your XSL into your XML – because you get all of the data in one XML dataset. However if you think about a solution you would end up with: replacing the starting root node with itself and the stylesheet-node. With this workaround you can alter the XML dataset and display it along with XSLT. And this works with GAE too. String rootNode = "<rootNode>"; xmlString.replaceFirst(rootNode, "<?xml-stylesheet type=\"text/xsl\" href=\"stylesheets/detail\ Html.xsl\"?>" + rootNode); The example above is a little hack but you have to do this to add the stylesheet to the XML data. XML Processing and the Google App Engine 6 XML to PDF Converting an XML to a PDF is something simple too: with XSLT you create an XSL-FO (FO for Formatted Objects) document from your XML. An FO document is an XML using element names (node names) from the FO namespace. After this you can send your resulting FO document to a render- engine (for example Apache FOP) and you get your PDF. Sounds simple however GAE does not allow some of the classes which are used by Apache FOP (for example AWT graphics). So there is need for another workaround. iText is a good alternative to FOP however it does not handle FO documents. Nevertheless, iText has an XmlWorker project which should be used to render XML (XHTML) documents. So this sounds very good so I gave it a try. To get an XHTML from the XML I used again XSLT. Unfortunately I had some problems with applying the re- quired CSS to the XHTML output (some of them worked, some not) and as far as I can remember the XmlWorker had some problems with displaying the required images too. And beside the images there is a requirement of specific fonts to use when displaying the texts – and this is hardly manageable too when it comes to XHTML to PDF conversion (or at least I did not find a good-enough solution). So I ended up creating the PDF manually with iText added each element on it’s own, programmatically. To achieve this I created a custom XML extractor which split the provided XML result document into some classes (grouped by coher- ence) and added display-information to these classes. This was the least time-consuming solution.
Load more

Recommended publications
  • SDM 7.61 Open Source and Third-Party Licenses
    Structured Data Manager Software Version 7.61 Open Source and Third-party Licenses Document Release Date: February 2019 Software Release Date: February 2019 Open Source and Third-party Licenses Legal notices Copyright notice © Copyright 2017-2019 Micro Focus or one of its affiliates. The only warranties for products and services of Micro Focus and its affiliates and licensors (“Micro Focus”) are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Micro Focus shall not be liable for technical or editorial errors or omissions contained herein. The information contained herein is subject to change without notice. Adobe™ is a trademark of Adobe Systems Incorporated. Microsoft® and Windows® are U.S. registered trademarks of Microsoft Corporation. UNIX® is a registered trademark of The Open Group. This product includes an interface of the 'zlib' general purpose compression library, which is Copyright © 1995-2002 Jean-loup Gailly and Mark Adler. Documentation updates The title page of this document contains the following identifying information: l Software Version number, which indicates the software version. l Document Release Date, which changes each time the document is updated. l Software Release Date, which indicates the release date of this version of the software. You can check for more recent versions of a document through the MySupport portal. Many areas of the portal, including the one for documentation, require you to sign in with a Software Passport. If you need a Passport, you can create one when prompted to sign in. Additionally, if you subscribe to the appropriate product support service, you will receive new or updated editions of documentation.
    [Show full text]
  • Full-Graph-Limited-Mvn-Deps.Pdf
    org.jboss.cl.jboss-cl-2.0.9.GA org.jboss.cl.jboss-cl-parent-2.2.1.GA org.jboss.cl.jboss-classloader-N/A org.jboss.cl.jboss-classloading-vfs-N/A org.jboss.cl.jboss-classloading-N/A org.primefaces.extensions.master-pom-1.0.0 org.sonatype.mercury.mercury-mp3-1.0-alpha-1 org.primefaces.themes.overcast-${primefaces.theme.version} org.primefaces.themes.dark-hive-${primefaces.theme.version}org.primefaces.themes.humanity-${primefaces.theme.version}org.primefaces.themes.le-frog-${primefaces.theme.version} org.primefaces.themes.south-street-${primefaces.theme.version}org.primefaces.themes.sunny-${primefaces.theme.version}org.primefaces.themes.hot-sneaks-${primefaces.theme.version}org.primefaces.themes.cupertino-${primefaces.theme.version} org.primefaces.themes.trontastic-${primefaces.theme.version}org.primefaces.themes.excite-bike-${primefaces.theme.version} org.apache.maven.mercury.mercury-external-N/A org.primefaces.themes.redmond-${primefaces.theme.version}org.primefaces.themes.afterwork-${primefaces.theme.version}org.primefaces.themes.glass-x-${primefaces.theme.version}org.primefaces.themes.home-${primefaces.theme.version} org.primefaces.themes.black-tie-${primefaces.theme.version}org.primefaces.themes.eggplant-${primefaces.theme.version} org.apache.maven.mercury.mercury-repo-remote-m2-N/Aorg.apache.maven.mercury.mercury-md-sat-N/A org.primefaces.themes.ui-lightness-${primefaces.theme.version}org.primefaces.themes.midnight-${primefaces.theme.version}org.primefaces.themes.mint-choc-${primefaces.theme.version}org.primefaces.themes.afternoon-${primefaces.theme.version}org.primefaces.themes.dot-luv-${primefaces.theme.version}org.primefaces.themes.smoothness-${primefaces.theme.version}org.primefaces.themes.swanky-purse-${primefaces.theme.version}
    [Show full text]
  • Getting Started with Flying Saucer
    Getting Started with Flying Saucer Table of Contents What it is and How it Works ......................................................................................... 1 What it is ............................................................................................................1 What it does ........................................................................................................ 2 What you can do with it ........................................................................................ 2 What it does not do .............................................................................................. 2 License and Dependencies ..................................................................................... 3 Requirements for Running and Using Flying Saucer ................................................... 3 Setting your Classpath ...........................................................................................3 Using Flying Saucer ..................................................................................................... 4 Basic Usage .........................................................................................................4 Sample Applications ............................................................................................. 6 Configuration ...............................................................................................................6 The Flying Saucer Configuration File .....................................................................
    [Show full text]
  • SEARCH-BASED XML TESTING Bear This Notice and the Full Citation on the first Page
    XMLMate: Evolutionary XML Test Generation Nikolas Havrikov · Matthias Höschele · Juan Pablo Galeotti · Andreas Zeller Saarland University, Saarbrücken, Germany {havrikov, hoeschele, galeotti, zeller}@st.cs.uni-saarland.de private void transcribeSection(final Part part, final Section section, ABSTRACT final int sectionNumber, Generating system inputs satisfying complex constraints is still a final boolean numbering) { final int staffCount = section.getStaffCount(); challenge for modern test generators. We present XMLMATE, a for (int staffIndex = 0; staffIndex < staffCount; staffIndex++) { search-based test generator specially aimed at XML-based systems. final Staff staff = section.getStaff(staffIndex); XMLMATE leverages program structure, existing XML schemas, if (transcriber.getCurrentColumn() > 0) transcriber.newLine(); and XML inputs to generate, mutate, recombine, and evolve valid XML inputs. Over a set of seven XML-based systems, XMLMATE if (numbering && staffIndex == 0) { detected 31 new unique failures in production code, all triggered # transcriber.printString(new UpperNumber(sectionNumber)); # transcriber.spaceOrNewLine(); by system inputs and thus true alarms. # transcriber.printString( # new LowerRange(section.getFirstMeasureNumber(), Video: http://youtu.be/-yKom5mbft0 # section.getLastMeasureNumber())); # transcriber.spaceOrNewLine(); } else { Categories and Subject Descriptors transcriber.indentTo(2); D.2.5 [Software Engineering]: Testing and Debugging—Testing } // more code... tools (e.g., data generators, coverage testing)
    [Show full text]
  • MCS: Modular, Cross-Platform and Secure Application for Remote Assistance and Monitoring Tasks Xavier Vallejo Eusebi Calle Anton
    MCS: Modular, Cross-platform and Secure application for Remote Assistance and Monitoring Tasks Xavier Vallejo Eusebi Calle Antonio Bueno Jose L. Marzo Institute of Informatics Institute of Informatics Institute of Informatics Institute of Informatics and Applications and Applications and Applications and Applications University of Girona University of Girona University of Girona University of Girona 17071 Girona 17071 Girona 17071 Girona 17071 Girona [email protected] [email protected] [email protected] [email protected] Abstract using basic broadband Internet connections and readily available hardware. It is the MCS is a multimedia tool for rehabilitation continuation of the work carried out in the specialists, that allows remote assistance and TRiEM project, a joint effort in Girona in monitoring of patient’s activities. This tool 2005-06 between the BCDS research group of is the evolution of the work done in 2005- the UdG [1] and the Multiple Sclerosis Foun- 06 between the BCDS research group of the dation (FEM in Spanish and Catalan), to de- UdG and the Multiple Sclerosis Foundation velop a tool for MS telerehabilitation.1 (FEM in Spanish) in Girona under the TRiEM project. Multiple Sclerosis (MS) is a neu- 1.1 The Illness rodegenerative disease (ND) that can provoke significant exhaustion in patients even by the Multiple sclerosis (MS) is a chronic, inflam- simple act of going to the medical centre for matory disease that affects the central nervous rehabilitation or regular check ups. The tool system. MS can cause a variety of symptoms, presented in this paper allows the medical including changes in sensation, visual prob- staff to carry out distance patient consulta- lems, muscle weakness, depression, difficulties tions and other activities from their home, with coordination and speech, severe fatigue, minimizing house calls and travel time for the cognitive impairment, problems with balance, medical consultant.
    [Show full text]
  • S1000D Transformation Toolkit Guide for Developers
    S1000D Transformation Toolkit Guide for Developers Tyler Shumaker Software Engineer 1 CTC Overview • 501(c)(3) nonprofit established in 1987 • Staff of 1,400+ professionals • More than 50 locations • 900,000 sq. ft., including labs & demonstration space • Top 100 Government Contractor • Quality/EH&S Management System comprised of industry-best models: ISO 9001 (Quality) and 14001 (Environmental), AS9100 (Aerospace), and CMMI- SE/SW (Systems/Software Engineering) • Nationally recognized security capabilities with 300,000+ sq. ft. of Top Secret/Sensitive Compartmented Information Facility Space, JWICS, SIPRNet, and NIPRNet access 2 The Bridge Project • An OSD-funded project managed by the ADL Initiative. – Improves lifecycle management of technical training content. – Uses the S1000D technical data specification to manage learning content. • The Bridge Project enables enterprise training content management: – CONOPS: Sets parameters and operational scenarios for an S1000D development environment. – Bridge API: Connects learning content development tools to common source databases – AIM/SCO Workbench S1000D Support: Authoring tools write directly to S1000D. – S1000D Transformation Toolkit – Product Data Identification Service: Locates all S1000D based product support data linked to engineering change proposals. • Cost Benefits Analysis – 10% reduction in total ownership cost due to quicker data identification. 3 S1000D Transformation Toolkit • Problem Statement Driver – There is no interoperable publishing activity that supports the building
    [Show full text]
  • Unit Test Virtualization with VMVM
    Unit Test Virtualization with VMVM Jonathan Bell Gail Kaiser Columbia University Columbia University 500 West 120th St, MC 0401 500 West 120th St, MC 0401 New York, NY USA New York, NY USA [email protected] [email protected] ABSTRACT by the test suite. These tests are added to existing unit test Testing large software packages can become very time in- suites and in an ideal continuous integration environment, tensive. To address this problem, researchers have inves- executed regularly (e.g., upon code check-ins, or nightly). tigated techniques such as Test Suite Minimization. Test Because developers are often creating new tests, as software Suite Minimization reduces the number of tests in a suite grows in size and complexity, its test suite frequently grows by removing tests that appear redundant, at the risk of a similarly. Software can reach a point where its test suite has reduction in fault-finding ability since it can be difficult to gotten so large that it takes too long to regularly execute identify which tests are truly redundant. We take a com- | previous work has reported test suites in industry taking pletely different approach to solving the same problem of several weeks to execute fully [36]. long running test suites by instead reducing the time needed To cope with long running test suites, testers might turn to execute each test, an approach that we call Unit Test to Test Suite Minimization or Test Suite Prioritization [43]. Virtualization. With Unit Test Virtualization, we reduce Test Suite Minimization techniques such as [14, 15, 22, 23, the overhead of isolating each unit test with a lightweight 27, 28, 38, 41] seek to reduce the total number of tests to virtualization container.
    [Show full text]
  • Getting Started
    Docx4j - Getting Started Contents What is docx4j? ............................................................................................................................................. 3 What sorts of things can you do with docx4j? .............................................................................................. 4 Is docx4j for you? .......................................................................................................................................... 5 docx4j.NET .................................................................................................................................................... 5 What Word documents does it support? ..................................................................................................... 6 Handling legacy binary .doc files ................................................................................................................... 6 Getting Help: the docx4j forum .................................................................................................................... 6 Using docx4j via Maven ................................................................................................................................ 7 Using docx4j binaries .................................................................................................................................... 8 docx4j dependencies ...................................................................................................................................
    [Show full text]
  • Methodologies and Architecture for the Implementation of a Web Application
    Methodologies and Architecture for the Implementation of a Web Application Metoder och arkitektur för utveckling av en webbapplikation Lisan Chen Gabriel Banfalvi Examensarbete inom information- och programvarusystem Kandidat Degree Project in Information and Software Systems Stockholm, Sweden 2012 Kurs II121X, 15hp TRITA-ICT-EX-2012:328 Methodologies and Architecture for the Implementation of a Web Application Bachelor’s Thesis Lisan Chen & Gabriel Banfalvi For Kungliga Tekniska Högskolan Spring – Summer 2011 This is a bachelor’s project for KTH researching the usability of web applications and the usefulness of a number of new web-related technologies. To do this, it intends to rewrite a collaboration desktop application as a web application and evaluate the process. This report discusses the project and provides valuable information to developers and researchers seeking to write their own web applications. Detta är ett examensarbete vid Kungliga Tekniska Högskolan, som undersöker användbarheten av webbapplikationer och ett antal nya webbteknologier. Som del av arbetet kommer ett traditionellt datorprogram skrivas om som en webbapplikation, och denna process kommer att utvärderas och dokumenteras skriftligt. Denna rapport syftar till att ge värdefull information för utvecklare och forskare som är intresserade av webbapplikationer i en djupare nivå. 1 Contents 1. Introduction ................................................................................................................................... 1 1.1 Thesis Background ..............................................................................................................
    [Show full text]
  • Itext in Action Second Edition
    Covers iText 5 SECOND EDITION Bruno Lowagie MANNING Praise for the First Edition Each aspect is explained with numerous examples that can be applied to real-world problems right away. —Ulf Ditmer, JavaRanch Any developer who is making serious use of iText would be a fool not to buy this book. —Dave Gilbert, jfree.org Thorough and complete ... will be a long-running, valuable resource for iText and PDF. —Alan Dennis Software Architect, MyFamily.com One of the best technical books I have ever read! Great work! —Oliver Zeigermann Technical Trainer, CoreMedia AG I wholeheartedly recommend it. —Doug James eReporting Team Lead, Benefitfocus.com, Inc. Impressive! It provides depth without all the noise. —Justin Lee, President, Antwerkz Inc. Valuable to any developer using PDF. —Stuart Caborn, Consultant, Thoughtworks Licensed to Bruno Lowagie <[email protected]> Licensed to Bruno Lowagie <[email protected]> iText in Action Second Edition BRUNO LOWAGIE MANNING Greenwich (74° w. long.) Licensed to Bruno Lowagie <[email protected]> For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 180 Broad Street Suite 1323 Stamford, CT 06901 Email: [email protected] ©2011 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks.
    [Show full text]
  • Amon Millner Phd Thesis Proposal
    Computer as Chalk: Children Designing Interactive Experiences by Programming Connections between Digital and Physical Media for Expressive Community Building Amon Millner Thesis Proposal for the degree of Doctor of Philosophy in Media Arts and Sciences at the Massachusetts Institute of Technology 2009.08.18 Thesis Advisor Mitchel Resnick LEGO Papert Professor of Learning Research MIT Media Laboratory Thesis Reader Leah Buechley Assistant Professor MIT Media Laboratory Thesis Reader Melvin King Senior Lecturer Emeritus MIT Department of Urban Studies and Planning 1 Abstract This thesis explores how chalk can serve as a model for how people can harness the expressive power of computers. Chalk is well-known for being central to creative activities such as making sidewalk art or sketching personalized hopscotch grids. A goal of my work is to support children from a diversity of backgrounds to use computers for creatively expressing themselves, similarly to how they use chalk. I am motivated to do this work for reasons George Washington Carver explains: "new developments are the products of a creative mind, so we must therefore stimulate and encourage that type of mind in every way possible." At the core of this study is the Hook-ups System, a set of technologies and activities that I designed to enable young people to create interactive experiences by programming connections between physical and digital media. With the Hook-ups System, young people integrate sensors with a myriad of materials to create their own tangible interfaces. These interfaces control digital images and sounds based on computer programs the young people write to make projects such as games or responsive art pieces.
    [Show full text]
  • Analysing the Visual Complexity of Web Pages Using Document Structure
    Analysing the Visual Complexity of Web Pages Using Document Structure Simon Harper∗ Caroline Jay Eleni Michailidou Huangmao Quan Abstract The perception of the visual complexity of World Wide Web (Web) pages is a topic of significant interest. Previous work has examined the relationship between complexity and various aspects of presen- tation, including font styles, colours and images, but automatically quantifying this dimension of a Web page at the level of the document remains a challenge. In this paper we demonstrate that areas of high complexity can be identified by detecting areas, or `chunks', of a Web page high in block-level elements. We report a computational algorithm that captures this metric and places Web pages in a sequence that shows an 86% correlation with the sequences generated through user judgements of complexity. The work shows that structural aspects of a Web page influence how complex a user perceives it to be, and presents a straightforward means of determining complexity through examining the DOM. Keywords Visual Perception, Perceived Visual Complexity, Pragmatic Computational Technique, World Wide Web, Block-Level HTML. 1 Introduction Web page designers focus on good visual presentation to implicitly help users navigate, understand, and interact with the content. But current advances in Web page development (such as mash-ups1) encourage the concatenation of block-level elements dynamically drawn from other sites. Our previous work (see x 2.2) has lead us to theorise that this blocky visual chunking (Cascading Style Sheet{CSS, block-level2 structure) of Web pages is a major factor in determining perceived complexity (Yesilada et al., 2008).
    [Show full text]