UNIVERSITY OF CALIFORNIA, SAN DIEGO Web Application Creation Made Easy: A SQL-driven Rapid Development Framework and a Do-It-Yourself Platform A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Computer Science by Kian Win Ong Committee in charge: Professor Alin Deutsch, Chair Professor Vish Krishnan Professor Ingolf Krueger Professor Bertram Lud¨ascher Professor Yannis Papakonstantinou 2010 Copyright Kian Win Ong, 2010 All rights reserved. The dissertation of Kian Win Ong is approved, and it is acceptable in quality and form for publication on micro- film and electronically: Chair University of California, San Diego 2010 iii TABLE OF CONTENTS Signature Page.................................. iii Table of Contents................................. iv List of Figures.................................. vi List of Tables................................... viii Acknowledgements................................ ix Vita........................................x Abstract of the Dissertation........................... xi Chapter 1 Introduction............................1 1.1 Rapid Application Development Framework.......2 1.2 Do-It-Yourself (DIY) Platform...............4 1.3 Roadmap..........................6 Chapter 2 Rapid Application Development Framework..........7 2.1 Introduction.........................7 2.1.1 Ajax background..................8 2.1.2 Framework and language contributions...... 12 2.1.3 System and optimization contributions...... 14 2.2 The FORWARD framework and scope.......... 15 2.2.1 Running example.................. 15 2.2.2 Architecture..................... 16 2.2.3 Data layer...................... 18 2.2.4 Visual layer..................... 21 2.3 Incremental page refresh.................. 24 2.3.1 Leveraging and extending incremental view main- tenance....................... 25 2.3.2 Incremental maintenance of the visual layer... 33 2.4 Evaluation.......................... 35 2.5 Related work........................ 38 2.6 Summary and Future Work................ 39 Chapter 3 Data Model............................ 42 3.1 Syntax............................ 44 3.2 Type System......................... 46 3.2.1 Data Trees...................... 46 3.2.2 Constraints..................... 49 iv 3.2.3 Primary keys & Context.............. 52 3.2.4 Data Consistency.................. 55 3.3 Data Diffs.......................... 57 Chapter 4 Query Language.......................... 60 4.1 Syntax Analysis....................... 62 4.2 Semantic Analysis...................... 68 4.2.1 Variable reference checking............. 68 4.2.2 Type checking / inference............. 70 4.2.3 Group by checking................. 71 4.2.4 Primary key checking / inference......... 71 4.3 Query Evaluation...................... 72 4.3.1 Decomposition of input data............ 72 4.3.2 Relational decomposition of query......... 78 4.3.3 Evaluating the generators............. 83 4.3.4 Reconstructing the nested output data tree.... 84 4.4 Data Sources........................ 86 4.4.1 Local versus Remote Data Sources........ 86 4.4.2 Data Source Catalog................ 88 Chapter 5 Incremental View Maintenance................. 90 5.1 Delta Tuples......................... 90 5.2 Leveraging Relational Incremental View Maintenance.. 92 5.3 Rewrite Rules to Create Incremental Queries....... 93 5.4 Evaluation of incremental query.............. 108 5.5 Reconstruction of the new view.............. 114 Chapter 6 Do-It-Yourself Platform..................... 116 6.1 Introduction......................... 116 6.2 Framework and Scope.................... 124 6.2.1 Reports....................... 125 6.2.2 Contextual Actions................. 127 6.2.3 User Group Definitions............... 129 6.3 Do-It-Yourself Design Facility............... 129 6.3.1 Derived Properties................. 130 6.3.2 Page-Driven Design................. 131 6.3.3 Workflow-Driven Design.............. 137 6.3.4 Automated Report Creation............ 141 6.4 Related Work........................ 147 Bibliography................................... 152 v LIST OF FIGURES Figure 2.1: Review Proposals page...................... 15 Figure 2.2: FORWARD architecture...................... 16 Figure 2.3: Page data tree........................... 18 Figure 2.4: Page schema............................ 18 Figure 2.5: Unit tree configuration....................... 23 Figure 2.6: Page state computation...................... 26 Figure 3.1: BNF for data tree......................... 44 Figure 3.2: BNF for schema tree........................ 44 Figure 3.3: Example data tree......................... 45 Figure 3.4: Example schema tree........................ 45 Figure 3.5: Grammar for Data Tree...................... 46 Figure 3.6: Grammar for Schema Tree..................... 47 Figure 3.7: Grammar for Constraints..................... 49 Figure 3.8: Example top-level collection.................... 52 Figure 3.9: Context............................... 54 Figure 3.10: Original Data Tree......................... 58 Figure 3.11: Data Diffs.............................. 58 Figure 4.1: Syntax of query (SELECT clause)................. 63 Figure 4.2: Syntax of query (expressions)................... 64 Figure 4.3: AST (queries)............................ 65 Figure 4.4: AST (SELECT clause)........................ 66 Figure 4.5: AST (expressions)......................... 67 Figure 4.6: Nested schema before decomposition............... 74 Figure 4.7: Schema after normalization.................... 75 Figure 4.8: Schema after concatenation (decomposed schema)........ 76 Figure 4.9: Data and Schema Tree for Local / Remote Data Sources.... 88 Figure 5.1: Example base relations....................... 111 Figure 5.2: Example decomposed old view.................. 111 Figure 5.3: Example flat deltas of base relations............... 112 Figure 5.4: Example generator tree with incremental queries........ 112 Figure 5.5: Example incremental view deltas................. 113 Figure 6.1: Submit Startup Page....................... 119 Figure 6.2: Evaluate Startups Page..................... 119 Figure 6.3: Advisor Comments Page...................... 120 Figure 6.4: TechCrunch50 Workflow Visualization (in ppt; see video demo for the actual one)............ 123 Figure 6.5: Page Wizard for Submit Startup Page............. 132 Figure 6.6: WYSIWYG Page Design..................... 134 vi Figure 6.7: Workflow-Driven Design...................... 139 Figure 6.8: Automated Report Extension on Evaluate Startups Page.. 142 vii LIST OF TABLES Table 2.1: Strawman implementation..................... 36 Table 2.2: FORWARD implementation.................... 36 viii ACKNOWLEDGEMENTS Chapter2 contains material from \Ajax-based Report Pages as Incremen- tally Rendered Views", by Yupeng Fu, Keith Kowalczykowski, Kian Win Ong, Kevin Keliang Zhao and Yannis Papakonstantinou, which appears in Proceedings of the ACM SIGMOD International Conference on Management of Data, SIG- MOD 2010. The dissertation author was the primary investigator and author of this paper. The material in this chapter is copyright c 2010 by the Association for Computing Machinery, Inc. (ACM). Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page in print or the first screen in digital media. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Publications Dept., ACM, Inc., fax +1 (212) 869-0481, or email [email protected]. Chapters3,4 and5 contain material from the technical report \Techni- cal Specifications of the FORWARD Framework", by Yupeng Fu, Kian Win Ong, Kevin Keliang Zhao, Yannis Papakonstantinou and Michalis Petropoulos. The dis- sertation author was the primary investigator and author of the relevant chapters in the technical report. Chapter6 contains material from \Do-It-Yourself Database-Driven Web Applications", by Keith Kowalczykowski, Kian Win Ong, Kevin Keliang Zhao, Alin Deutsch, Yannis Papakonstantinou and Michalis Petropoulos, which appears in Proceedings of the Fourth Biennial Conference on Innovative Data Systems Research, CIDR 2009. The dissertation author was the primary investigator and author of this paper. The paper is published under a Creative Commons License Agreement (http://creativecommons.org/licenses/by/3.0/). Permission is granted to copy, distribute, display, and perform the work, make derivative works and make commercial use of the work, but the work must be attributed to the authors and CIDR 2009. ix VITA 2003 B. Eng. in Computer Engineering, National University of Singapore. 2003-2004 Research Assistant, Hong Kong University of Science and Technology. 2004-2010 Graduate Student Researcher, University of California, San Diego. 2010 Ph. D. in Computer Science, University of California, San Diego. PUBLICATIONS Y. Fu, K. Kowalczykowski, K. W. Ong, K. K. Zhao, Y. Papakonstantinou. \Ajax- based Report Pages as Incrementally Rendered Views". ACM Management of Data International Conference (SIGMOD), 2010 G. Bhatia, Y. Fu, K. Kowalczykowski, K. W. Ong, K. K. Zhao, A. Deutsch, Y. Pa- pakonstantinou. \FORWARD: Design Specification Techniques for Do-It-Yourself Application