CLEARWATER: AN EXTENSIBLE, PLIABLE, AND CUSTOMIZABLE APPROACH TO CODE GENERATION A Dissertation Presented to The Academic Faculty By Galen Steen Swint In Partial Fulfillment Of the Requirements for the Degree Doctor of Philosophy in Computer Science Georgia Institute of Technology August 2006 CLEARWATER: AN EXTENSIBLE, PLIABLE, AND CUSTOMIZABLE APPROACH TO CODE GENERATION Approved by: Dr. Calton Pu, Advisor Dr. Ling Liu College of Computing College of Computing Georgia Institute of Technology Georgia Institute of Technology Dr. Karsten Schwan Dr. Olin Shivers College of Computing College of Computing Georgia Institute of Technology Georgia Institute of Technology Dr. Donald F. Ferguson IBM Software Group IBM Corporation Date Approved: June 30, 2006 ACKNOWLEDGEMENTS One of the most wonderful, extraordinary qualities of a computer program is its synthesis of ideas from many sources into a useful, operational whole. To a certain degree, I think of a computer program as a bit like a knowledge machine in which every programmer and every designer create the patterns and parts that comprise the whole. Perhaps not too surprisingly, just as computer programmers often must sledge, hone, and burnish our ideas into the precision-fit final design, so too do many take part in the creation of a neo-Ph.D. Of my family, I wish to thank both my wife and my parents. My wife, Melissa, has had the fortitude, good nature, and stamina to keep me on track, to cheer me on when my motivation might flag. I thank her for that and hope I can re-pay her patience. My parents, too, were invaluable in their support, but more importantly, they set me on the path to achieve this Ph.D. – a concerted 29-year effort. I thank them for their guidance and motivation to move down the path to the highest education. Calton, my advisor, is also key to my success as a student. Without his vision and perception, I might very well have labored in these gold mines without producing an ounce of refined product; there were numerous times when his perspective was vital to extracting the true value of my research for presentation. Most importantly, Calton never failed to provide the resources and opportunities for me to succeed. Other students students and researchers have been instrumental in my Ph.D. work, as fellow programmers, idea generators, and social support. Thanks especially to Young, Wenchang, and Gueyoung for their tireless programming and willing technical discussions that led to the ISG and ACCT/Mulini. Thanks also to the DiSL group and iii Enterprise Research Group for serving as test audiences for the presentation of material before conferences. Thank you also to Ling Liu, for her additional guidance and input and to Charles Consel for the opportunity to collaborate with him and study abroad. Finally, in recognition of the funding without which this work could not have occurred: this work was partially funded by DARPA/IXO as a project in the PCES program, by DoE as a project in the SciDAC’s Scientific Data Management Center, by NSF/CISE as a project in the CCR division’s Distributed Systems program, IIS division’s Data and Application Security program, and the ITR program. This work was partially supported by NSF/CISE IIS and CNS divisions through grants IDM-0242397 and ITR- 0219902, DARPA ITO and IXO through contract F33615-00-C-3049 and N66001-00-2- 8901, and Hewlett-Packard. iv TABLE OF CONTENTS Acknowledgements ..........................................................................................................iii List of Tables ..................................................................................................................viii List of Figures.................................................................................................................... x Summary......................................................................................................................... xiv Chapter 1 Introduction..................................................................................................... 1 1.1. Code Generation for Heterogeneous Distributed Systems ................................. 1 1.2. Target Domains................................................................................................... 3 1.2.1. Information Flow Applications................................................................... 4 1.2.2. Distributed Enterprise Application Management ....................................... 6 1.3. Code Generation Challenges............................................................................... 7 1.4. Solution Requirements...................................................................................... 10 1.5. Thesis................................................................................................................ 13 Chapter 2 Clearwater..................................................................................................... 16 2.1. Clearwater Features .......................................................................................... 17 2.1.1. Architectural Overview............................................................................. 17 2.1.2. XML: Extensible Domain Specification................................................... 18 2.1.3. XSLT: Pliable Code Generation................................................................. 4 2.1.4. XML+XSLT: Flexible Customization through Weaving......................... 13 2.2. Implementations................................................................................................ 17 2.2.1. A Quick Look: The ISG............................................................................ 18 2.2.2. A Quick Look: ACCT and Mulini............................................................ 19 2.3. Work Related to Clearwater.............................................................................. 20 2.4. Summary for the Clearwater Approach............................................................ 22 Chapter 3 The ISG: Clearwater for Infopipes............................................................. 23 3.1. Code Generation for Infopipes.......................................................................... 23 3.2. Lessons from RPC ............................................................................................ 24 3.3. The Infopipes Abstraction................................................................................. 27 3.4. The Infopipes Toolkit ....................................................................................... 29 3.5. Spi: Specifying Infopipes.................................................................................. 31 3.6. XIP: XML for Infopipes ................................................................................... 33 3.7. ISG: The Infopipes Stub Generator .................................................................. 35 3.7.1. The Base Generator................................................................................... 35 3.7.2. The AXpect Weaver ................................................................................. 39 3.8. Benchmark Comparisons.................................................................................. 51 3.8.1. Communication Software ......................................................................... 51 3.8.2. Metrics of Interest..................................................................................... 54 3.8.3. Benchmark Environment .......................................................................... 58 3.8.4. Benchmark Execution............................................................................... 59 v 3.8.5. Benchmark Transfer Data......................................................................... 60 3.8.6. Two Node Synthetic Benchmarks ............................................................ 61 3.8.7. Three Node Synthetic Benchmarks .......................................................... 75 3.8.8. Application Based Benchmarks................................................................ 84 3.8.9. Benchmarks Conclusion ........................................................................... 87 3.9. Application 1: Distributed Linear Road............................................................ 88 3.9.1. Scenario..................................................................................................... 88 3.9.2. Implementation ......................................................................................... 91 3.9.3. Evaluation ................................................................................................. 94 3.9.4. Linear Road Summary.............................................................................. 96 3.10. Application 2: A MultiUAV Scenario.......................................................... 97 3.10.1. Scenario..................................................................................................... 97 3.10.2. Implementation ......................................................................................... 99 3.10.3. Results..................................................................................................... 104 3.11. Work Related to Infopipes.......................................................................... 106 3.12. Summary of ISG Research.......................................................................... 110 Chapter 4 ACCT and Mulini: Clearwater for Distributed Enterprise Application Management .................................................................................................................