DOCSLIB.ORG

Advanced Concepts and Applications of the UB-Tree

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

ddd d d

¨
INSTITUT FUR INFORMATIK

d d d d

  • d
  • d

d d d d d d d

  • ¨
  • ¨

d d

DER TECHNISCHEN UNIVERSITAT MUNCHEN

Advanced Concepts and Applications of the UB-Tree

Robert Josef Widhopf-Fenk

Institut fu¨r Informatik der Technischen Universit¨at Mu¨nchen

Advanced Concepts and Applications of the UB-Tree

Robert Josef Widhopf-Fenk

Vollsta¨ndiger Abdruck der von der Fakult¨at fu¨r Informatik der Technischen Universit¨at Mu¨nchen zur Erlangung des akademischen Grades eines

Doktors der Naturwissenschaften

genehmigten Dissertation.

  • Vorsitzende:
  • Univ.-Prof. G. J. Klinker, Ph.D.

Pru¨fer der Dissertation: 1. Univ.-Prof. R. Bayer, Ph.D., emeritiert
2. Univ.-Prof. A. Kemper, Ph.D.

Die Dissertation wurde am 05.08.2004 bei der Technischen Universita¨t Mu¨nchen eingereicht und durch die Fakult¨at fu¨r Informatik am 07.02.2005 angenommen.

Acknowledgements

First of all, I want to thank my supervisor Rudolf Bayer for his advice, guidance, support, and patience. Without his invitation to be a Ph.D. student, I would have never been working on database research.
I owe my former colleagues in the MISTRAL project many thanks for fruitful discussions, proofreading, inspirations, and motivating me: Volker Markl, Frank Ramsak, Martin Zirkel and our “external” member Roland Pieringer.
I like to thank my students, Oliver Nickel, Chris Hodges and Werner Unterhofer contributing to this work by supporting the implementation of algorithms and performing experiments.
Special thanks to our project partners at TransAction, Teijin, GfK, and the members of the EDITH project.
And last but not least, many thanks to Michael Bauer, Bernd Reiner, and Charlie Hahn for discussions, distraction and small talk. Specifically: to Charlie for the ice cream; to Alex S. for the Gauloise; to Evi Kollmann for all your help, even when you were loaded with other work you always were at hand. Also thanks to everyone I accidently forgot.

To my love Birgit for your patience, for our children Matilda and Valentin, and for going through this with me!

Abstract

The UB-Tree is an index structure for multidimensional point data. By name, it claims to be universal, but this imposes a huge burden, as there are few things which really prove to be universal. This thesis takes a closer look at aspects where the UB-Tree is not universal at a first glance.
The first aspect is the discussion of space filling curves (SFC), in particular comparing the Z-curve and the Hilbert-curve. The Z-curve is used to cluster data indexed by the UB-Tree and we highlight its advantages in comparison to other SFCs. While the Hilbertcurve provides better clustering, the Z-curve is superior w.r. to other metrics, i.e., it is significantly more efficient to calculate addresses and the mapping of queries to SFC- segments, and it is able to space efficiently index arbitrary universes. Thus the Z-curve is more universal here.
The second aspect are bulk operations on UB-Trees. Especially for data warehousing the bulk insertion and deletion are crucial operations. We present efficient algorithms for incremental insertion and deletion.
The third aspect is the comparison of the UB-Tree with bitmap indexes used for an example data warehousing application. We show how performance of bitmap indexes is increased by clustering the base table according to a SFC. Still the UB-Tree proves to be superior.
The fourth aspect is the efficient management of data with skewed data distributions.
The UB-Tree adapts its partitioning to the actual data distribution, but in comparison to the R-Tree, it suffers from being not able to prune search path leading to unpopulated space (= dead space). This is caused by partitioning the complete universe with separators. We present a novel index structure, the bounding UB-Tree (BUB-Tree), which is a variant of the UB-Tree inheriting its worst case guarantees for the basic operations while efficiently addressing queries on dead space. In comparison to R-Trees, its query performance is similar while offering superior maintenance performance and logarithmic worst case guarantees, thus being more universal than the R∗-Tree.
The last aspect addressed in this thesis is the management of spatial data. The UB-Tree is an index designed for point data, however also spatial objects can be indexed efficiently with it by mapping them to higher dimensional points. We discuss different mapping methods and their performance in comparison to the RI-Tree and R∗-Tree.

Our conclusion: The UB-Tree comes closer to being an universal index than any other competing index structure. It is flexible, dynamic, relatively easy to integrate into a DBMS kernel, and provides logarithmic worst case guarantees for the basic operations of insertion, deletion, and update. By extending its concepts to the BUB-Tree it is also able to efficiently support skewed queries on skewed data distributions.

Contents

  • Preface
  • 1

  • 1 Introduction
  • 3

344555667778
1.1 The Objective: Getting Information Fast . . . . . . . . . . . . . . . . . . . 1.2 The Problem: The Properties of Real World Data . . . . . . . . . . . . . .
1.2.1 Complex Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Skewed Data Distribution . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Huge Data Amounts . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.4 Continuous Change . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 The Solution: Multidimensional Access Methods . . . . . . . . . . . . . . . 1.4 Objective and Outline of this Thesis . . . . . . . . . . . . . . . . . . . . . 1.5 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5.1 RDBMSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5.2 Multidimensional Access Methods . . . . . . . . . . . . . . . . . . .
1.6 The MISTRAL Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  • 2 Terminology and Basic Concepts
  • 11

2.1 General Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.1 Tuple, Relation, Universe . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.2 Query, Result Set, and Selectivity . . . . . . . . . . . . . . . . . . . 14
2.2 Storage Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3 Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.5 Access Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.5.1 Clustering Index, Primary Index, Secondary Index . . . . . . . . . . 24 2.5.2 Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.5.3 Accessed Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

  • 3 Space-filling Curves
  • 29

3.1 Space Filling Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.1.1 Compound Curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.1.2 Snake-Curve and Zig-Zag-Curve . . . . . . . . . . . . . . . . . . . . 33 3.1.3 Fractal Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

iii iv

CONTENTS

3.1.4 Curves created by order preserving Bit-Permutations . . . . . . . . 39
3.2 Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.1 Address calculations . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.2.2 Indexing a Space Filling Curve . . . . . . . . . . . . . . . . . . . . 44 3.2.3 SFC Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.3.1 [Jag90a] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.3.2 [MJF+01] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.3.3 [HW02] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.3.4 [MAK02] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.3.5 [Law00] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.3.6 Summary of Related Work . . . . . . . . . . . . . . . . . . . . . . . 57
3.4 Comparison of SFC Properties . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.4.1 Region Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 3.4.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

  • 4 The UB-Tree
  • 63

4.1 Basic Concepts of UB-Trees . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.1.1 Z-regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.1.2 Range Query Processing . . . . . . . . . . . . . . . . . . . . . . . . 66
4.2 UB-Tree Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.2.1 UBAPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

  • 4.2.2 Transbase
  • Hypercube . . . . . . . . . . . . . . . . . . . . . . . . 68

4.2.3 RFDBMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.3 Bulk Insertion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.3.2 General Problem Description . . . . . . . . . . . . . . . . . . . . . 72 4.3.3 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.3.4 DBMS-Kernel Integration . . . . . . . . . . . . . . . . . . . . . . . 78 4.3.5 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.3.6 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 80 4.3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.4 Bulk Deletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.4.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.4.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.4.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

  • 5 Bitmap-Indexes on Clustered Data
  • 95

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.1.1 Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.1.2 Query Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

CONTENTS

v
5.1.3 Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.1.4 Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.2 Bitmaps on clustered data . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 5.3 Evaluation with the GfK DWH . . . . . . . . . . . . . . . . . . . . . . . . 103
5.3.1 Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 5.3.2 Query Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 5.5 Multi-User environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

  • 6 The BUB-Tree
  • 123

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 6.3 BUB-Tree regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.4 Point Query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 6.5 Insertion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.6 Page Split . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 6.7 Deletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 6.8 Range Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 6.9 Bulk Insertion and Bulk Deletion . . . . . . . . . . . . . . . . . . . . . . . 136
6.9.1 Initial Bulk Loading . . . . . . . . . . . . . . . . . . . . . . . . . . 136 6.9.2 Incremental Bulk Loading . . . . . . . . . . . . . . . . . . . . . . . 136 6.9.3 Bulk Deletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.10 Reorganization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.10.1 Tightening Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 6.10.2 Optimizing the Index Coverage . . . . . . . . . . . . . . . . . . . . 138 6.10.3 Increasing the Page Utilization . . . . . . . . . . . . . . . . . . . . 139
6.11 Implementation-Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.12 Comparison of BUB-Tree and R-Tree . . . . . . . . . . . . . . . . . . . . . 141
6.12.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 6.12.2 {R}-Tree coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 6.12.3 Bounding Technique . . . . . . . . . . . . . . . . . . . . . . . . . . 143 6.12.4 Supported Data Types . . . . . . . . . . . . . . . . . . . . . . . . . 144 6.12.5 Dimensionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 6.12.6 Multi-User Support . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 6.12.7 Page Split . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 6.12.8 Index Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 6.12.9 Maintenance Performance . . . . . . . . . . . . . . . . . . . . . . . 145 6.12.10Query Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.13 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
6.13.1 Random insertion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 6.13.2 Bulk Loading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 6.13.3 Query Procession . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 6.13.4 Point Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 6.13.5 Random Range Queries . . . . . . . . . . . . . . . . . . . . . . . . 171 6.13.6 Street Section Queries . . . . . . . . . . . . . . . . . . . . . . . . . 173
6.14 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

  • 7 Management of Spatial Objects
  • 177

7.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
7.1.1 RDBMS Implementations . . . . . . . . . . . . . . . . . . . . . . . 178 7.1.2 Interval Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 7.1.3 Sub-Cube Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 7.1.4 Previous UB-Tree based solutions . . . . . . . . . . . . . . . . . . . 181
7.2 Mapping Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
7.2.1 Approximation by a MBB . . . . . . . . . . . . . . . . . . . . . . . 181 7.2.2 Approximation by SFC-segments . . . . . . . . . . . . . . . . . . . 186 7.2.3 Indexing the Parameter Space . . . . . . . . . . . . . . . . . . . . . 188
7.3 Experiments: Interval Data . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.3.1 Artificial Interval Data Sets . . . . . . . . . . . . . . . . . . . . . . 189 7.3.2 Artificial Interval Query Sets . . . . . . . . . . . . . . . . . . . . . . 190 7.3.3 Interval Index Structures . . . . . . . . . . . . . . . . . . . . . . . . 191 7.3.4 Qualitative Comparison of Interval-Index Structures . . . . . . . . . 191 7.3.5 Interval Measurement Results . . . . . . . . . . . . . . . . . . . . . 192 7.3.6 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 7.3.7 Real World Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 7.3.8 GEO Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 7.3.9 Endpoint Transformation . . . . . . . . . . . . . . . . . . . . . . . . 201 7.3.10 Comparison of Point-Transformation Variants . . . . . . . . . . . . 204 7.3.11 Approximation by SFC-segments . . . . . . . . . . . . . . . . . . . 204
7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

  • 8 Summary
  • 207

8.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

  • A Description of the Test System
  • 209

A.1 System Sunbayer69 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 A.2 System Sunwibas0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 A.3 System Atmistral7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

  • B Real World Data Sets
  • 211

B.1 GfK Datawarehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 B.2 GEO Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

C Measures of RFDBMS Bibliography
224 239

List of Figures

2.1 Example for a two dimensional Universe and Queries on it . . . . . . . . . 15 2.2 Physical Structure of a Hard-Disk . . . . . . . . . . . . . . . . . . . . . . . 18 2.3 Percentage of Positioning Cost with growing Page Size . . . . . . . . . . . 19 2.4 Random and sequential Access Times on different Hardware Configurations 20 2.5 Different Types of Clustering for Lexical Order and the Interval Query [c, f] 23

2.6 Clustering Page-Based Index vs. Secondary Index . . . . . . . . . . . . . . 25

2.7 Range Query and actually affected Area of the Universe . . . . . . . . . . . 27 3.1 Simple SFCs for 8x8 universe with range query . . . . . . . . . . . . . . . . 29 3.2 Example relations of ∆M to ∆S . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3 Universe, quad tree partitioning, and address . . . . . . . . . . . . . . . . . 34 3.4 Z-curve for 8x8 universe . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.5 bit-interleaving for dimensions with differing domains . . . . . . . . . . . . 37 3.6 Hilbert-curve for 8x8 universe . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.7 Hilbert-curve indexing an universe with differing dimension cardinalities . . 38 3.8 Gray-Code-curve for 8x8 universe . . . . . . . . . . . . . . . . . . . . . . . 40 3.9 Brother-father-search with Z-curve . . . . . . . . . . . . . . . . . . . . . . 43 3.10 Cost of post filtering vs. NJI/NJO calculation . . . . . . . . . . . . . . . . 48 3.11 Space partitioning for Z- and Hilbert-curve . . . . . . . . . . . . . . . . . . 50 3.12 B-Tree traversal: Page accesses . . . . . . . . . . . . . . . . . . . . . . . . 51 3.13 Percentage of segments in SFCs . . . . . . . . . . . . . . . . . . . . . . . . 56 3.14 SFC-Regions consisting of disjoint areas in Ω . . . . . . . . . . . . . . . . . 60

4.1 A UB-Tree Partitioning the World into Z-regions . . . . . . . . . . . . . . 64 4.2 Z-regions of a UB-Tree for a given Data Distribution . . . . . . . . . . . . 66 4.3 A Bulk Loading Architecture for one dimensional clustering Indexes . . . . 73 4.4 Estimated Number of Page Reads and Writes . . . . . . . . . . . . . . . . 79 4.5 Changes of UB-Tree Space Partitioning during Incremental Loading . . . . 80 4.6 The Bulk Loading Architecture as implemented for UB-Trees . . . . . . . . 81 4.7 Data Distribution and Times for Copying resp. Pipe-Lining . . . . . . . . . 83 4.8 Address Calculation, Sorting and Loading Performance for GfK DWH . . . 83 4.9 Page Statistics for Bulk Loading of GfK DWH . . . . . . . . . . . . . . . . 86 4.10 Deleting and Freeing Subtrees . . . . . . . . . . . . . . . . . . . . . . . . . 91

vii viii

LIST OF FIGURES

5.1 Query Processing with Bitmap Indexes . . . . . . . . . . . . . . . . . . . . 98 5.2 RLE Bitmap Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.3 Cost for Address Calculation and Sorting . . . . . . . . . . . . . . . . . . . 105 5.4 Maintenance Cost for Bitmap Indexesc . . . . . . . . . . . . . . . . . . . . 107 5.5 Bitmaps mapped to Images . . . . . . . . . . . . . . . . . . . . . . . . . . 108 5.6 Overall Maintenance Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.7 PG SERIES: Result Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 5.8 PG SERIES: Elapsed Time . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.9 PG SERIES: Page Accesses w.r. to Query . . . . . . . . . . . . . . . . . . 116 5.10 Time per Logical Page Accesses . . . . . . . . . . . . . . . . . . . . . . . . 117 5.11 PG SERIES: Page Accesses w.r. to Result Set Size . . . . . . . . . . . . . 118 5.12 PG SERIES Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.13 2Y PG SERIES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

Recommended publications
  • Today's Problems, Yesterday's Toolkit

    Today's Problems, Yesterday's Toolkit

    TODAY’S PROBLEMS, YESTERDAY’S TOOLKIT Developed by Owned by and working for Australian and New Zealand Governments. 1 THE ROADMAP: TODAY’S PROBLEMS, YESTERDAY’S TOOLKIT 4 THE PUBLIC PROBLEM SOLVING IMPERATIVE 5 The Death of Trust 6 Why Public Problem Solving is So Urgent 7 Why Building Skills will Restore Trust 10 How Public Problems Differ 12 The Public Problem-Solving Pathway 14 INNOVATION SKILLS IN THE PUBLIC SECTOR 16 Definitions of Innovation Skills in the Public Sector 17 The Public Entrepreneur’s Skillset 18 Training for Innovation Skills 21 ANZSOG Survey of Innovation Skills 27 THE INSTITUTIONAL ENVIRONMENT 35 Features of Innovative Institutions 37 New Innovation Institutions 41 Policies to Catalyse Innovative Institutions 46 Talent Mobility: Moving Brains Around 48 Sustaining Innovative Institutions 50 CONCLUSION: LOOKING TO TOMORROW 53 Acknowledgements 56 About the Authors 57 ANNEX I 58 Section I. Respondent’s characteristics 60 Section II. Awareness of skill & training 63 Section III. Use of skill 67 Section IV. Interest in learning skills and learning preferences 71 Section V. Enabling environment 74 Section VI. Relationship between skill practice, training and environment 79 ANNEX II — AUSTRALIA AND NEW ZEALAND INNOVATION SKILLS SURVEY INSTRUMENT 83 ANNEX III — LIST OF INTERVIEWEES 106 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying or otherwise, without the prior permission of the publisher. © 2019 The Australia and New Zealand School of Government (ANZSOG) and the authors. 3 THE ROADMAP: TODAY’S PROBLEMS, YESTERDAY’S TOOLKIT Governments of all political stripes are being The report begins by explaining why public problem- buffeted by technological and societal change.
  • (12) United States Patent (10) Patent No.: US 8.473450 B2 Bakalash Et Al

    (12) United States Patent (10) Patent No.: US 8.473450 B2 Bakalash Et Al

    USOO847345OB2 (12) United States Patent (10) Patent No.: US 8.473450 B2 Bakalash et al. (45) Date of Patent: *Jun. 25, 2013 (54) RELATIONAL DATABASE MANAGEMENT (52) U.S. Cl. SYSTEM (RDBMS) EMPLOYING USPC ........................................... 707/600; 707/770 MULTI-DIMENSIONAL DATABASE (MDDB) (58) Field of Classification Search FOR SERVICING QUERY STATEMENTS None THROUGHONE ORMORE CLIENT See application file for complete search history. MACHINES (56) References Cited (75) Inventors: Reuven Bakalash, Beer Sheva (IL); Guy U.S. PATENT DOCUMENTS Shaked, Shdema (IL); Joseph Caspi, 4,590,465. A 5, 1986 Fuchs Herzlyia (IL) 4,598.400 A 7, 1986 Hillis (73) Assignee: Yanicklo Technology Limited Liability (Continued) Company, Wilmington, DE (US) FOREIGN PATENT DOCUMENTS (*) Notice: Subject to any disclaimer, the term of this EP O 314 279 5, 1989 patent is extended or adjusted under 35 EP O 657 052 6, 1995 U.S.C. 154(b) by 13 days. (Continued) OTHER PUBLICATIONS This patent is Subject to a terminal dis claimer. Scheuermann, P. J. Shim and R. Vingralek “WATCHMAN: A Data Warehouse Intelligent Cache Manager'. Proceedings of the 22nd International Conference on Very Large Databases (VLDB), 1996, (21) Appl. No.: 12/455,665 pp. 51-62.* (22) Filed: Jun. 4, 2009 (Continued) (65) Prior Publication Data Primary Examiner — Robert Timblin (74) Attorney, Agent, or Firm — Knobbe, Martens, Olson & US 2009/027641.0 A1 Nov. 5, 2009 Bear LLP Related U.S. Application Data (57) ABSTRACT A relational database management system (RDBMS) for ser (63) Continuation of application No. 1 1/888,904, filed on vicing query statements through one or more client machines.
  • Vasili Korol

    Vasili Korol

    Vasili Korol Senior Software Developer Odense, Denmark Age: 35 mob.: +45 20 68 50 23 Married, have son (born 2010) e-mail: [email protected] ​ Personal Statement ⚬ Strong IT skills (16+ years of versatile experience) ⚬ Background in physics research ⚬ Work effectively both as team member and leader ⚬ Enthusiastic and committed ⚬ Spoken languages: Russian (native), English (fluent), Danish (Prøve i Dansk 3 / level B2) ​ ​ ​ ​ Education 2006–2008: Master’s degree (with distinction) in applied physics. ​ 2002–2006: Bachelor’s degree (with distinction) in applied physics. Under- to postgraduate student at St. Petersburg State Polytechnical University, Faculty of Physics and Technology, Dept. of Cosmic Physics. The thesis “Search for possible space-time variations of the fine-structure constant and isotopic shifts” (a supervisor Prof. ​ M.G. Kozlov). ​ 1992-2002: School education in St. Petersburg, Russia and Belfast, UK (in 1993). Professional Career 2015 – Feb 2021: Software developer in the QuantBio research group at the University of ​ ​ ​ ​ Southern Denmark (SDU), Institute of Physics, Chemistry and Pharmacy (HPC section). I am the principal developer of VIKING, a service providing a web interface for configuring ​ ​ ​ and running scientific computational tasks on supercomputers. I designed the software architecture, developed the system core and coordinated the work of several developers. 2014 – 2015: Lead programmer (Perl) at Internet Projects LLC, russian informational portals subscribe.ru and sendsay.ru (St. Petersburg, Russia). ​ ​ ​ Worked with a team of developers on projects targeted at developing an API for news aggregation and content processing services. This involved integration with various online platforms (Facebook, Twitter, Vkontakte, LiveJournal, Google Analytics), web scraping and designing instruments for user publications at the portals and beyond.
  • A Location Service for Partial Spatial Replicas Implementing an R-Tree in a Relational Database

    A Location Service for Partial Spatial Replicas Implementing an R-Tree in a Relational Database

    A Location Service for Partial Spatial Replicas Implementing an R-Tree in a Relational Database Yun Tian Department of Computer Science, Eastern Washington University Cheney, WA, USA 99004 Philip J. Rhodes Department of Computer and Information Science, University of Mississippi, University, MS, USA 38677 Abstract As parallel computing has become increasingly common, the need for scalable and efficient ways of storing and locating data has become increas- ingly acute. For years, both grid and cloud computing have distributed data across machines and even clusters at different geographic locations (sites). However not all sites need all of the data in a particular data set, or have the (perhaps specialized) processing capabilities required. These facts chal- lenge the conventional wisdom that we should always move the computation to the data rather than the data to the computation. Sometimes the data actually required is small. In other cases, the site with specialized processing capabilities (such as a GPU equipped cluster) cannot handle the demands placed on it unless a way is found to let that cluster select the data that is actually needed, even if it is not stored locally. Keywords: Globus Toolkit, Replica Location Service, R-tree, Prefetching, Spatial, Replica Email addresses: [email protected] (Yun Tian), [email protected] (Philip J. Rhodes) Preprint submitted to Journal of Parallel and Distributed Computing 1. Introduction Although the problems just described are general, this paper concentrates on the problem of computation with spatial data. Such data is often the product of large computations and simulations, and also natural observations. In any case, spatial data sets present special problems, including not only size but also the mismatch between the n-dimensional nature of the dataset and its representation on disk.
  • Integrating the UB-Tree Into a Database System Kernel

    Integrating the UB-Tree Into a Database System Kernel

    Integrating the UB-Tree into a Database System Kernel Frank Ramsak1, Volker Markl1, Robert Fenk1, Martin Zirkel2, Klaus Elhardt3, Rudolf Bayer1,2 1Bayerisches Forschungszentrum 2Institut für Informatik 3TransAction Software GmbH für Wissensbasierte Systeme TU München Gustav-Heinemann-Ring 109, Orleansstraße 34, Orleansstraße 34, D-81739 München, Germany D- 81667 München, Germany D-81667 München, Germany {frank.ramsak, robert.fenk, volker.markl}@forwiss.de, {zirkel, bayer}@in.tum.de, [email protected] Abstract Multidimensional access methods have shown 1 Introduction high potential for significant performance im- Various research approaches in the past have shown that provements in various application domains. multidimensional access methods (MAMs) have a high However, only few approaches have made their impact on different database application domains like data way into commercial products. In commercial warehousing, data mining, or geographical information database management systems (DBMSs) the B- systems. However, despite the vast research effort MAMs Tree is still the prevalent indexing technique. have not made their way into commercial database Integrating new indexing methods into existing management systems on a broad scale. This is mostly due database kernels is in general a very complex to the fact that the integration of these complex data and costly task. Exceptions exist, as our experi- structures into an existing database kernel is fairly ence of integrating the UB-Tree into TransBase, complicated. Especially concurrency and recovery issues, a commercial DBMS, shows. The UB-Tree is a which are as important as performance issues for very promising multidimensional index, which commercial systems, are major obstacles. For most has shown its superiority over traditional access MAMs new solutions to these problems, e.g., locking for methods in different scenarios, especially in R-Trees [KB95, CM98], have to be developed, as the new OLAP applications.
  • Transbase®: a Leading-Edge ROLAP Engine Supporting Multidimensional Indexing and Hierarchy Clustering∇ R

    Transbase®: a Leading-Edge ROLAP Engine Supporting Multidimensional Indexing and Hierarchy Clustering∇ R

    Transbase®: A leading-edge ROLAP Engine supporting multidimensional Indexing and Hierarchy Clustering∇ R. Pieringer1, K. Elhardt1, F. Ramsak2, V. Markl3, R. Fenk2, R. Bayer2 1 Transaction 2 Bayerisches 3IBM Almaden Software GmbH Forschungszentrum für Research Center, Thomas-Dehler- Wissensbasierte Systeme K55/B1, Str. 18 Boltzmannstr. 3 650 Harry Road, San D-81737 München D-85747 München Jose, CA 95120- {pieringer,elhardt}@ {ramsak,fenk}@forwiss.de, 6099 transaction.de [email protected] [email protected] Abstract: Analysis-oriented database applications, such as data warehousing or customer relationship management, play a crucial role in the database area. In general, the multidimensional data model is used in these applications, realized as star or snow-flake schemata in the relational world. The so-called star queries are the prevalent type of queries on such schemata. All database vendors have extended their products to support star queries efficiently. However, mostly reporting queries benefit from the optimizations, like pre-aggregation, while ad-hoc queries usually lack efficient support. We present the DBMS Transbase® in this paper, which provides a new physical organization of the data based on hierarchical clustering and multidimensional clustering combined with multidimensional indexing. In combination with new query optimizations (e.g., hierarchical pre-grouping) significant performance improvements are achieved. The paper describes how the new technology is implemented in the Transbase® product and how it is made available to the user as transparently as possible. The benefits are illustrated with a real-world data warehousing scenario. 1 Introduction Data warehousing (DW), online analytical processing (OLAP), and customer relationship management (CRM), have become a major market in the database area through the last decade.
  • Transbase® System Guide Transbase System Guide Version 8.1

    Transbase® System Guide Transbase System Guide Version 8.1

    Transbase® System Guide Transbase System Guide Version 8.1 Publication date 2018-05-14 Copyright © 2018 Transaction Software GmbH ALL RIGHTS RESERVED. While every precaution has been taken in the preparation of this document, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. Table of Contents About This Manual ...................................................................................................... vii 1. The Transbase DBMS ................................................................................................ 1 1.1. Transbase Packages ......................................................................................... 1 1.2. Transbase Platforms ........................................................................................ 1 1.2.1. Service Platforms .................................................................................. 2 1.2.2. Client Platforms ................................................................................... 2 2. Applications and Client Software ................................................................................. 3 2.1. Client Software ............................................................................................... 3 2.1.1. Database Browsers ................................................................................ 3 2.1.2. Programming Interfaces ......................................................................... 3 2.2. Database Connections
  • Case Study Transbase CD Customer AZ Direct Adress Data

    Case Study Transbase CD Customer AZ Direct Adress Data

    CASE STUDY Direct marketing supplier AZ Direct relies on Transbase® and Transaction Software's data- base expertise AZ Direct is one of Europe's leading direct marketing companies. AZ Direct offers innovative products and services relating to addresses, information, and information processing. Over almost four decades, the company has grown from a traditional direct marketing company to a universal information pro- vider for marketing. Many companies and agencies have used the ideas and solutions offered by AZ Direct to initiate a successful dialogue with their cus- tomers. AZ Direct is a subsidiary of arvato AG, the internationally networked media service provider of Bertelsmann AG. Bertelsmann AG has 42,000 employees worldwide. For further information, please visit: www.az-direct.de ConData arvato services is a division of AZ Direct. ConData maintains close relationships with top suppliers throughout the direct marketing value chain. For further information, please visit: www.arvato-condata.de Overview The ConTo application Industry: ConTo is a new concept in direct marketing. For the first time, this application Direct marketing combines one of the strongest and best-known consumer databases in Information service Germany with a powerful tool on one program DVD and two data DVDs. Clients receive a comprehensive and constantly updated database embedded Application: in the user-friendly ConTo program. The ConTo application is provided free of ConTo charge. With ConTo, users can easily work with a number of different feature combinations to assemble selections that are specifically customized for end Product: clients, or to optimize consumer address records by comparing them against Transbase® CD reference records.
  • Kiev, Ukraine, September 14-17, 1993)

    Kiev, Ukraine, September 14-17, 1993)

    DOCUMENT RESUME ED 367 314 IR 016 592 AUTHOR Petrushin, V., Ed.; Dovgiallo, A., Ed. TITLE Computer Technologies in Education. Proceedinga of the International Conference on Computer Technologies in Education (Kiev, Ukraine, September 14-17, 1993). PUB DATE 93 NOTE 217p. PUB TYPE Collected Works Conference Proceedings (021) EDRS PRICE MF01/PC09 Plus Postage. DESCRIPTORS Artificial Intelligence; *Computer Assisted Instruction; Computer Literacy; Computer Simulation; Distance Education; Educational Media; *Educational Technology; Elementary Secondary Education; Foreign Countries; Higher Education; Hypermedia; *Information Technology; Learning Processes; Models; Multimedia Instruction; Programming IDENTIFIERS Intelligent Tutoring Systems; Knowledge Representation; Multimedia Materials; Ukraine; United States ABSTRACT The conference reported in this document provides a meeting place for researchers from around the world, where the emphasis is on new ideas connected to computer technologies in education. This volume contains 140 extended abstracts selected by the program committee and organized into the following categories: (1) educational informational technologies (14 titles);(2) multimedia and hypermedia learning environments (24 titles); (3) distance learning (14 titles);(4) knowledge-based systems in education (23 titles);(3) intelligent tutoring systems and their components (19 titles); (6) student modelling (10 titles); (7) simulation environments and teaching science (15 titles); (8) teaching languages and humanities (7 titles); and
  • Transbase® CD

    Transbase® CD

    PRODUCT SHEET Publishing Central Databases Widely with Transbase® CD Transbase® CD is a special variant of the database system Transbase®, desi- gned and optimized for databases directly on CD-ROM or DVD-ROM. The range of use of Transbase® CD offers many advantages for the develop- ment and the operration of applications on CD and DVD. Transbase® CD provides for full SQL functionality (SQL 2003) Transbase® CD permits space saving distribution of database contents on CD/DVD-ROM Transbase® CD runs directly with the compressed database contents, thus minimizing local memory requirements. The integrated encryption and the privilege concepts provide an effective security for sensitive stored data. Transbase® CD offers the possibility to continuously update the data via internet using its update mechanism. Transbase® CD is perfect for applications which distribute central databases widely for autonomous operation. Publication of data on CD/DVD-ROM CD-ROM and DVD-ROM remain widely used for distributing information. These media prove themselves to be an important channel for distribution, even in the age of intensive internet usage. These media guarantee an auto- nomous use of the information in the event that use on-line is restricted or even impossible. This is an important aspect which has particular relevance to product information such as technical handbooks, service information and catalogues Database operation direct on CD/DVD-ROM For a long time SQL was considered too cumbersome and complex for these relatively slow media. Transbase® CD, however, is convincing with its full SQL functionality and impressive speed and performance. Page 1 of 6 PRODUCT SHEET This permits not only simple and clear programming, but also the immediate use of all the more pleasing features of SQL database systems: Automatic net- workability, multi-user support, data security monitored by the system, trans- actions and the powerful descriptive query language SQL.
  • Modelo Para a Gestão E Controlo Da Entrega Na Distribuição Logística Da Empresa Transbase

    Modelo Para a Gestão E Controlo Da Entrega Na Distribuição Logística Da Empresa Transbase

    Projecto Mestrado em Computação Móvel Modelo para a gestão e controlo da entrega na distribuição logística da empresa Transbase Sérgio Duarte Mota Leiria, Setembro de 2011 Projecto Mestrado em Comuptação Móvel Modelo para a gestão e controlo da entrega na distribuição logística da empresa Transbase Sérgio Duarte Mota Projecto de Mestrado realizada sob a orientação do Doutor Rui Rijo, Professor da Escola Superior de Tecnologia e Gestão do Instituto Politécnico de Leiria. Leiria, Setembro de 2011 “Acho que deve existir um mercado mundial para, talvez, cinco computadores.” Autor Thomas Watson, dirigente da IBM, 1943 Agradecimentos O sucesso na realização deste trabalho foi alcançado com o apoio e a colaboração de todos os que se enumeram de seguida. A todos gostaria de exprimir os meus mais sinceros agradecimentos e aqui reconhecer o seu contributo. Aos meus familiares e amigos, em especial à minha mãe que sempre me motivou e incentivou. À Ana Ribeiro que sempre me acompanhou nas diversas noites e fins de semana que passei a trabalhar, dando-me energia para continuar. Ao meu orientador Professor Doutor Rui Pedro Charters Lopes Rijo, pela sua disponibilidade, incentivo e motivação. A todos os colaboradores da Transbase que estiveram envolvidos na elaboração deste trabalho e sempre se encontraram disponíveis para me auxiliarem. O meu muito obrigado a todos. ix Resumo A evolução acentuada das tecnologias de informação e comunicação, que se tem vindo a assistir nas últimas décadas, tem permitido que estas se encontrem implementadas em praticamente todas as áreas de negócio. Assim, é com naturalidade que as tecnologias de informação tenham sido adoptadas na logística empresarial.
  • Modern B-Tree Techniques Contents

    Modern B-Tree Techniques Contents

    Foundations and TrendsR in Databases Vol. 3, No. 4 (2010) 203–402 c 2011 G. Graefe DOI: 10.1561/1900000028 Modern B-Tree Techniques By Goetz Graefe Contents 1 Introduction 204 1.1 Perspectives on B-trees 204 1.2 Purpose and Scope 206 1.3 New Hardware 207 1.4 Overview 208 2 Basic Techniques 210 2.1 Data Structures 213 2.2 Sizes, Tree Height, etc. 215 2.3 Algorithms 216 2.4 B-trees in Databases 221 2.5 B-trees Versus Hash Indexes 226 2.6 Summary 230 3 Data Structures and Algorithms 231 3.1 Node Size 232 3.2 Interpolation Search 233 3.3 Variable-length Records 235 3.4 Normalized Keys 237 3.5 Prefix B-trees 239 3.6 CPU Caches 244 3.7 Duplicate Key Values 246 3.8 Bitmap Indexes 249 3.9 Data Compression 253 3.10 Space Management 256 3.11 Splitting Nodes 258 3.12 Summary 259 4 Transactional Techniques 260 4.1 Latching and Locking 265 4.2 Ghost Records 268 4.3 Key Range Locking 273 4.4 Key Range Locking at Leaf Boundaries 280 4.5 Key Range Locking of Separator Keys 282 4.6 Blink-trees 283 4.7 Latches During Lock Acquisition 286 4.8 Latch Coupling 288 4.9 Physiological Logging 289 4.10 Non-logged Page Operations 293 4.11 Non-logged Index Creation 295 4.12 Online Index Operations 296 4.13 Transaction Isolation Levels 300 4.14 Summary 304 5 Query Processing 305 5.1 Disk-order Scans 309 5.2 Fetching Rows 312 5.3 Covering Indexes 313 5.4 Index-to-index Navigation 317 5.5 Exploiting Key Prefixes 324 5.6 Ordered Retrieval 327 5.7 Multiple Indexes for a Single Table 329 5.8 Multiple Tables in a Single Index 333 5.9 Nested Queries and Nested Iteration 334