DOCSLIB.ORG

Distributed Computing on Event-Sourced Graphs

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Distributed Computing on Event-Sourced Graphs Distributed Computing on Event-Sourced Graphs Dissertation zur Erlangung des Doktorgrades Dr.rer.nat. der Fakultät für Ingenieurwissenschaften, Informatik und Psychologie der Universität Ulm vorgelegt von Benjamin Erb aus Bonn Institut für Verteilte Systeme Fakultät für Ingenieurwissenschaften, Informatik und Psychologie Universität Ulm 2019 DISTRIBUTEDCOMPUTING ONEVENT-SOURCEDGRAPHS BENJAMINERB Institut für Verteilte Systeme Universität Ulm DISTRIBUTEDCOMPUTING ONEVENT-SOURCEDGRAPHS DISSERATION zur Erlangung des Doktorgrades Dr. rer. nat. der Fakultät für Ingenieurwissenschaften, Informatik und Psychologie der Universität Ulm vorgelegt von BENJAMINERB aus Bonn, Deutschland 2 0 1 9 Amtierender Dekan Prof. Dr.-Ing. Maurits Ortmanns Gutachter Prof. Dr. rer. nat. Frank Kargl Prof. Dr. rer. nat. Dr. h. c. Kurt Rothermel Tag der Promotion 04.12.2019 kontakt [email protected] www.benjamin-erb.de ABSTRACT Modern applications with increasingly connected domain topologies require Motivation. processing and programming abstractions that reflect the network structure inherent to these applications. At the same time, data-intensive applications necessitate more and more online processing capabilities when consuming in- coming streams of events to execute continuous computations and provide fast results. Only very few systems have taken into account the combined chal- lenge of executing graph processing on a dynamically evolving graph. How- ever, this is a critical capability as timely computations enable reactive applica- tion behaviors upon graph changes. In addition, no existing system supports processing on a live graph and on past version of that evolving graph at the same time. The distinct characteristics of event processing and graph computing, as Problem statement. well as batch processing and live processing yield specific requirements for any computing platform that unifies these approaches. Solutions require (i) data management strategies to keep track of the continuous graph evolution, (ii) ap- propriate graph processing models that can simultaneously handle computa- tions and graph updates, and (iii) an efficient platform implementation that provides the necessary performance at runtime. To this end, this thesis suggests a combination of an adapted actor model, Approach. an event-sourced persistence layer, and a vertex-based, asynchronous live pro- gramming model. The event-sourced actor model enables highly concurrent computations in which the full application history is implicitly persisted. This model is then refined into a live graph processing model with a particular fo- cus on asynchronicity, liveness, and parallel execution support. At the same time, the use of event sourcing enables the model to reconstruct global and consistent graph representations from arbitrary points of the timeline. These graph representations form the basis for decoupled, non-live graph processing models. The chronograph platform represents an implementation of the event- Result. sourced graph model. The platform ingests external update streams and main- tains a live graph representation as part of a user-specified graph application. It thus enables live and offline computations on event-driven, history-aware graphs and supports different processing models on the evolving graph. This thesis provides the following contributions: (i) a distributed comput- Conclusion & ing approach with history support based on the actor model and event sourc- contributions. ing, as wall as corresponding and supplementary concepts, (ii) a data man- agement approach for evolving graphs that builds on the event-sourced actor model, (iii) a set of novel and adapted programming and processing models that integrate well with event-sourced graphs, (iv) a distributed platform archi- tecture that implements the event-sourced graph model; and (v) an evaluation framework for such live graph processing systems. ZUSAMMENFASSUNG Moderne Software-Systeme bilden immer komplexere Szenarien mit zuneh- Motivation. mend vernetzteren Anwendungsdaten ab. Dies erfordert neue Verabreitungs- modelle, welche die Vernetzung der Daten miteinbezieht. Gleichzeitig erfor- dern datenintensive Anwendungen vermehrt eine echtzeitnahe Verarbeitung, um auf eingehenden Datenströmen kontinuierliche Berechnungen mit zeitna- hen Berechnungen durchzuführen. Kaum ein System erlaubt bisher die Kom- bination beider Ansätze – kontinuierliche Berechnungen auf Graphen, die sich zugleich auf Basis von externen Ereignisströmen verändern. Dies ist je- doch eine wichtige Fähigkeit, wenn Anwendungen kontinuierlich auf dynami- schen Graphen operieren und auf Veränderungen reagieren müssen. Zudem unterstützt keines der existierenden Systeme sowohl Berechnungen auf dem Live-Graph als auch auf älteren Versionen des Graphs. Ein solches System muss einerseits die Anforderungen von Event Proces- Problemstellung. sing und Graph Processing gegeneinander abwägen, als auch die Konzepte echtzeitnaher und stapelbasierter Berechnungen zusammenführen. Dies er- fordert (i) Datenverwaltungsstrategien zur Verfolgung des sich entwickelnden Graphs, (ii) angemessene Ausführungsmodelle, die gleichzeitig Berechnun- gen und Graph-Änderungen verarbeiten können sowie (iii) eine Implemen- tierung der Plattform, die zur Laufzeit die erforderliche Leistung bereitstellt. In dieser Arbeit wird die Kombination des Aktor-Modells, Event Sourcing, Eigener Ansatz. sowie einem vertex-basierten, asynchronen Programmiermodell vorgeschla- gen. Das resultierende Modell ermöglicht ermöglicht hochgradig parallele Be- rechnungen bei gleichzeitiger Persisitierung der vollständingen Zustandshis- torie de Aktoren. Dieses Modell dient als Grundlage für ein darauf aufbau- endes Verarbeitungsmodell für Graphen mit besonderem Schwerpunkt auf Asynchronität, Reaktionsvermögen und Unterstützung der parallelen Ausfüh- rung. Gleichzeitig ermöglicht der Einsatz von Event Sourcing die konsistente und globale Rekonstruktion beliebiger Graph-Zustände aus der Zeitleiste, wel- che als Grundlage für weitere, entkoppelte Berechnungen. chronograph implementiert als Plattform das konzeptuelle Modell und Ergebnis. führt benutzerdefinierte Graph-Anwendungen auf sich ändernden Graphen aus. Externe Datenströme dienen dabei als Quelle für Änderungen, welche vom System direkt eingepflegt werden. chronograph unterstützt gleichzeitig verschiedene Berechnungsmodelle, sowohl auf dem Live-Graphen, als auch auf älteren Graph-Versionen. Diese Arbeit bietet die folgenden Beiträge: (i) ein Konzept für verteilte, Zusammenfassung persistierte Berechnungen auf Basis des Aktor-Modells und Event Sourcing, der Beiträge. (ii) ein darauf aufbauender Data-Management-Ansatz für sich ändernde Gra- phen, (iii) eine Reihe von neuartigen und darauf angepassten Programmier- und Verarbeitungsmodellen für Graphen, (iv) eine verteilte Plattformarchi- tektur, die das konzeptuelle Modell implementiert und (v) ein Framework zur Evaluation solcher Systeme. PUBLICATIONS Parts of this thesis have been published in the following publications on dis- Peer-reviewed tributed graph computing and event sourcing. The list also contains other conference papers, workshop papers, and peer-reviewed publications of the author up to the time of submitting this journal articles by the thesis, including contributions on concurrent programming, distributed and author. mobile applications, scalable database systems, computer networks, network security, and psychological aspects of privacy: 1. Erb, B., Kaufmann, S., Schlecht, T., Schaub, F., Weber, M., “diretto: A Toolkit for Distributed Reporting and Collaboration.” In: Mensch & Computer 2011: überMEDIEN ÜBERmorgen. Chemnitz: Oldenbourg Verlag, 2011, pp. 151–160. doi: 10.1524/9783486712742.151. 2. Erb, B., Kargl, F., Domaschka, J., “Concurrent Programming in Web Applications.”In: it-Information Technology 56.3 (2014), pp. 119–126. doi: 10.1515/itit-2013-1030. 3. Erb, B., Kargl, F., “Combining Discrete Event Simulations and Event Sourcing.” In: Proceedings of the 7th International ICST Conference on Simulation Tools and Techniques. SIMUTools ’14. Lisbon, Portugal: ICST (Institute for Computer Sciences, Social-Informatics and Telecommuni- cations Engineering), 2014, pp. 51–55. doi: 10 . 4108 / icst . simutools . 2014.254624. 4. Erb, B., Kargl, F., “A Conceptual Model for Event-sourced Graph Com- puting.” In: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems. DEBS ’15. Oslo, Norway: ACM, 2015, pp. 352–355. doi: 10.1145/2675743.2776773. 5. Erb, B. Towards Distributed Processing on Event-sourced Graphs. Ex- tended Abstract. Doctoral Symposium of the 9th ACM International Conference on Distributed Event-Based Systems. Ulm University, 2015. doi: 10.13140/RG.2.1.3273.3289. 6. Erb, B., Habiger, G., Hauck, F. J., “On the Potential of Event Sourcing for Retroactive Actor-based Programming.” In: First Workshop on Pro- gramming Models and Languages for Distributed Computing. PMLDC ’16. Rome, Italy: ACM, 2016, pp. 1–5. doi: 10.1145/2957319.2957378. 7. Erb, B., Kargl, F., “Chronograph: A Distributed Platform for Event- Sourced Graph Computing.” In: Proceedings of the Posters and De- mos Session of the 17th International Middleware Conference. Middle- ware Posters and Demos ’16. Trento, Italy: ACM, 2016, pp. 15–16. doi: 10.1145/3007592.3007601. xii 8. Erb, B., Meißner, D., Pietron, J., Kargl, F., “Chronograph: A Distributed Processing Platform for Online and Batch Computations on Event- sourced Graphs.” In: Proceedings of the 11th ACM International Con-
Load more

Recommended publications
  • NUMA-Aware Thread Migration for High Performance NVMM File Systems
    NUMA-Aware Thread Migration for High Performance NVMM File Systems Ying Wang, Dejun Jiang and Jin Xiong SKL Computer Architecture, ICT, CAS; University of Chinese Academy of Sciences fwangying01, jiangdejun, [email protected] Abstract—Emerging Non-Volatile Main Memories (NVMMs) out considering the NVMM usage on NUMA nodes. Besides, provide persistent storage and can be directly attached to the application threads accessing file system rely on the default memory bus, which allows building file systems on non-volatile operating system thread scheduler, which migrates thread only main memory (NVMM file systems). Since file systems are built on memory, NUMA architecture has a large impact on their considering CPU utilization. These bring remote memory performance due to the presence of remote memory access and access and resource contentions to application threads when imbalanced resource usage. Existing works migrate thread and reading and writing files, and thus reduce the performance thread data on DRAM to solve these problems. Unlike DRAM, of NVMM file systems. We observe that when performing NVMM introduces extra latency and lifetime limitations. This file reads/writes from 4 KB to 256 KB on a NVMM file results in expensive data migration for NVMM file systems on NUMA architecture. In this paper, we argue that NUMA- system (NOVA [47] on NVMM), the average latency of aware thread migration without migrating data is desirable accessing remote node increases by 65.5 % compared to for NVMM file systems. We propose NThread, a NUMA-aware accessing local node. The average bandwidth is reduced by thread migration module for NVMM file system.
    [Show full text]
  • Facebook's Virtual Reality Ambitions Could Be Threatened by Court Order
    Mitch Shelowitz Quoted on Historic Oculus/Facebook $500 Million Copyright Infringement Case For more information about the case and the importance of software copyright registration, please contact Mitch at [email protected] and/or 212-655-9384. Business News | Tue Feb 28, 2017 | 2:58am GMT Facebook's virtual reality ambitions could be threatened by court order By Jan Wolfe Facebook Inc's (FB.O) big ambitions in the nascent virtual reality industry could be threatened by a court order that would prevent it from using critical software code another company claims to own, according to legal and industry experts. Last Thursday, video game publisher ZeniMax Media Inc asked a Dallas federal judge to issue an order barring Facebook unit Oculus from using or distributing the disputed code, part of the software development kit that Oculus provides to outside companies creating games for its Rift VR headset. A decision is likely a few months away, but intellectual property lawyers said ZeniMax has a decent chance of getting the order, which would mean Facebook faces a tough choice between paying a possibly hefty settlement or fighting on at risk of jeopardizing its position in the sector. For now, Facebook is fighting on. Oculus spokeswoman Tera Randall said last Thursday the company would challenge a $500 million jury verdict on Feb. 1 against Oculus and its co-founders Palmer Luckey and Brendan Iribe for infringing ZeniMax's copyrighted code and violating a non-disclosure agreement. Randall said Oculus would possibly file an appeal that would "allow us to put this litigation behind us." She did not respond to a request for comment for this article.
    [Show full text]
  • Unravel Data Systems Version 4.5
    UNRAVEL DATA SYSTEMS VERSION 4.5 Component name Component version name License names jQuery 1.8.2 MIT License Apache Tomcat 5.5.23 Apache License 2.0 Tachyon Project POM 0.8.2 Apache License 2.0 Apache Directory LDAP API Model 1.0.0-M20 Apache License 2.0 apache/incubator-heron 0.16.5.1 Apache License 2.0 Maven Plugin API 3.0.4 Apache License 2.0 ApacheDS Authentication Interceptor 2.0.0-M15 Apache License 2.0 Apache Directory LDAP API Extras ACI 1.0.0-M20 Apache License 2.0 Apache HttpComponents Core 4.3.3 Apache License 2.0 Spark Project Tags 2.0.0-preview Apache License 2.0 Curator Testing 3.3.0 Apache License 2.0 Apache HttpComponents Core 4.4.5 Apache License 2.0 Apache Commons Daemon 1.0.15 Apache License 2.0 classworlds 2.4 Apache License 2.0 abego TreeLayout Core 1.0.1 BSD 3-clause "New" or "Revised" License jackson-core 2.8.6 Apache License 2.0 Lucene Join 6.6.1 Apache License 2.0 Apache Commons CLI 1.3-cloudera-pre-r1439998 Apache License 2.0 hive-apache 0.5 Apache License 2.0 scala-parser-combinators 1.0.4 BSD 3-clause "New" or "Revised" License com.springsource.javax.xml.bind 2.1.7 Common Development and Distribution License 1.0 SnakeYAML 1.15 Apache License 2.0 JUnit 4.12 Common Public License 1.0 ApacheDS Protocol Kerberos 2.0.0-M12 Apache License 2.0 Apache Groovy 2.4.6 Apache License 2.0 JGraphT - Core 1.2.0 (GNU Lesser General Public License v2.1 or later AND Eclipse Public License 1.0) chill-java 0.5.0 Apache License 2.0 Apache Commons Logging 1.2 Apache License 2.0 OpenCensus 0.12.3 Apache License 2.0 ApacheDS Protocol
    [Show full text]
  • Artificial Intelligence for Understanding Large and Complex
    Artificial Intelligence for Understanding Large and Complex Datacenters by Pengfei Zheng Department of Computer Science Duke University Date: Approved: Benjamin C. Lee, Advisor Bruce M. Maggs Jeffrey S. Chase Jun Yang Dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science in the Graduate School of Duke University 2020 Abstract Artificial Intelligence for Understanding Large and Complex Datacenters by Pengfei Zheng Department of Computer Science Duke University Date: Approved: Benjamin C. Lee, Advisor Bruce M. Maggs Jeffrey S. Chase Jun Yang An abstract of a dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science in the Graduate School of Duke University 2020 Copyright © 2020 by Pengfei Zheng All rights reserved except the rights granted by the Creative Commons Attribution-Noncommercial Licence Abstract As the democratization of global-scale web applications and cloud computing, under- standing the performance of a live production datacenter becomes a prerequisite for making strategic decisions related to datacenter design and optimization. Advances in monitoring, tracing, and profiling large, complex systems provide rich datasets and establish a rigorous foundation for performance understanding and reasoning. But the sheer volume and complexity of collected data challenges existing techniques, which rely heavily on human intervention, expert knowledge, and simple statistics. In this dissertation, we address this challenge using artificial intelligence and make the case for two important problems, datacenter performance diagnosis and datacenter workload characterization. The first thrust of this dissertation is the use of statistical causal inference and Bayesian probabilistic model for datacenter straggler diagnosis.
    [Show full text]
  • Characterizing, Modeling, and Benchmarking Rocksdb Key-Value
    Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook Zhichao Cao, University of Minnesota, Twin Cities, and Facebook; Siying Dong and Sagar Vemuri, Facebook; David H.C. Du, University of Minnesota, Twin Cities https://www.usenix.org/conference/fast20/presentation/cao-zhichao This paper is included in the Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST ’20) February 25–27, 2020 • Santa Clara, CA, USA 978-1-939133-12-0 Open access to the Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST ’20) is sponsored by Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook Zhichao Cao†‡ Siying Dong‡ Sagar Vemuri‡ David H.C. Du† †University of Minnesota, Twin Cities ‡Facebook Abstract stores is still challenging. First, there are very limited studies of real-world workload characterization and analysis for KV- Persistent key-value stores are widely used as building stores, and the performance of KV-stores is highly related blocks in today’s IT infrastructure for managing and storing to the workloads generated by applications. Second, the an- large amounts of data. However, studies of characterizing alytic methods for characterizing KV-store workloads are real-world workloads for key-value stores are limited due to different from the existing workload characterization stud- the lack of tracing/analyzing tools and the difficulty of collect- ies for block storage or file systems. KV-stores have simple ing traces in operational environments. In this paper, we first but very different interfaces and behaviors. A set of good present a detailed characterization of workloads from three workload collection, analysis, and characterization tools can typical RocksDB production use cases at Facebook: UDB (a benefit both developers and users of KV-stores by optimizing MySQL storage layer for social graph data), ZippyDB (a dis- performance and developing new functions.
    [Show full text]
  • Myrocks in Mariadb
    MyRocks in MariaDB Sergei Petrunia <[email protected]> MariaDB Shenzhen Meetup November 2017 2 What is MyRocks ● #include <Yoshinori’s talk> ● This talk is about MyRocks in MariaDB 3 MyRocks lives in Facebook’s MySQL branch ● github.com/facebook/mysql-5.6 – Will call this “FB/MySQL” ● MyRocks lives there in storage/rocksdb ● FB/MySQL is easy to use if you are Facebook ● Not so easy if you are not :-) 4 FB/mysql-5.6 – user perspective ● No binaries, no packages – Compile yourself from source ● Dependencies, etc. ● No releases – (Is the latest git revision ok?) ● Has extra features – e.g. extra counters “confuse” monitoring tools. 5 FB/mysql-5.6 – dev perspective ● Targets a CentOS-type OS – Compiler, cmake version, etc. – Others may or may not [periodically] work ● MariaDB/Percona file pull requests to fix ● Special command to compile – https://github.com/facebook/mysql-5.6/wiki/Build-Steps ● Special command to run tests – Test suite assumes a big machine ● Some tests even a release build 6 Putting MyRocks in MariaDB ● Goals – Wider adoption – Ease of use – Ease of development – Have MyRocks in MariaDB ● Use it with MariaDB features ● Means – Port MyRocks into MariaDB – Provide binaries and packages 7 Status of MyRocks in MariaDB 8 Status of MyRocks in MariaDB ● MariaDB 10.2 is GA (as of May, 2017) ● It includes an ALPHA version of MyRocks plugin – Working to improve maturity ● It’s a loadable plugin (ha_rocksdb.so) ● Packages – Bintar, deb, rpm, win64 zip + MSI – deb/rpm have MyRocks .so and tools in a separate package. 9 Packaging for MyRocks in MariaDB 10 MyRocks and RocksDB library ● MyRocks is tied RocksDB@revno MariaDB – RocksDB is a github submodule – No compatibility with other versions MyRocks ● RocksDB is always compiled with RocksDB MyRocks S Z n ● l i And linked-in statically a b p ● p Distros have a RocksDB package y – Not using it.
    [Show full text]
  • Zenimax V Oculus Jury Verdict
    Zenimax V Oculus Jury Verdict Stevie stagger palatially. Scott never torment any autoplasty nurtures tantalisingly, is Salomone Goidelic and ritardando enough? Is Perry unimplored when Bennett worth mistrustfully? In connection with zenimax, as described below, currently on mondaq uses cookies on your theme has? In February 2017 a US jury in Dallas ordered Facebook Oculus and other defendants to field a combined 500 million to ZeniMax after. Facebook on Losing Side of 500M Virtual Reality Headset. We had just that leases could do. Receive email alerts for new posts. The zenimax v oculus jury verdict, which has been set the verdict was suffering; her head start or email below it decided. Oculus must pay Zenimax half a billion dollars as manual case. Ceo mark zuckerberg owes a sympathetic face. West Bengal Elections 2021 Bengaluru News IND vs AUS 3rd Test. Today bracket has posted a lengthy response now my case has concluded. Clicking the title she will take you to the source define the post. Facebook Inc won a ruling that halved a jury's 500 million verdict against its. Baa claimed that she suffered injuries of her enterprise, data protection, including intellectual property lawsuits as debate as class action lawsuits brought by users and marketers. Sporting Goods, Dallas Division. With mock judge ruling that Rift sales should be allowed to what and. Carmack could connect some newer cooler stuff does fine. Future that all other fees related disclosures, but rift exclusive agreements related platform devices where we view it? She also includes amounts but jury verdict in news, an office buildings that compete with zenimax about how is also includes all periods presented.
    [Show full text]
  • High-Performance Play: the Making of Machinima
    High-Performance Play: The Making of Machinima Henry Lowood Stanford University <DRAFT. Do not cite or distribute. To appear in: Videogames and Art: Intersections and Interactions, Andy Clarke and Grethe Mitchell (eds.), Intellect Books (UK), 2005. Please contact author, [email protected], for permission.> Abstract: Machinima is the making of animated movies in real time through the use of computer game technology. The projects that launched machinima embedded gameplay in practices of performance, spectatorship, subversion, modification, and community. This article is concerned primarily with the earliest machinima projects. In this phase, DOOM and especially Quake movie makers created practices of game performance and high-performance technology that yielded a new medium for linear storytelling and artistic expression. My aim is not to answer the question, “are games art?”, but to suggest that game-based performance practices will influence work in artistic and narrative media. Biography: Henry Lowood is Curator for History of Science & Technology Collections at Stanford University and co-Principal Investigator for the How They Got Game Project in the Stanford Humanities Laboratory. A historian of science and technology, he teaches Stanford’s annual course on the history of computer game design. With the collaboration of the Internet Archive and the Academy of Machinima Arts and Sciences, he is currently working on a project to develop The Machinima Archive, a permanent repository to document the history of Machinima moviemaking. A body of research on the social and cultural impacts of interactive entertainment is gradually replacing the dismissal of computer games and videogames as mindless amusement for young boys. There are many good reasons for taking computer games1 seriously.
    [Show full text]
  • Dmon: Efficient Detection and Correction of Data Locality
    DMon: Efficient Detection and Correction of Data Locality Problems Using Selective Profiling Tanvir Ahmed Khan and Ian Neal, University of Michigan; Gilles Pokam, Intel Corporation; Barzan Mozafari and Baris Kasikci, University of Michigan https://www.usenix.org/conference/osdi21/presentation/khan This paper is included in the Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation. July 14–16, 2021 978-1-939133-22-9 Open access to the Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation is sponsored by USENIX. DMon: Efficient Detection and Correction of Data Locality Problems Using Selective Profiling Tanvir Ahmed Khan Ian Neal Gilles Pokam Barzan Mozafari University of Michigan University of Michigan Intel Corporation University of Michigan Baris Kasikci University of Michigan Abstract cally at run time. In fact, as we (§6.2) and others [2,15,20,27] Poor data locality hurts an application’s performance. While demonstrate, compiler-based techniques can sometimes even compiler-based techniques have been proposed to improve hurt performance when the assumptions made by those heuris- data locality, they depend on heuristics, which can sometimes tics do not hold in practice. hurt performance. Therefore, developers typically find data To overcome the limitations of static optimizations, the locality issues via dynamic profiling and repair them manually. systems community has invested substantial effort in devel- Alas, existing profiling techniques incur high overhead when oping dynamic profiling tools [28,38, 57,97, 102]. Dynamic used to identify data locality problems and cannot be deployed profilers are capable of gathering detailed and more accurate in production, where programs may exhibit previously-unseen execution information, which a developer can use to identify performance problems.
    [Show full text]
  • Real-Time LSM-Trees for HTAP Workloads
    Real-Time LSM-Trees for HTAP Workloads Hemant Saxena Lukasz Golab University of Waterloo University of Waterloo [email protected] [email protected] Stratos Idreos Ihab F. Ilyas Harvard University University of Waterloo [email protected] [email protected] ABSTRACT We observe that a Log-Structured Merge (LSM) Tree is a natu- Real-time data analytics systems such as SAP HANA, MemSQL, ral fit for a lifecycle-aware storage engine. LSM-Trees are widely and IBM Wildfire employ hybrid data layouts, in which dataare used in key-value stores (e.g., Google’s BigTable and LevelDB, Cas- stored in different formats throughout their lifecycle. Recent data sandra, Facebook’s RocksDB), RDBMSs (e.g., Facebook’s MyRocks, are stored in a row-oriented format to serve OLTP workloads and SQLite4), blockchains (e.g., Hyperledger uses LevelDB), and data support high data rates, while older data are transformed to a stream and time-series databases (e.g., InfluxDB). While Cassandra column-oriented format for OLAP access patterns. We observe that and RocksDB can simulate columnar storage via column families, a Log-Structured Merge (LSM) Tree is a natural fit for a lifecycle- we are not aware of any lifecycle-aware LSM-Trees in which the aware storage engine due to its high write throughput and level- storage layout can change throughout the lifetime of the data. We oriented structure, in which records propagate from one level to fill this gap in our work, by extending the capabilities ofLSM- the next over time. To build a lifecycle-aware storage engine using based systems to efficiently serve real-time analytics and HTAP an LSM-Tree, we make a crucial modification to allow different workloads.
    [Show full text]
  • UNITED STATES DISTRICT COURT NORTHERN DISTRICT of TEXAS DALLAS DIVISION ZENIMAX MEDIA INC. and ID SOFTWARE LLC, Plaintiffs, V. O
    Case 3:14-cv-01849-K Document 1012 Filed 05/05/17 Page 1 of 35 PageID 49652 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF TEXAS DALLAS DIVISION ZENIMAX MEDIA INC. and ID SOFTWARE LLC, Case No.: 3:14-cv-01849-K Plaintiffs, Hon. Ed Kinkeade v. OCULUS VR, LLC, PALMER LUCKEY, FACEBOOK, INC., BRENDAN IRIBE and JOHN CARMACK, Defendants. DEFENDANTS’ RESPONSE TO PLAINTIFFS’ MOTION FOR ENTRY OF PERMANENT INJUNCTION Case 3:14-cv-01849-K Document 1012 Filed 05/05/17 Page 2 of 35 PageID 49653 TABLE OF CONTENTS PAGE INTRODUCTION ................................................................................................................................ 1 ARGUMENT ...................................................................................................................................... 2 I. ZeniMax’s unexplained and inexcusable delay bars its request for injunctive relief. ................................................................................................................................... 2 II. ZeniMax cannot satisfy any of the four injunction factors. ................................................ 5 A. ZeniMax’s claimed injuries are not irreparable. ..................................................... 5 1. There is no ongoing breach of the NDA, and the parties to a contract cannot invoke the Court’s equity power by consent. .................... 6 2. ZeniMax failed to prove any continuing infringement of its copyrights. ..................................................................................................
    [Show full text]
  • Head Tracking for the Oculus Rift
    Head Tracking for the Oculus Rift Steven M. LaValle1 Anna Yershova1 Max Katsev1 Michael Antonov Oculus VR, Inc. 19800 MacArthur Blvd Irvine, CA 92612 USA Abstract— We present methods for efficiently maintaining human head orientation using low-cost MEMS sensors. We particularly address gyroscope integration and compensa- tion of dead reckoning errors using gravity and magnetic fields. Although these problems have been well-studied, our performance criteria are particularly tuned to optimize user experience while tracking head movement in the Oculus Rift Development Kit, which is the most widely used virtual reality headset to date. We also present novel predictive tracking methods that dramatically reduce effective latency (time lag), which further improves the user experience. Experimental results are shown, along with ongoing research on positional tracking. I. INTRODUCTION In June 2012, Palmer Luckey’s prototype headset generated widespread enthusiasm and hopes for trans- formative virtual reality (VR) as John Carmack used it to develop and show a compelling Doom demo at the Electronic Entertainment Expo (E3). This convinced in- dustry leaders that a low-cost, high-fidelity VR experience could be achieved by leveraging MEMS sensing and video display technology from the smart phone industry. One important aspect of the design is a wide viewing angle, which greatly increases the sense of immersion over most prior headsets. Momentum has continued to build since that time, with broad media coverage progress on the potential for consumer VR, the dispersal of 50,000 Oculus Rift Development Kits, and the active cooperation of developers and researchers around the world. Although Fig. 1. The Oculus Rift Development Kit tracks head movement to originally targeted at the game industry, it has been finding present the correct virtual-world image to the eyes.
    [Show full text]