Author

Thomas Neumann

246 papers · 1982–2026

1982peak 20 in 20192026

Frequent co-authors

Papers

showing 1–25 of 246 RSS

How good are query optimizers, really?

Viktor Leis, Andrey Gubichev, Atanas Mirchev +3 more

Signal Processing · Proceedings of the VLDB Endowment · November 2015 · cited 673× · DOI ↗ · more like this ·

Finding a good join order is crucial for query performance. In this paper, we introduce the Join Order Benchmark (JOB) and experimentally revisit the main components in the classic query optimizer architecture using a complex, real-world data set and realistic multi-join queries. We investigate the quality of industrial-strength cardinality estimators and find that all estimators routinely produce large errors. We further show that while estimates are essential for finding a good join order, query performance is unsatisfactory if the query engine relies too heavily on these estimates. Using another set of experiments that measure the impact of the cost model, we find that it has much less influence on query performance than the cardinality estimates. Finally, we investigate plan enumeration techniques comparing exhaustive dynamic programming with heuristic algorithms and find that…

HyPer: A hybrid OLTP&amp;OLAP main memory database system based on virtual memory snapshots

Alfons Kemper, Thomas Neumann

Computer Networks and Communications · April 2011 · cited 592× · DOI ↗ · more like this ·

The two areas of online transaction processing (OLTP) and online analytical processing (OLAP) present different challenges for database architectures. Currently, customers with high rates of mission-critical transactions have split their data into two separate systems, one database for OLTP and one so-called data warehouse for OLAP. While allowing for decent transaction rates, this separation has many disadvantages including data freshness issues due to the delay caused by only periodically initiating the Extract Transform Load-data staging and excessive resource consumption due to maintaining two separate information systems. We present an efficient hybrid system, called HyPer, that can handle both OLTP and OLAP simultaneously by using hardware-assisted replication mechanisms to maintain consistent snapshots of the transactional data. HyPer is a main-memory database system that…

RDF-3X

Thomas Neumann, Gerhard Weikum

Artificial Intelligence · Proceedings of the VLDB Endowment · August 2008 · cited 561× · DOI ↗ · more like this ·

RDF is a data representation format for schema-free structured information that is gaining momentum in the context of Semantic-Web corpora, life sciences, and also Web 2.0 platforms. The "pay-as-you-go" nature of RDF and the flexible pattern-matching capabilities of its query language SPARQL entail efficiency and scalability challenges for complex queries including long join paths. This paper presents the RDF-3X engine, an implementation of SPARQL that achieves excellent performance by pursuing a RISC-style architecture with a streamlined architecture and carefully designed, puristic data structures and operations. The salient points of RDF-3X are: 1) a generic solution for storing and indexing RDF triples that completely eliminates the need for physical-design tuning, 2) a powerful yet simple query processor that leverages fast merge joins to the largest possible extent, and 3) a query…

Efficiently compiling efficient query plans for modern hardware

Thomas Neumann

Hardware and Architecture · Proceedings of the VLDB Endowment · June 2011 · cited 463× · DOI ↗ · more like this ·

As main memory grows, query performance is more and more determined by the raw CPU costs of query processing itself. The classical iterator style query processing technique is very simple and exible, but shows poor performance on modern CPUs due to lack of locality and frequent instruction mispredictions. Several techniques like batch oriented processing or vectorized tuple processing have been proposed in the past to improve this situation, but even these techniques are frequently out-performed by hand-written execution plans. In this work we present a novel compilation strategy that translates a query into compact and efficient machine code using the LLVM compiler framework. By aiming at good code and data locality and predictable branch layout the resulting code frequently rivals the performance of hand-written C++ code. We integrated these techniques into the HyPer main memory…

The adaptive radix tree: ARTful indexing for main-memory databases

Viktor Leis, Alfons Kemper, Thomas Neumann

Artificial Intelligence · April 2013 · cited 416× · DOI ↗ · more like this ·

Main memory capacities have grown up to a point where most databases fit into RAM. For main-memory database systems, index structure performance is a critical bottleneck. Traditional in-memory data structures like balanced binary search trees are not efficient on modern hardware, because they do not optimally utilize on-CPU caches. Hash tables, also often used for main-memory indexes, are fast but only support point queries. To overcome these shortcomings, we present ART, an adaptive radix tree (trie) for efficient indexing in main memory. Its lookup performance surpasses highly tuned, read-only search trees, while supporting very efficient insertions and deletions as well. At the same time, ART is very space efficient and solves the problem of excessive worst-case space consumption, which plagues most radix trees, by adaptively choosing compact and efficient data structures for…

Morsel-driven parallelism

Viktor Leis, Peter Boncz, Alfons Kemper +1 more

Signal Processing · Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands · 18 Jun 2014 · cited 255× · DOI ↗ · more like this ·

With modern computer architecture evolving, two problems conspire against the state-of-the-art approaches in parallel query execution: (i) to take advantage of many-cores, all query work must be distributed evenly among (soon) hundreds of threads in order to achieve good speedup, yet (ii) dividing the work evenly is difficult even with accurate data statistics due to the complexity of modern out-of-order cores. As a result, the existing approaches for plan-driven parallelism run into load balancing and context-switching bottlenecks, and therefore no longer scale. A third problem faced by many-core architectures is the decentralization of memory controllers, which leads to Non-Uniform Memory Access (NUMA). In response, we present the morsel-driven query execution framework, where scheduling becomes a fine-grained run-time task that is NUMA-aware. Morsel-driven query processing takes…

RadixSpline: A Single-Pass Learned Index

Andreas Kipf, Ryan Marcus, Alexander van Renen +4 more

cs.DB · April 2020 · cited 232× · arXiv ↗ · more like this ·

Recent research has shown that learned models can outperform state-of-the-art index structures in size and lookup performance. While this is a very promising result, existing learned structures are often cumbersome to implement and are slow to build. In fact, most approaches that we are aware of require multiple training passes over the data. We introduce RadixSpline (RS), a learned index that can be built in a single pass over the data and is competitive with state-of-the-art learned index models, like RMI, in size and lookup performance. We evaluate RS using the SOSD benchmark and show that it achieves competitive results on all datasets, despite the fact that it only has two parameters.

Scalable join processing on very large RDF graphs

Thomas Neumann, Gerhard Weikum

Artificial Intelligence · Max Planck Institute for Plasma Physics · 29 Jun 2009 · cited 232× · DOI ↗ · more like this ·

With the proliferation of the RDF data format, engines for RDF query processing are faced with very large graphs that contain hundreds of millions of RDF triples. This paper addresses the resulting scalability problems. Recent prior work along these lines has focused on indexing and other physical-design issues. The current paper focuses on join processing, as the fine-grained and schema-relaxed use of RDF often entails star- and chain-shaped join queries with many input streams from index scans.

Massively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems

Martina-Cezara Albutiu, Alfons Kemper, Thomas Neumann

cs.DB · 30 Jun 2012 · cited 231× · arXiv ↗ · more like this ·

Two emerging hardware trends will dominate the database system technology in the near future: increasing main memory capacities of several TB per server and massively parallel multi-core processing. Many algorithmic and control techniques in current database technology were devised for disk-based systems where I/O dominated the performance. In this work we take a new look at the well-known sort-merge join which, so far, has not been in the focus of research in scalable massively parallel multi-core data processing as it was deemed inferior to hash joins. We devise a suite of new massively parallel sort-merge (MPSM) join algorithms that are based on partial partition-based sorting. Contrary to classical sort-merge joins, our MPSM algorithms do not rely on a hard to parallelize final merge step to create one complete sort order. Rather they work on the independently created runs in…

Characteristic sets: Accurate cardinality estimation for RDF queries with multiple joins

Thomas Neumann, Guido Moerkotte

Computer Networks and Communications · April 2011 · cited 229× · DOI ↗ · more like this ·

Accurate cardinality estimates are essential for a successful query optimization. This is not only true for relational DBMSs but also for RDF stores. An RDF database consists of a set of triples and, hence, can be seen as a relational database with a single table with three attributes. This makes RDF rather special in that queries typically contain many self joins. We show that relational DBMSs are not well-prepared to perform cardinality estimation in this context. Further, there are hardly any special cardinality estimation methods for RDF databases. To overcome this lack of appropriate cardinality estimation methods, we introduce characteristic sets together with new cardinality estimation methods based upon them. We then show experimentally that the new methods are-in the RDF context-highly superior to the estimation methods employed by commercial DBMSs and by the open-source RDF…

Fast Serializable Multi-Version Concurrency Control for Main-Memory Database Systems

Thomas Neumann, Tobias Mühlbauer, Alfons Kemper

Computer Networks and Communications · 27 May 2015 · cited 223× · DOI ↗ · more like this ·

Multi-Version Concurrency Control (MVCC) is a widely employed concurrency control mechanism, as it allows for execution modes where readers never block writers. However, most systems implement only snapshot isolation (SI) instead of full serializability. Adding serializability guarantees to existing SI implementations tends to be prohibitively expensive.

Preventing bad plans by bounding the impact of cardinality estimation errors

Guido Moerkotte, Thomas Neumann, Gabriele Steidl

Computer Networks and Communications · Proceedings of the VLDB Endowment · August 2009 · cited 174× · DOI ↗ · more like this ·

Query optimizers rely on accurate estimations of the sizes of intermediate results. Wrong size estimations can lead to overly expensive execution plans. We first define the q-error to measure deviations of size estimates from actual sizes. The q-error enables the derivation of two important results: (1) We provide bounds such that if the q-error is smaller than this bound, the query optimizer constructs an optimal plan. (2) If the q-error is bounded by a number q , we show that the cost of the produced plan is at most a factor of q 4 worse than the optimal plan. Motivated by these findings, we next show how to find the best approximation under the q-error. These techniques can then be used to build synopsis for size estimates. Finally, we give some experimental results where we apply the developed techniques.

RadixSpline

Andreas Kipf, Ryan Marcus, Alexander van Renen +4 more

Artificial Intelligence · 3 Jun 2020 · cited 154× · DOI ↗ · more like this ·

The mixed workload CH-benCHmark

Richard L. Cole, Florian Funke, Leo Giakoumakis +9 more

Computer Networks and Communications · 13 Jun 2011 · cited 141× · DOI ↗ · more like this ·

While standardized and widely used benchmarks address either operational or real-time Business Intelligence (BI) workloads, the lack of a hybrid benchmark led us to the definition of a new, complex, mixed workload benchmark, called mixed workload CH-benCHmark. This benchmark bridges the gap between the established single-workload suites of TPC-C for OLTP and TPC-H for OLAP, and executes a complex mixed workload: a transactional workload based on the order entry processing of TPC-C and a corresponding TPC-H-equivalent OLAP query suite run in parallel on the same tables in a single database system. As it is derived from these two most widely used TPC benchmarks, the CH-benCHmark produces results highly relevant to both hybrid and classic single-workload systems.

Data Blocks

Harald Lang, Tobias Mühlbauer, Florian Funke +3 more

Computer Networks and Communications · VU Research Portal · 14 Jun 2016 · cited 137× · DOI ↗ · more like this ·

This work aims at reducing the main-memory footprint in high performance hybrid OLTP & OLAP databases, while retaining high query performance and transactional throughput. For this purpose, an innovative compressed columnar storage format for cold data, called Data Blocks is introduced. Data Blocks further incorporate a new light-weight index structure called Positional SMA that narrows scan ranges within Data Blocks even if the entire block cannot be ruled out. To achieve highest OLTP performance, the compression schemes of Data Blocks are very light-weight, such that OLTP transactions can still quickly access individual tuples. This sets our storage scheme apart from those used in specialized analytical databases where data must usually be bit-unpacked. Up to now, high-performance analytical systems use either vectorized query execution or just-in-time (JIT) query compilation. The…

x-RDF-3X

Thomas Neumann, Gerhard Weikum

Computer Networks and Communications · Proceedings of the VLDB Endowment · September 2010 · cited 137× · DOI ↗ · more like this ·

The RDF data model is gaining importance for applications in computational biology, knowledge sharing, and social communities. Recent work on RDF engines has focused on scalable performance for querying, and has largely disregarded updates. In addition to incremental bulk loading, applications also require online updates with flexible control over multi-user isolation levels and data consistency. The challenge lies in meeting these requirements while retaining the capability for fast querying. This paper presents a comprehensive solution that is based on an extended deferred-indexing method with integrated versioning. The version store enables time-travel queries that are efficiently processed without adversely affecting queries on the current data. For flexible consistency, transactional concurrency control is provided with options for either snapshot isolation or full serializability.…

Efficient top-k querying over social-tagging networks

Ralf Schenkel, Tom Crecelius, Mouna Kacimi +4 more

Information Systems · Max Planck Institute for Plasma Physics · 20 Jul 2008 · cited 134× · DOI ↗ · more like this ·

Online communities have become popular for publishing and searching content, as well as for finding and connecting to other users. User-generated content includes, for example, personal blogs, bookmarks, and digital photos. These items can be annotated and rated by different users, and these social tags and derived user-specific scores can be leveraged for searching relevant content and discovering subjectively interesting items. Moreover, the relationships among users can also be taken into consideration for ranking search results, the intuition being that you trust the recommendations of your close friends more than those of your casual acquaintances.

SOSD: A Benchmark for Learned Indexes

Andreas Kipf, Ryan Marcus, Alexander van Renen +4 more

cs.DB · 29 Nov 2019 · cited 119× · arXiv ↗ · more like this ·

A groundswell of recent work has focused on improving data management systems with learned components. Specifically, work on learned index structures has proposed replacing traditional index structures, such as B-trees, with learned models. Given the decades of research committed to improving index structures, there is significant skepticism about whether learned indexes actually outperform state-of-the-art implementations of traditional structures on real-world data. To answer this question, we propose a new benchmarking framework that comes with a variety of real-world datasets and baseline implementations to compare against. We also show preliminary results for selected index structures, and find that learned models indeed often outperform state-of-the-art implementations, and are therefore a promising direction for future research.

Managing Non-Volatile Memory in Database Systems

Alexander van Renen, Viktor Leis, Alfons Kemper +6 more

Computer Networks and Communications · 25 May 2018 · cited 118× · DOI ↗ · more like this ·

Non-volatile memory (NVM) is a new storage technology that combines the performance and byte addressability of DRAM with the persistence of traditional storage devices like flash (SSD). While these properties make NVM highly promising, it is not yet clear how to best integrate NVM into the storage layer of modern database systems. Two system designs have been proposed. The first is to use NVM exclusively, i.e., to store all data and index structures on it. However, because NVM has a higher latency than DRAM, this design can be less efficient than main-memory database systems. For this reason, the second approach uses a page-based DRAM cache in front of NVM. This approach, however, does not utilize the byte addressability of NVM and, as a result, accessing an uncached tuple on NVM requires retrieving an entire page.

The more the merrier

Manuel Then, Moritz Kaufmann, Fernando Chirigati +5 more

Computer Vision and Pattern Recognition · Proceedings of the VLDB Endowment · December 2014 · cited 118× · DOI ↗ · more like this ·

Graph analytics on social networks, Web data, and communication networks has been widely used in a plethora of applications. Many graph analytics algorithms are based on breadth-first search (BFS) graph traversal, which is not only time-consuming for large datasets but also involves much redundant computation when executed multiple times from different start vertices. In this paper, we propose Multi-Source BFS (MS-BFS), an algorithm that is designed to run multiple concurrent BFSs over the same graph on a single CPU core while scaling up as the number of cores increases. MS-BFS leverages the properties of small-world networks , which apply to many real-world graphs, and enables efficient graph traversal that: (i) shares common computation across concurrent BFSs; (ii) greatly reduces the number of random memory accesses; and (iii) does not incur synchronization costs. We demonstrate how…

The ART of practical synchronization

Viktor Leis, Florian Scheibner, Alfons Kemper +1 more

Computer Networks and Communications · June 2016 · cited 110× · DOI ↗ · more like this ·

The performance of transactional database systems is critically dependent on the efficient synchronization of in-memory data structures. The traditional approach, fine-grained locking, does not scale on modern hardware. Lock-free data structures, in contrast, scale very well but are extremely difficult to implement and often require additional indirections. In this work, we argue for a middle ground, i.e., synchronization protocols that use locking, but only sparingly. We synchronize the Adaptive Radix Tree (ART) using two such protocols, Optimistic Lock Coupling and Read-Optimized Write EXclusion (ROWEX). Both perform and scale very well while being much easier to implement than lock-free techniques.

Exploiting hardware transactional memory in main-memory databases

Viktor Leis, Alfons Kemper, Thomas Neumann

Computer Networks and Communications · March 2014 · cited 101× · DOI ↗ · more like this ·

So far, transactional memory-although a promising technique-suffered from the absence of an efficient hardware implementation. The upcoming Haswell microarchitecture from Intel introduces hardware transactional memory (HTM) in mainstream CPUs. HTM allows for efficient concurrent, atomic operations, which is also highly desirable in the context of databases. On the other hand HTM has several limitations that, in general, prevent a one-to-one mapping of database transactions to HTM transactions. In this work we devise several building blocks that can be used to exploit HTM in main-memory databases. We show that HTM allows to achieve nearly lock-free processing of database transactions by carefully controlling the data layout and the access patterns. The HTM component is used for detecting the (infrequent) conflicts, which allows for an optimistic, and thus very low-overhead execution of…

How good are modern spatial analytics systems?

Varun Pandey, Andreas Kipf, Thomas Neumann +1 more

Signal Processing · Proceedings of the VLDB Endowment · July 2018 · cited 99× · DOI ↗ · more like this ·

Spatial data is pervasive. Large amount of spatial data is produced every day from GPS-enabled devices such as cell phones, cars, sensors, and various consumer based applications such as Uber, location-tagged posts in Facebook, In-stagram, Snapchat, etc. This growth in spatial data coupled with the fact that spatial queries, analytical or transactional, can be computationally extensive has attracted enormous interest from the research community to develop systems that can efficiently process and analyze this data. In recent years a lot of spatial analytics systems have emerged. Existing work compares either limited features of these systems or the studies are outdated since new systems have emerged. In this work, we first explore the available modern spatial processing systems and then thoroughly compare them based on features and queries they support, using real-world datasets.

High-Speed Query Processing over High-Speed Networks

Wolf Roediger, Tobias Muehlbauer, Alfons Kemper +1 more

cs.DB · February 2015 · cited 97× · arXiv ↗ · more like this ·

Modern database clusters entail two levels of networks: connecting CPUs and NUMA regions inside a single server in the small and multiple servers in the large. The huge performance gap between these two types of networks used to slow down distributed query processing to such an extent that a cluster of machines actually performed worse than a single many-core server. The increased main-memory capacity of the cluster remained the sole benefit of such a scale-out. The economic viability of high-speed interconnects such as InfiniBand has narrowed this performance gap considerably. However, InfiniBand's higher network bandwidth alone does not improve query performance as expected when the distributed query engine is left unchanged. The scalability of distributed query processing is impaired by TCP overheads, switch contention due to uncoordinated communication, and load imbalances resulting…

Dynamic programming strikes back

Guido Moerkotte, Thomas Neumann

Computer Networks and Communications · Max Planck Institute for Plasma Physics · 9 Jun 2008 · cited 97× · DOI ↗ · more like this ·

Two highly efficient algorithms are known for optimally ordering joins while\navoiding cross products:\nDPccp, which is based on dynamic programming, and Top-Down Partition Search, \nbased on memoization.\nBoth have two severe limitations:\nThey handle only (1) simple (binary) join predicates and (2) inner joins.\nHowever, real queries may contain complex join predicates, involving more than \ntwo relations,\nand outer joins as well as other non-inner joins.\n\nTaking the most efficient known join-ordering algorithm, DPccp, as a starting \npoint,\nwe first develop a new algorithm, DPhyp,\nwhich is capable to handle complex join predicates efficiently.\nWe do so by modeling the query graph as a (variant of a) hypergraph and then \nreason about its\nconnected subgraphs.\nThen, we present a technique to exploit this capability to efficiently handle\nthe widest class of non-inner joins…