arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Sam Lightstone

31 papers · 2001–2025

2001: 12002: 12003: 32004: 82005: 12006: 42007: 52008: 12009: 02010: 12011: 02012: 02013: 12014: 22015: 12016: 02017: 12018: 02019: 02020: 02021: 02022: 02023: 02024: 02025: 1
2001peak 8 in 20042025

Frequent co-authors

Papers

showing 1–25 of 31 RSS

DB2 with BLU acceleration

Vijayshankar Raman, Gopi K. Attaluri, Ronald Barber +9 more

Computer Networks and Communications · Proceedings of the VLDB Endowment · · cited 254× · DOI ↗ · more like this ·

DB2 with BLU Acceleration deeply integrates innovative new techniques for defining and processing column-organized tables that speed read-mostly Business Intelligence queries by 10 to 50 times and improve compression by 3 to 10 times, compared to traditional row-organized tables, without the complexity of defining indexes or materialized views on those tables. But DB2 BLU is much more than just a column store. Exploiting frequency-based dictionary compression and main-memory query processing technology from the Blink project at IBM Research - Almaden, DB2 BLU performs most SQL operations - predicate application (even range predicates and IN-lists), joins, and grouping - on the compressed values, which can be packed bit-aligned so densely that multiple values fit in a register and can be processed simultaneously via SIMD (single-instruction, multipledata) instructions. Designed and built…

Adaptive self-tuning memory in DB2

Adam Storm, Christian Garcia-Arellano, Sam Lightstone +2 more

Computer Networks and Communications · Very Large Data Bases · · cited 113× · DOI ↗ · more like this ·

DB2 for Linux, UNIX, and Windows Version 9.1 introduces the Self-Tuning Memory Manager (STMM), which provides adaptive self tuning of both database memory heaps and cumulative database memory allocation. This technology provides state-ofthe-art memory tuning combining control theory, runtime simulation modeling, cost-benefit analysis, and operating system resource analysis. In particular, the novel use of cost-benefit analysis and control theory techniques makes STMM a breakthrough technology in database memory management. The cost-benefit analysis allows STMM to tune memory between radically different memory consumers such as compiled statement cache, sort, and buffer pools. These methods allow for the fast convergence of memory settings while also providing stability in the presence of system noise. The tuning model has been found in numerous experiments to tune memory allocation as…

Memory-efficient hash joins

Ronald Barber, Guy M. Lohman, Ippokratis Pandis +6 more

Computer Networks and Communications · Proceedings of the VLDB Endowment · · cited 84× · DOI ↗ · more like this ·

We present new hash tables for joins, and a hash join based on them, that consumes far less memory and is usually faster than recently published in-memory joins. Our hash join is not restricted to outer tables that fit wholly in memory. Key to this hash join is a new concise hash table (CHT), a linear probing hash table that has 100% fill factor, and uses a sparse bitmap with embedded population counts to almost entirely avoid collisions. This bitmap also serves as a Bloom filter for use in multi-table joins. We study the random access characteristics of hash joins, and renew the case for non-partitioned hash joins. We introduce a variant of partitioned joins in which only the build is partitioned, but the probe is not, as this is more efficient for large outer tables than traditional partitioned joins. This also avoids partitioning costs during the probe, while at the same time…

Toward autonomic computing with DB2 universal database

Sam Lightstone, Guy M. Lohman, Danny Zilio

Computer Networks and Communications · ACM SIGMOD Record · · cited 50× · DOI ↗ · more like this ·

As the cost of both hardware and software falls due to technological advancements and economies of scale, the cost of ownership for database applications is increasingly dominated by the cost of people to manage them. Databases are growing rapidly in scale and complexity, while skilled database administrators (DBAs) are becoming rarer and more expensive. This paper describes the self-managing or autonomic technology in IBM's DB2 Universal Database® for UNIX and Windows to illustrate how self-managing technology can reduce complexity, helping to reduce the total cost of ownership (TCO) of DBMSs and improve system performance.

Using MIMO linear control for load balancing in computing systems

Yixin Diao, Joseph L. Hellerstein, A.J. Storm +4 more

Computer Networks and Communications · · cited 49× · DOI ↗ · more like this ·

Load balancing is widely used in computing systems as a way to optimize performance by equalizing loads to reduce delays, such as adjusting the size of memory pools to balance resource demands in a database management system. Load balancing is generally approached as a nonlinear constrained optimization in which dynamics are ignored. We approach load balancing differently - as a feedback controller design problem using a multiple input multiple output linear quadratic regulator (LQR) that achieves the constrained optimization objective. Such an approach allows us to exploit well-established techniques for handling disturbances (e.g., due to changes in workloads) and to incorporate the cost of control (e.g., throughput reductions due to resizing buffer pools) by properly selecting the LQR Q and R matrices. From studies of DB2 Universal Database Server using industry standard benchmarks…

Incorporating cost of control into the design of a load balancing controller

Yixin Diao, Joseph L. Hellerstein, A.J. Storm +4 more

Computer Networks and Communications · · cited 39× · DOI ↗ · more like this ·

Load balancing is widely used in computing systems as a way to optimize performance by reducing bottleneck utilizations, such as adjusting the size of buffer pools to balance resource demands in a database management system. Load balancing is generally approached as a constrained optimization problem in which only the benefits of load balancing are considered. However, the costs of control are important as well. Herein, we study the value of including in controller design the trade-off between the cost of transient imbalances in resource utilizations and the cost of changing resource allocations. An example of the latter are actions such as resizing buffer pools that can reduce throughputs. This is because requests for data in pools whose memory is reduced immediately have longer access times whereas requests for data in pools whose memory is increased must fill this memory with data…

Comparative studies of load balancing with control and optimization techniques

Yixin Diao, Chai Wah Wu, Joseph L. Hellerstein +8 more

Information Systems · · cited 37× · DOI ↗ · more like this ·

Load balancing is a widely used technique to optimizing distributed computing system performance. System response delays are reduced by equalizing the loads, such as adjusting memory pool sizes to balance disk access demands in a database management system. In this paper we formulate load balancing as a constrained optimization problem and investigate two load balancing controllers based on feedback control theory and optimization theory. We show the difference and equivalence between their design methods and criteria. Furthermore, our studies on a DB2 universal database server reveal their performance difference regarding to system noise and workload variations.

Benchmarking autonomic capabilities: promises and pitfalls

Aaron Brown, Joseph M. Hellerstein, Matt Hogstrom +4 more

Artificial Intelligence · · cited 37× · DOI ↗ · more like this ·

Benchmarks provide a way to quantify progress in a field. Our goal is to produce a suite of benchmarks covering the four categories of autonomic capabilities: self-configuring, self-healing, self-optimizing, and self-protecting (IBM, 2003). This is not an easy task, however, and in this paper we identify several of the challenges and pitfalls that must be confronted to extend benchmarking technology beyond its traditional basis in performance evaluation.

Automatic Database Configuration for DB2 Universal Database: Compressing Years of Performance Expertise into Seconds of Execution.

Eva Kwan, Sam Lightstone, K. Bernhard Schiefer +2 more

Computer Networks and Communications · BTW · · cited 31× · OpenAlex ↗ · more like this ·

This paper describes the DB2 Configuration Advisor, an expert tool for the configuration of DB2 databases. This advisor has shown dramatic results for tuning and configuring DB2 servers on UNIX and Windows platforms. The recognition of the essential need for administration and design tools has spurred renewed interest among leading relational database management system (RDBMS) vendors. The DB2 Configuration Advisor is a key feature in DB2's Autonomic Computing self-managing technology portfolio. This paper discusses the purpose and features of this expert advisor. Experimental results are presented with a description of the advisor's interfaces.

Throttling utilities in the IBM DB2 universal database server

S. Parekh, Kevin C. Rose, Yixin Diao +4 more

Computer Networks and Communications · · cited 26× · DOI ↗ · more like this ·

This paper describes a control system that provides the "utilities throttling" feature in the IBM/spl reg/ DBS /spl reg/ Universal Database/spl trade/ v8.1. Administrative utilities (e.g., file system and database backups, antivirus scan) are essential to the operation of production systems. Unfortunately, production work can be severely degraded by the concurrent execution of such utilities. Hence, it is desirable for the system to self-manage its utilities to limit their performance impact, with only high-level policy input from the administrator. We focus on policies of the form "there should be no more than an x% degradation of production work due to utility execution." We have designed a throttling mechanism called self-imposed sleep (SIS), which forces utilities to slow down their processing, by a configurable amount. We design a feedback control system based on online…

Foundations of Autonomic Computing Development

Sam Lightstone

Artificial Intelligence · · cited 25× · DOI ↗ · more like this ·

The complexity of modern middleware, and software solutions, is growing at an exponential rate. Only self-managing, or autonomic computing technology can reasonably stem the confusion this complexity brings to bear on human administrators. While much has been published on "architecture" and "function" for producing such systems, little has been written about the engineering of self-managing systems as a distinct paradigm. In this paper we suggest a straw-man for engineering of autonomic systems that is based on two essential tracks: a set of engineering principles that should guide the planning of autonomic systems and their interfaces and secondly a set of mathematical foundations upon which such systems can best be constructed. These foundational attributes are intended to guide the thinking of R&D organizations pursuing the development of autonomic computing capability. The role of…

Control Theory: a Foundational Technique for Self Managing Databases

Sam Lightstone, Maheswaran Surendra, Yixin Diao +5 more

Computer Networks and Communications · · cited 24× · DOI ↗ · more like this ·

Control theory is a well established discipline that has emerged from aeronautical, electrical, and mechanical engineering to provide a formal approach to building robust systems. While similar robustness concerns exist in database management systems, control theory is rarely used due to the lack of canonical control models and a dearth of control theory expertise among database researchers. We discuss our experience with using control theory to build self managing databases, showing experimental results, discussing pitfalls and limitations, and contrasting formal models against with feedback loops. While our experience indicates that control theory is a good paradigm for database self management, control theory should be used Judiciously since its techniques are not suited to all problems in database administration.

In-memory BLU acceleration in IBM's DB2 and dashDB: Optimized for modern workloads and hardware architectures

Ronald Barber, Guy M. Lohman, Vijayshankar Raman +3 more

Computer Networks and Communications · · cited 20× · DOI ↗ · more like this ·

Although the DRAM for main memories of systems continues to grow exponentially according to Moore's Law and to become less expensive, we argue that memory hierarchies will always exist for many reasons, both economic and practical, and in particular due to concurrent users competing for working memory to perform joins and grouping. We present the in-memory BLU Acceleration used in IBM's DB2 for Linux, UNIX, and Windows, and now also the dashDB cloud offering, which was designed and implemented from the ground up to exploit main memory but is not limited to what fits in memory and does not require manual management of what to retain in memory, as its competitors do. In fact, BLU Acceleration views memory as too slow, and is carefully engineered to work in higher levels of the system cache by keeping the data encoded and packed densely into bit-aligned vectors that can exploit SIMD…

Joins on encoded and partitioned data

Jae-Gil Lee, Gopi K. Attaluri, Ronald Barber +9 more

Artificial Intelligence · Proceedings of the VLDB Endowment · · cited 20× · DOI ↗ · more like this ·

Compression has historically been used to reduce the cost of storage, I/Os from that storage, and buffer pool utilization, at the expense of the CPU required to decompress data every time it is queried. However, significant additional CPU efficiencies can be achieved by deferring decompression as late in query processing as possible and performing query processing operations directly on the still-compressed data. In this paper, we investigate the benefits and challenges of performing joins on compressed (or encoded) data. We demonstrate the benefit of independently optimizing the compression scheme of each join column, even though join predicates relating values from multiple columns may require translation of the encoding of one join column into the encoding of the other. We also show the benefit of compressing "payload" data other than the join columns "on the fly," to minimize the…

Usability and design considerations for an autonomic relational database management system

Richard J. Telford, Randy Horman, Sam Lightstone +3 more

Information Systems · IBM Systems Journal · · cited 16× · DOI ↗ · more like this ·

Autonomic systems offer numerous advantages over non-autonomic systems, and many of these advantages relate to ease of use. The advantages regarding ease of use include reducing the number of low-level system administration tasks, simplifying the system administrator's interface, handling exceptions which would otherwise have resulted in system alerts, and the learning, by the system, of actions taken by the administrator. However, human intervention must still be factored in, and care must be taken in the design of autonomic systems not to make the system administrator's task more difficult. This paper examines the ease-of-use ramifications of autonomic computing in the context of relational databases in general, and of the IBM® DB2® Universal Database™ Version 8.1 autonomic computing system in particular.

Self-managing technology in IBM DB2 universal database

Daniel C. Zilio, Sam Lightstone, Kelly Lyons +1 more

Computer Networks and Communications · · cited 13× · DOI ↗ · more like this ·

As the cost of both hardware and software falls due to technological advancements and economies of scale, the cost of ownership for database applications is increasingly dominated by the cost of people to manage them. Databases are growing rapidly in scale and complexity, while skilled database administrators (DBAs) are becoming rarer and more expensive. The scope of responsibility of DBAs is indeed daunting. This paper describes the self-managing technology in IBM DB2 Universal Database to illustrate how self-managing technology can enhance the usability of enterprise middleware and reduce the total cost of ownership (TCO).

Autonomic features of the IBM DB2 universal database for linux, UNIX, and windows

Christian Garcia-Arellano, Sam Lightstone, Guy M. Lohman +2 more

Computer Networks and Communications · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · · cited 12× · DOI ↗ · more like this ·

Thevast majority of the world's structured data are now stored and managed by relational database management systems (RDBMSs). These systems provide powerful data management capabilities. However, as the power and functionality of these systems has grown, so has the complexity of their administration. In this paper, we show how the IBM DB2 Universal Database for Linux, UNIX, and Windows (DB2 UDB) product has exploited autonomic computing to reduce this administration complexity and become more self-managing. We survey the major autonomic computing features in the DB2 UDB product and describe the benefits, with experimental data in some cases

Making DB2Products Self-Managing: Strategies and Experiences.

Sam Lightstone, Guy M. Lohman, Peter J. Haas +5 more

Computer Networks and Communications · IEEE Data(base) Engineering Bulletin · · cited 12× · OpenAlex ↗ · more like this ·

This paper evaluates the impact of the DB2 Autonomic Computing project at the IBM Toronto Software Lab, Almaden Research Center, and Watson Research Center. It describes the key ideas behind the many self-managing features added to the IBM R©DB2 R©for Linux R©, UNIX R©, and Windows R©products, and evaluates the degree to which these features have been accepted by the DB2 user community. We offer lessons learned from this experience, our conclusions, and future directions for self-managing databases. 1

Trends in automating physical database design

Daniel C. Zilio, Sam Lightstone, Guy M. Lohman

Computer Networks and Communications · · cited 6× · DOI ↗ · more like this ·

A database administrator (DBA) must make many decisions about what is the best decision for a database and its users - including what is the best physical DB design decision. The DBA must determine what auxiliary data structures to create - such as indexes and materialized views - as well as the optimal way to partition each table in a shared-nothing partitioned system, to maximize workload performance while minimizing any increase in disk space or maintenance costs. Given the number of possible configurations, the possible interactions between each choice, and the associated constraints on the decision, the problem of selecting an optimal design is daunting for even the most skilled DBA. It has long been recognized that DBAs need tools to help them make these complex decisions, but until recently little progress was made. We present our vision for the future of automating physical DB…

The need for ease: development principles for successful autonomic computing projects

Sam Lightstone

Computer Networks and Communications · · cited 3× · DOI ↗ · more like this ·

Summary form only given. Autonomic computing has emerged as a paradigm for self managing IT systems to stem the tide of rapidly increasing administration costs in the face of rising IT system complexity. The development of self managing systems poses special challenges to research and development teams. This paper explores some of the development paradigms that lead to successful software projects in autonomic computing. Examples from IBM's DB2 UDB Autonomic Computing project is used so as to highlight recent successes illustrating software engineering principles, algorithmic/mathematical techniques such as online simulation and close loop adaptive control, and experimental results

Poster session: Simplifying business intelligence with a hybrid appliance: the IBM balanced warehouse

Haider Rizvi, Joyce Coleman, Sam Lightstone

Computer Networks and Communications · · cited 2× · DOI ↗ · more like this ·

Business Intelligence systems require a complex configuration of robust components, including analytical and warehousing software, database systems, servers, and storage. These systems are consistently at the frontier of the most complex query reporting and analysis engines over very large data sets. As a result such systems typically present significant administrative challenges and therefore require simplified deployment and administration. A key consideration is the balance between system resources, which if poorly configured can easily waste time and money. Two fundamental strategies have been used in the past to architect database systems for these environments: traditional relational database servers, with their concomitant configuration and design flexibility and complexity, and data warehouse appliances at the other extreme, with their superb ease of use but well known…

Making Big Data Simple with dashDB Local

Sam Lightstone, Russ Ohanian, Michael Haide +3 more

Computer Networks and Communications · · cited 1× · DOI ↗ · more like this ·

In this paper we introduce dashDB Local, a new Big Data & SQL warehousing technology from IBM designed to provide dramatic simplification over traditional data warehousing deployments, while offering next generation query performance and the analytic richness of the Apache Spark ecosystem. Designed as a Software Defined Technology, dashDB Local automatically adapts to hardware platforms to provide simplified deployments. In experiments one multiple workloads, we have achieved deployment times for large clusters in <; 30 minutes while providing dramatic workload performance speedups of several factors on multiple workloads under study compared to industry leading appliance technology and market leading data warehouses. We see dashDB Local providing five novel value propositions: 1. Improved deployment with automatic adaptation to the target hardware enabling fast deployment in minutes of…

MEET DB2

Reynold Xin, William McLaren, Patrick Dantressangle +3 more

Computer Networks and Communications · Proceedings of the VLDB Endowment · · cited 1× · DOI ↗ · more like this ·

Commercial databases compete for market share, which is composed of not only net-new sales to those purchasing a database for the first time, but also competitive "win-backs" and migrations. Database migration, or the act of moving both application code and its underlying database platform from one database to another, presents a serious administrative and application development challenge fraught with large manual costs. Migration is typically a high cost effort due to incompatibilities between database platforms. Incompatibilities are caused most often by product specific extensions to language support, procedural logic, DDL, and administrative interfaces. The migration evaluation is the first step in any competitive database migration process. Historically this has been a manual process, with the high costs and subjective results. This has led us to reexamine traditional practices…

Poster Session Problem definition for effective workload management

Adrian M. Teisanu, Sam Lightstone, Mariano P. Consens +3 more

Computer Networks and Communications · · cited 1× · DOI ↗ · more like this ·

The paper introduces the problem of designing dynamic workload management (WM) tools that are aware of the diversity of classes of users and their diverse access patterns. Our approach should be contrasted with the current WM tools and their ability of detecting performance degradations in accordance with the user's preference goals. We define the problem and suggest a formal definition that allows the further development of algorithms and architectures that allow the implementation of effective on-line database tuning strategies.