arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Adam Kalai

22 papers · 1999–2024

1999: 22000: 12001: 02002: 22003: 02004: 02005: 12006: 12007: 02008: 12009: 02010: 02011: 02012: 02013: 12014: 22015: 02016: 42017: 32018: 12019: 02020: 12021: 12022: 02023: 02024: 1
1999peak 4 in 20162024

Frequent co-authors

Papers

showing 1–22 of 22 RSS

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

Tolga Bolukbasi, Kai-Wei Chang, James Zou +2 more

cs.CL · · cited 3,901× · arXiv ↗ · more like this ·

The blind application of machine learning runs the risk of amplifying biases present in data. Such a danger is facing us with word embedding, a popular framework to represent text data as vectors which has been used in many machine learning and natural language processing tasks. We show that even word embeddings trained on Google News articles exhibit female/male gender stereotypes to a disturbing extent. This raises concerns because their widespread use, as we describe, often tends to amplify these biases. Geometrically, gender bias is first shown to be captured by a direction in the word embedding. Second, gender neutral words are shown to be linearly separable from gender definition words in the word embedding. Using these properties, we provide a methodology for modifying an embedding to remove gender stereotypes, such as the association between between the words receptionist and…

Noise-Tolerant Learning, the Parity Problem, and the Statistical Query Model

Avrim Blum, Adam Kalai, Hal Wasserman

cs.LG · · cited 827× · arXiv ↗ · more like this ·

We describe a slightly sub-exponential time algorithm for learning parity functions in the presence of random classification noise. This results in a polynomial-time algorithm for the case of parity functions that depend on only the first O(log n log log n) bits of input. This is the first known instance of an efficient noise-tolerant algorithm for a concept class that is provably not learnable in the Statistical Query model of Kearns. Thus, we demonstrate that the set of problems learnable in the statistical query model is a strict subset of those problems learnable in the presence of noise in the PAC model. In coding-theory terms, what we give is a poly(n)-time algorithm for decoding linear k by n codes in the presence of random noise for the case of k = c log n loglog n for some c > 0. (The case of k = O(log n) is trivial since one can just individually check each of the 2^k possible…

Quantifying and Reducing Stereotypes in Word Embeddings

Tolga Bolukbasi, Kai-Wei Chang, James Zou +2 more

cs.CL · · cited 64× · arXiv ↗ · more like this ·

Machine learning algorithms are optimized to model statistical properties of the training data. If the input data reflects stereotypes and biases of the broader society, then the output of the learning algorithm also captures these stereotypes. In this paper, we initiate the study of gender stereotypes in {\em word embedding}, a popular framework to represent text data. As their use becomes increasingly common, applications can inadvertently amplify unwanted stereotypes. We show across multiple datasets that the embeddings contain significant gender stereotypes, especially with regard to professions. We created a novel gender analogy task and combined it with crowdsourcing to systematically quantify the gender bias in a given embedding. We developed an efficient algorithm that reduces gender stereotype using just a handful of training examples while preserving the useful geometric…

Supervising Unsupervised Learning

Vikas K. Garg, Adam Kalai

cs.AI · · cited 31× · arXiv ↗ · more like this ·

We introduce a framework to leverage knowledge acquired from a repository of (heterogeneous) supervised datasets to new unsupervised datasets. Our perspective avoids the subjectivity inherent in unsupervised learning by reducing it to supervised learning, and provides a principled way to evaluate unsupervised algorithms. We demonstrate the versatility of our framework via simple agnostic bounds on unsupervised problems. In the context of clustering, our approach helps choose the number of clusters and the clustering algorithm, remove the outliers, and provably circumvent the Kleinberg's impossibility result. Experimental results across hundreds of problems demonstrate improved performance on unsupervised data with simple algorithms, despite the fact that our problems come from heterogeneous domains. Additionally, our framework lets us leverage deep networks to learn common features from…

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

Shyam Upadhyay, Kai-Wei Chang, Matt Taddy +2 more

cs.CL · · cited 26× · arXiv ↗ · more like this ·

Word embeddings, which represent a word as a point in a vector space, have become ubiquitous to several NLP tasks. A recent line of work uses bilingual (two languages) corpora to learn a different vector for each sense of a word, by exploiting crosslingual signals to aid sense identification. We present a multi-view Bayesian non-parametric algorithm which improves multi-sense word embeddings by (a) using multilingual (i.e., more than two languages) corpora to significantly improve sense embeddings beyond what one achieves with bilingual information, and (b) uses a principled approach to learn a variable number of senses per word, in a data-driven manner. Ours is the first approach with the ability to leverage multilingual corpora efficiently for multi-sense representation learning. Experiments show that multilingual training significantly improves performance over monolingual and…

Learn to Expect the Unexpected: Probably Approximately Correct Domain Generalization

Vikas K. Garg, Adam Kalai, Katrina Ligett +1 more

cs.LG · · cited 6× · arXiv ↗ · more like this ·

Domain generalization is the problem of machine learning when the training data and the test data come from different data domains. We present a simple theoretical model of learning to generalize across domains in which there is a meta-distribution over data distributions, and those data distributions may even have different supports. In our model, the training data given to a learning algorithm consists of multiple datasets each from a single domain drawn in turn from the meta-distribution. We study this model in three different problem settings---a multi-domain Massart noise setting, a decision tree multi-dataset setting, and a feature selection setting, and find that computationally efficient, polynomial-sample domain generalization is possible in each. Experiments demonstrate that our feature selection algorithm indeed ignores spurious correlations and improves generalization.

Counterfactual Language Model Adaptation for Suggesting Phrases

Kenneth C. Arnold, Kai-Wei Chang, Adam T. Kalai

cs.CL · · cited 6× · arXiv ↗ · more like this ·

Mobile devices use language models to suggest words and phrases for use in text entry. Traditional language models are based on contextual word frequency in a static corpus of text. However, certain types of phrases, when offered to writers as suggestions, may be systematically chosen more often than their frequency would predict. In this paper, we propose the task of generating suggestions that writers accept, a related but distinct task to making accurate predictions. Although this task is fundamentally interactive, we propose a counterfactual setting that permits offline training and evaluation. We find that even a simple language model can capture text characteristics that improve acceptability.

Actively Avoiding Nonsense in Generative Models

Steve Hanneke, Adam Kalai, Gautam Kamath +1 more

cs.LG · · cited 2× · arXiv ↗ · more like this ·

A generative model may generate utter nonsense when it is fit to maximize the likelihood of observed data. This happens due to "model error," i.e., when the true data generating distribution does not fit within the class of generative models being learned. To address this, we propose a model of active distribution learning using a binary invalidity oracle that identifies some examples as clearly invalid, together with random positive examples sampled from the true distribution. The goal is to maximize the likelihood of the positive examples subject to the constraint of (almost) never generating examples labeled invalid by the oracle. Guarantees are agnostic compared to a class of probability distributions. We show that, while proper learning often requires exponentially many queries to the invalidity oracle, improper distribution learning can be done using polynomially many queries.

Feature Multi-Selection among Subjective Features

Sivan Sabato, Adam Kalai

cs.LG · · cited 1× · arXiv ↗ · more like this ·

When dealing with subjective, noisy, or otherwise nebulous features, the "wisdom of crowds" suggests that one may benefit from multiple judgments of the same feature on the same object. We give theoretically-motivated `feature multi-selection' algorithms that choose, among a large set of candidate features, not only which features to judge but how many times to judge each one. We demonstrate the effectiveness of this approach for linear regression on a crowdsourced learning task of predicting people's height and weight from photos, using features such as 'gender' and 'estimated weight' as well as culturally fraught ones such as 'attractive'.

OpenAI o1 System Card

OpenAI, :, Aaron Jaech +262 more

cs.AI · · cited 0× · arXiv ↗ · more like this ·

The o1 model series is trained with large-scale reinforcement learning to reason using chain of thought. These advanced reasoning capabilities provide new avenues for improving the safety and robustness of our models. In particular, our models can reason about our safety policies in context when responding to potentially unsafe prompts, through deliberative alignment. This leads to state-of-the-art performance on certain benchmarks for risks such as generating illicit advice, choosing stereotyped responses, and succumbing to known jailbreaks. Training models to incorporate a chain of thought before answering has the potential to unlock substantial benefits, while also increasing potential risks that stem from heightened intelligence. Our results underscore the need for building robust alignment methods, extensively stress-testing their efficacy, and maintaining meticulous risk…

Efficient Learning with Arbitrary Covariate Shift

Adam Kalai, Varun Kanade

cs.LG · · cited 0× · arXiv ↗ · more like this ·

We give an efficient algorithm for learning a binary function in a given class C of bounded VC dimension, with training data distributed according to P and test data according to Q, where P and Q may be arbitrary distributions over X. This is the generic form of what is called covariate shift, which is impossible in general as arbitrary P and Q may not even overlap. However, recently guarantees were given in a model called PQ-learning (Goldwasser et al., 2020) where the learner has: (a) access to unlabeled test examples from Q (in addition to labeled samples from P, i.e., semi-supervised learning); and (b) the option to reject any example and abstain from classifying it (i.e., selective classification). The algorithm of Goldwasser et al. (2020) requires an (agnostic) noise tolerant learner for C. The present work gives a polynomial-time PQ-learning algorithm that uses an oracle to a…

This conf...

Brendan Juba, Adam T. Kalai, Sanjeev Khanna

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

Compression is a fundamental goal of both human language and digital communication, yet natural language is very different from compression schemes employed by modern computers. We partly explain this difference using the fact that information theory generally assumes a common prior probability distribution shared by the encoder and decoder, whereas human communication has to be robust to the fact that a speaker and listener may have different prior beliefs about what a speaker may say. We model this information-theoretically using the following question: what type of compression scheme would be effective when the encoder and decoder have (boundedly) different prior probability distributions. The resulting compression scheme resembles natural language to a far greater extent than existing digital communication protocols. We also use

Carnegie Mellon

Miaomiao Wen, Nancy Baym, Omer Tamuz +3 more

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

In this paper we explore Computer-Aided Humor (CAH), where a computer and a human collaborate to be humorous. CAH systems support people’s natural desire to be funny by helping them express their own idiosyncratic sense of humor. Artificial intelligence research has tried for years to create systems that are funny, but found the problem to be extremely hard. We show that by combining the strengths of a computer and a human, CAH can foster humor better than either alone. We present CAHOOTS, an online chat system that suggests humorous images to its users to include in the conversation. We compare CAHOOTS to a regular chat system, and to a system that automatically inserts funny images using an artificial humor-bot. Users report that CAHOOTS made their conversations more enjoyable and funny, and helped them to express their personal senses of humor. Computer-Aided Humor offers an example…

c Kluwer Academic Publishers Boston Manufactured in The Netherlands A Note on Learning from MultipleInstance Examples

Avrim Blum Avrim Cscmuedu, Adam Kalai Akalai Cscmuedu

Sociology and Political Science · · cited 0× · OpenAlex ↗ · more like this ·

We describe a simple reduction from the problem of PAC learning from multiple instance exam ples to that of PAC learning with one sided random classication noise Thus all concept classes learnable with one sided noise which includes all concepts learnable in the usual sided random noise model plus others such as the parity function are learnable from multiple instance exam ples We also describe a more e\ncient and somewhat technically more involved reduction to the Statistical Query model that results in a polynomial time algorithm for learning axis parallel rectangles with sample complexity Od r saving roughly a factor of r over the results of Auer et al

Beijing, China Trust-Based Recommendation Systems: an Axiomatic Approach

Reid Andersen, Christian Borgs, Jennifer Chayes +5 more

Sociology and Political Science · · cited 0× · OpenAlex ↗ · more like this ·

High-quality, personalized recommendations are a key feature in many online systems. Since these systems often have explicit knowledge of social network structures, the recommendations may incorporate this information. This paper focuses on networks that represent trust and recommendation systems that incorporate these trust relationships. The goal of a trust-based recommendation system is to generate personalized recommendations by aggregating the opinions of other users in the trust network. In analogy to prior work on voting and ranking systems, we use the axiomatic approach from the theory of social choice. We develop a set of five natural axioms that a trustbased recommendation system might be expected to satisfy. Then, we show that no system can simultaneously satisfy

ABSTRACT Boosting in the Presence of Noise

Adam Kalai, Rocco A. Servedio

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

Boosting algorithms are procedures that “boost ” low accuracy weak learning algorithms to achieve arbitrarily high accuracy. Over the past decade boosting has been widely used in practice and has become a major research topic in computational learning theory. In this paper we study boosting in the presence of random classification noise,giving both positive and negative results. We show that a modified version of a boosting algorithm due to Mansour and McAllester [14] can achieve accuracy arbitrarily close to the noise rate. We also give a matching lower bound by showing that no efficient black-box boosting algorithm can boost accuracy 1 beyond the noise rate (assuming that one-way functions exist). Finally,we consider a variant of the standard scenario for boosting in which the “weak learner ” satisfies a slightly stronger condition than the usual weak learning guarantee. We give an…

Graph model selection using maximum likelihood. ICML

Ivona Bezáková, Adam Kalai, Rahul Santhanam

Computer Vision and Pattern Recognition · · cited 0× · OpenAlex ↗ · more like this ·

In recent years, there has been a proliferation of theoretical graph models, e.g., preferential attachment and small-world models, motivated by real-world graphs such as the Internet topology. To address the natural question of which model is best for a particular data set, we propose a model selection criterion for graph models. Since each model is in fact a probability distribution over graphs, we suggest using Maximum Likelihood to compare graph models and select their parameters. Interestingly, for the case of graph models, computing likelihoods is a difficult algorithmic task. However, we design and implement MCMC algorithms for computing the maximum likelihood for four popular models: a power-law random graph model, a preferential attachment model, a small-world model, and a uniform random graph model. We hope that this novel use of ML will objectify comparisons between graph…

Efficient Estimators for Generalized Additive Models

Adam Kalai

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

Generalized additive models are a powerful generalization of linear and logistic regression models. In this paper we show that a natural regression graph learning algorithm efficiently learns generalized additive models. Efficiency is proven in two senses: the estimator’s future prediction accuracy approaches optimality at rate inverse polynomial in the size of the training data, and its runtime is polynomial in the size of the training data. Furthermore, the guarantees are nearly linear in terms of the dimensionality (number of regressors) of the problem, and hence the algorithm does not suffer from the “curse of dimensionality. ” The algorithm is a simple generalization of Mansour and McAllester’s classification algorithm that generates decision graphs, i.e., decision trees with merges. Our analysis is also viewed as defining a natural extension of the original classification boosting…

Geometric Algorithms for Online Optimization

Adam Kalai, Santosh Vempala

Computer Networks and Communications · · cited 0× · OpenAlex ↗ · more like this ·

In this paper, we consider a natural online version of linear optimization: the problem has to be solved repeatedly over a sequence of periods, where the objective direction for the upcoming period is unknown. This models online versions of optimization problems, such as max-cut, variants of clustering, and also the classic online binary search tree problem. We present algorithms for this problem that are (1 + epsilon)-competitive with the optimal static solution chosen in hindsight. Our algorithms and proofs are motivated by geometric considerations.

655 Efficient Pattern-Matching with Don't Cares

Adam Kalai

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

We present a randomized algorithm for the string matching with don't cares problem. Based on the simple fingerprint method of Kaxp and Rabin for ordinary string matching [4], our algorithm runs in time O(n log m) for a text of length n and a pattern of length m and is simpler and slightly faster than the previous algorithms [3, 5, 1]. 1 Introduction. We extend the simple randomized fingerprinting algorithm of Karp and Rabin [4] to the problem of string matching with don't cares. Our algorithm uses a single, simple convolution. This is optimal in the sense that

Omnivergent Stereo

Heung-yeung Shum, Adam Kalai, Steven M. Seitz

Computer Vision and Pattern Recognition · · cited 0× · OpenAlex ↗ · more like this ·

The notion of a virtual sensor for optimal 3D reconstruction is introduced. Instead of planar perspective images that collect many rays at a fixed viewpoint, omnivergent cameras collect a small number of rays at many different viewpoints. The resulting 2D manifold of rays are arranged into two multiple-perspective images for stereo reconstruction. We call such images omnivergent images, and the process of reconstructing the scene from such images omnivergent stereo. This procedure is shown to produce 3D scene models with minimal reconstruction error, due to the fact that for any point in the 3D scene, two rays with maximum vergence angle can be found in the omnivergent images. Furthermore, omnivergent images are shown to have horizontal epipolar lines, enabling the application of traditional stereo matching algorithms, without modification. Three types of omnivergent virtual sensors are…

Beating the Hold-Out: Bounds for K-fold and Progressive Cross-Validation

Avrim Blum, Adam Kalai, John Langford

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

The empirical error on a test set, the hold-out estimate, often is a more reliable estimate of generalization error than the observed error on the training set, the training estimate. K-fold cross validation is used in practice with the hope of being more accurate than the hold-out estimate without reducing the number of training examples. We argue that the k-fold estimate does in fact achieve this goal. Specifically, we show that for any nontrivial learning problem and learning algorithm that is insensitive to example ordering, the k-fold estimate is strictly more accurate than a single hold-out estimate on 1/k of the data, for 2 <k<n(k = n is leave-one-out), based on its variance and all higher moments. Previous bounds were termed sanitycheck because they compared the k-fold estimate to the training estimate and, further, restricted the VC dimension and required a notion…