Author

Alexandre Passos

24 papers · 2002–2023

2002peak 6 in 20122023

Frequent co-authors

Papers

showing 1–24 of 24 RSS

Scikit-learn: Machine Learning in Python

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort +16 more

cs.LG · January 2012 · cited 95,273× · arXiv ↗ · more like this ·

Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.org.

PaLM 2 Technical Report

Rohan Anil, Andrew M. Dai, Orhan Firat +125 more

cs.CL · May 2023 · cited 1,563× · arXiv ↗ · more like this ·

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables…

Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

Arvind Neelakantan, Jeevan Shankar, Alexandre Passos +1 more

cs.CL · 24 Apr 2015 · cited 472× · arXiv ↗ · more like this ·

There is rising interest in vector-space word embeddings and their use in NLP, especially given recent methods for their fast estimation at very large scale. Nearly all this work, however, assumes a single vector per word type ignoring polysemy and thus jeopardizing their usefulness for downstream tasks. We present an extension to the Skip-gram model that efficiently learns multiple embeddings per word type. It differs from recent related work by jointly performing word sense discrimination and embedding learning, by non-parametrically estimating the number of senses per word type, and by its efficiency and scalability. We present new state-of-the-art results in the word similarity in context task and demonstrate its scalability by training with one machine on a corpus of nearly 1 billion tokens in less than 6 hours.

Large scale distributed neural network training through online distillation

Rohan Anil, Gabriel Pereyra, Alexandre Passos +3 more

cs.LG · April 2018 · cited 459× · arXiv ↗ · more like this ·

Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model. However, due to increased test-time cost (for ensembles) and increased complexity of the training pipeline (for distillation), these techniques are challenging to use in industrial settings. In this paper we explore a variant of distillation which is relatively straightforward to use as it does not require a complicated multi-stage setup or many new hyperparameters. Our first claim is that online distillation enables us to use extra parallelism to fit very large datasets about twice as fast. Crucially, we can still speed up training even after we have already reached the point at which additional parallelism provides no benefit for synchronous or asynchronous stochastic gradient descent. Two neural networks trained on disjoint subsets of the data can share knowledge…

Lexicon Infused Phrase Embeddings for Named Entity Resolution

Alexandre Passos, Vineet Kumar, Andrew McCallum

cs.CL · 22 Apr 2014 · cited 317× · arXiv ↗ · more like this ·

Most state-of-the-art approaches for named-entity recognition (NER) use semi supervised information in the form of word clusters and lexicons. Recently neural network-based language models have been explored, as they as a byproduct generate highly informative vector representations for words, known as word embeddings. In this paper we present two contributions: a new form of learning word embeddings that can leverage information from relevant lexicons to improve the representations, and the first system to use neural word embeddings to achieve state-of-the-art results on named-entity recognition in both CoNLL and Ontonotes NER. Our system achieves an F1 score of 90.90 on the test set for CoNLL 2003---significantly better than any previous system trained on public data, and matching a system employing massive private industrial query-log data.

Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$

Adam Roberts, Hyung Won Chung, Anselm Levskaya +40 more

cs.LG · 31 Mar 2022 · cited 227× · arXiv ↗ · more like this ·

Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can be complicated due to various factors including the need to distribute computation on supercomputer clusters (e.g., TPUs), prevent bottlenecks when infeeding data, and ensure reproducible results. In this work, we present two software libraries that ease these issues: $\texttt{t5x}$ simplifies the process of building and training large language models at scale while maintaining ease of use, and $\texttt{seqio}$ provides a task-based API for simple creation of fast and reproducible training data and evaluation pipelines. These open-source libraries have been used to train models with hundreds of billions of parameters on datasets with multiple terabytes of training data. Along with the libraries, we release…

TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learning

Akshay Agrawal, Akshay Naresh Modi, Alexandre Passos +8 more

cs.PL · 27 Feb 2019 · cited 95× · arXiv ↗ · more like this ·

TensorFlow Eager is a multi-stage, Python-embedded domain-specific language for hardware-accelerated machine learning, suitable for both interactive research and production. TensorFlow, which TensorFlow Eager extends, requires users to represent computations as dataflow graphs; this permits compiler optimizations and simplifies deployment but hinders rapid prototyping and run-time dynamism. TensorFlow Eager eliminates these usability costs without sacrificing the benefits furnished by graphs: It provides an imperative front-end to TensorFlow that executes operations immediately and a JIT tracer that translates Python functions composed of TensorFlow operations into executable dataflow graphs. TensorFlow Eager thus offers a multi-stage programming model that makes it easy to interpolate between imperative and staged execution in a single package.

Faster Neural Network Training with Data Echoing

Dami Choi, Alexandre Passos, Christopher J. Shallue +1 more

cs.LG · July 2019 · cited 55× · arXiv ↗ · more like this ·

In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training. However, earlier stages of the training pipeline, such as disk I/O and data preprocessing, do not run on accelerators. As accelerators continue to improve, these earlier stages will increasingly become the bottleneck. In this paper, we introduce "data echoing," which reduces the total computation used by earlier pipeline stages and speeds up training whenever computation upstream from accelerators dominates the training time. Data echoing reuses (or "echoes") intermediate outputs from earlier pipeline stages in order to reclaim idle capacity. We investigate the behavior of different data echoing algorithms on various workloads, for various amounts of echoing, and for various batch sizes. We find that in all settings, at least one data echoing algorithm can…

Flexible Modeling of Latent Task Structures in Multitask Learning

Alexandre Passos, Piyush Rai, Jacques Wainer +1 more

cs.LG · 27 Jun 2012 · cited 52× · arXiv ↗ · more like this ·

Multitask learning algorithms are typically designed assuming some fixed, a priori known latent structure shared by all the tasks. However, it is usually unclear what type of latent task structure is the most appropriate for a given multitask learning problem. Ideally, the "right" latent task structure should be learned in a data-driven manner. We present a flexible, nonparametric Bayesian model that posits a mixture of factor analyzers structure on the tasks. The nonparametric aspect makes the model expressive enough to subsume many existing models of latent task structures (e.g, mean-regularized tasks, clustered tasks, low-rank or linear/non-linear subspace assumption on tasks, etc.). Moreover, it can also learn more general task structures, addressing the shortcomings of such models. We present a variational inference algorithm for our model. Experimental results on synthetic and…

FRUIT: Faithfully Reflecting Updated Information in Text

Robert L. Logan IV, Alexandre Passos, Sameer Singh +1 more

cs.CL · December 2021 · cited 48× · arXiv ↗ · more like this ·

Textual knowledge bases such as Wikipedia require considerable effort to keep up to date and consistent. While automated writing assistants could potentially ease this burden, the problem of suggesting edits grounded in external knowledge has been under-explored. In this paper, we introduce the novel generation task of *faithfully reflecting updated information in text* (FRUIT) where the goal is to update an existing article given new evidence. We release the FRUIT-WIKI dataset, a collection of over 170K distantly supervised data produced from pairs of Wikipedia snapshots, along with our data generation pipeline and a gold evaluation set of 914 instances whose edits are guaranteed to be supported by the evidence. We provide benchmark results for popular generation systems as well as EDIT5 -- a T5-based approach tailored to editing we introduce that establishes the state of the art. Our…

Learning Soft Linear Constraints with Application to Citation Field Extraction

Sam Anzaroot, Alexandre Passos, David Belanger +1 more

cs.CL · March 2014 · cited 30× · arXiv ↗ · more like this ·

Accurately segmenting a citation string into fields for authors, titles, etc. is a challenging task because the output typically obeys various global constraints. Previous work has shown that modeling soft constraints, where the model is encouraged, but not require to obey the constraints, can substantially improve segmentation performance. On the other hand, for imposing hard constraints, dual decomposition is a popular technique for efficient prediction given existing algorithms for unconstrained inference. We extend the technique to perform prediction subject to soft constraints. Moreover, with a technique for performing inference given soft constraints, it is easy to automatically generate large families of constraints and learn their costs with a simple convex optimization problem during training. This allows us to obtain substantial gains in accuracy on a new, challenging citation…

RAMEAU: A SYSTEM FOR AUTOMATIC HARMONIC ANALYSIS

Pedro Kröger, Alexandre Passos, Marcos Sampaio

Signal Processing · The Journal of the Abraham Lincoln Association · October 2009 · cited 13× · OpenAlex ↗ · more like this ·

Automated harmonic analysis is an important and interesting music research topic.Although many researchers have studied solutions to this problem, there is no comprehensive and systematic comparison of the many techniques proposed.In this paper we present Rameau, a framework for automatic harmonic analysis we are developing.With Rameau we are able to reimplement and analyze previous techniques, and develop new ones as well.We present a performance evaluation of ten algorithms on a corpus of 140 Bach chorales.We also evaluate four of them using precision and recall and discuss possible improvements.We also present a numeric codification for tonal music with interesting properties, such as easy transposition, preservation of enharmonic information and easy conversion to standard pitch-class notation.

Field hospitals to face COVID-19: Requirements and lessons learned in the design and construction of the Lagoa Barra Hospital - Brazil

Alexandre Passos Ribeiro, Kemmylle Sanny de Matos Ferreira, Christine Kowal Chinelli +1 more

Information Systems · Strategic Design Research Journal · 23 Dec 2020 · cited 6× · DOI ↗ · more like this ·

The construction of field hospitals has been a strategy adopted worldwide to face the consequences of the pandemic impacts caused by COVID-19 on healthcare systems. Specific characteristics of this pandemic, such as different ways and speeds of the disease transmission, and the implications at the population health and the productive system, has made the project, management, and construction of field hospitals for the patients with COVID-19 present specific features. In this work, for each phase of the FHLB implantation, we present the main concepts, premises, restrictions, and challenges, focusing mainly on the needs programs of the project that guided the configuration of the environments, the definition of the circulation flows, the typology of the beds, the main management tools used during the project's planning and control process, and in the lessons learned.

MAP Inference in Chains using Column Generation

David Belanger, Alexandre Passos, Sebastian Riedel +1 more

Artificial Intelligence · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · 3 Dec 2012 · cited 5× · OpenAlex ↗ · more like this ·

Linear chains and trees are basic building blocks in many applications of graphical models, and they admit simple exact maximum a-posteriori (MAP) inference algorithms based on message passing. However, in many cases this computation is prohibitively expensive, due to quadratic dependence on variables' domain sizes. The standard algorithms are inefficient because they compute scores for hypotheses for which there is strong negative local evidence. For this reason there has been significant previous interest in beam search and its variants; however, these methods provide only approximate results. This paper presents new exact inference algorithms based on the combination of column generation and pre-computed bounds on terms of the model's scoring function. While we do not improve worst-case performance, our method substantially speeds real-world, typical-case inference in chains and…

Universal Schema for Slot Filling and Cold Start: UMass IESL at TACKBP 2013.

Sameer Singh, Limin Yao, David Belanger +8 more

Artificial Intelligence · Theory and applications of categories · 2013 · cited 4× · OpenAlex ↗ · more like this ·

We employ universal schema for the TAC KBP slot filling and cold start tasks. The technique enlarges the set of relations in an ontology, e.g., TACKBP slots, to con-tain all surface patterns between pairs of entities in a large corpus. By factoriz-ing the matrix of co-occurrences between entity pairs and universal schema rela-tions, we are able to predict new target slots. This differs fundamentally from tra-ditional relation extraction approaches be-cause an entire knowledge base is con-structed jointly over train and test data. To produce submissions for the slot filling and cold start tasks, we simply query this knowledge base. We describe universal schema, our data preprocessing pipeline, and additional techniques we employ for predicting entities ’ attributes. 1

Message passing for soft constraint dual decomposition

David Belanger, Alexandre Passos, Sebastian Riedel +1 more

Artificial Intelligence · Uncertainty in Artificial Intelligence · 23 Jul 2014 · cited 2× · OpenAlex ↗ · more like this ·

Dual decomposition provides the opportunity to build complex, yet tractable, structured predic-tion models using linear constraints to link to-gether submodels that have available MAP infer-ence routines. However, since some constraints might not hold on every single example, such models can often be improved by relaxing the requirement that these constraints always hold, and instead replacing them with soft constraints that merely impose a penalty if violated. A dual objective for the resulting MAP inference prob-lem differs from the hard constraint problem’s associated dual decomposition objective only in that the dual variables are subject to box con-straints. This paper introduces a novel primal-dual block coordinate descent algorithm for min-imizing this general family of box-constrained objectives. Through experiments on two nat-ural language corpus-wide inference tasks, we…

What do you know? A topic-model approach to authority identification

Alexandre Passos, Jacques Wainer, Aria Haghighi

Information Systems · 2009 · cited 2× · OpenAlex ↗ · more like this ·

A signiﬁcant problem in the analysis of social media – e.g., web pages, discussion forums, blogs,product reviews, etc– is how to identify authoritative documents in a given domain. Broadly, au-thoritative documents are those which exhibit novel and relevant information relative to a documentcollection and generally demonstrate domain knowledge. Identifying authoritative documents andﬁnding useful content will become a key natural language processing (NLP) and social computingproblem as the amount of social media grows. In this work, we present a preliminary study of basicapproaches to the problem, including a surprisingly strong and simple baseline. Our best perform-ing approach is a hierarchical topic model [Blei et al., 2004] similar to Haghighi and Vanderwende2009.Previous work have explored graph-theoretical approaches to authority in networks where for-mal links are present between…

Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models

Luke Vilnis, Yury Zemlyanskiy, Patrick Murray +2 more

cs.CL · October 2022 · cited 1× · arXiv ↗ · more like this ·

Decoding methods for large language models often trade-off between diversity of outputs and parallelism of computation. Methods such as beam search and Gumbel top-k sampling can guarantee a different output for each element of the beam, but are not easy to parallelize. Alternatively, methods such as temperature sampling and its modifications (top-k sampling, nucleus sampling, typical decoding, and others), are embarrassingly parallel, but have no guarantees about duplicate samples. We present a framework for sampling according to an arithmetic code book implicitly defined by a large language model, compatible with common sampling variations, with provable beam diversity under certain conditions, as well as being embarrassingly parallel and providing unbiased and consistent expectations from the original model. We demonstrate the effectiveness of our approach on WMT machine translation…

Speeding up MAP with Column Generation and Block Regularization

David Belanger, Alexandre Passos, Sebastian Riedel +1 more

Artificial Intelligence · June 2012 · cited 1× · OpenAlex ↗ · more like this ·

In this paper, we show how the connections between max-product message passing and linear programming relaxations for MAP al-low for a more efficient exact algorithm than standard dynamic programming. Our pro-posed algorithm uses column generation to pass messages only on a small subset of the possible assignments to each variable, while guaranteeing to find the exact solution. This algorithm is more than two times faster than Viterbi decoding for part-of-speech tagging on WSJ data and equivalently fast as beam search with a beam of size two, while being exact. The empirical performance of column gener-ation depends on how quickly we can rule out entire sets of assignments to the edges of the chain, which is done by bounding the contri-bution of the pairwise factors to the score of the solution. This provides an opportunity at the intersection of inference and learning: at training…

A penny for your thoughts? The value of information in recommendation systems

Alexandre Passos, Jurgen Van Gael, Ralf Herbrich +1 more

Information Systems · 4 Dec 2015 · cited 0× · OpenAlex ↗ · more like this ·

Most recommendation systems are trained to predict behavioral data and then used to generate more such data by recommending items and receiving feedback on the quality of these recommendations. This data in then fed back into the training process. This creates a feedback loop: as long as the low-cost way to interact with the service is through the recommender, the recommender will only ever

Mathieu Blondel

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort +9 more

Computer Science Applications · 6 May 2012 · cited 0× · OpenAlex ↗ · more like this ·

Learning to Speed Up MAP Decoding with Column Generation

David Belanger, Alexandre Passos, Sebastian Riedel +1 more

Artificial Intelligence · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · 2012 · cited 0× · OpenAlex ↗ · more like this ·

In this paper, we show how the connections between max-product message passing for max-product and linear programming relaxations allow for a more efficient exact algorithm for the MAP problem. Our proposed algorithm uses column generation to pass messages only on a small subset of the possible assignments to each variable, while guaranteeing to find the exact solution. This algorithm is three times faster than Viterbi decoding for part-of-speech tagging on WSJ data and equivalently fast as beam search with a beam of size two while being exact. The empirical performance of column generation depends on how quickly we can rule out entire sets of assignments to the edges of the chain, which is done by bounding the contribution of the pairwise factors to the score of the solution. This provides an opportunity at the intersection of inference and learning: at training time, we can regularize…

<title>Object-oriented framework for rapid game prototyping</title>

Alexandre Passos, Richard Simpson

Computer Networks and Communications · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 24 May 2002 · cited 0× · DOI ↗ · more like this ·

Small game development groups and companies are faced with two important challenges in today's economy: creating a good game prototype as showcase for game publishers and meeting the time to market deadlines. These two challenges are sometimes the factors that will separate a successful group from one that is not. The learning curve in design and implementation is a significant component of these two challenges. If the learning curve is too steep then deadlines may not be met and the overall quality of the software is lowered. This study presents a new game- programming library called PGL that addresses the timing and learning factors that exist in game development.

Developing 3-d animated applications prototypes in the classroom

Alexandre Passos, Richard Simpson

Computer Graphics and Computer-Aided Design · Journal of computing sciences in colleges · April 2002 · cited 0× · OpenAlex ↗ · more like this ·

Simulated environments, such as those found in training centers, computer games, movies, advertisement and defense applications, require software companies to face two important challenges in today's economy: creating a good prototype as showcase for the end-user and meeting time to market deadlines. Graphics libraries and development environments provide to the application programmer the necessary interfaces used in the construction of graphics, animations, simulations and games. The true value of a graphics library can be measured from two issues: ease of use and extensibility. However, the special needs in an educational environment change the shape and form of graphics frameworks such that each library must be designed to accommodate ease of use and extensibility. Such frameworks provide the user-friendly environment required by, for example, novice game programmers. This study…