Author

Mario Marchand

65 papers · 1989–2026

1989peak 8 in 19932026

Frequent co-authors

Papers

showing 1–25 of 65 RSS

Domain-Adversarial Training of Neural Networks

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan +5 more

stat.ML · May 2015 · cited 12,302× · arXiv ↗ · more like this ·

We introduce a new representation learning approach for domain adaptation, in which data at training and test time come from similar but different distributions. Our approach is directly inspired by the theory on domain adaptation suggesting that, for effective domain transfer to be achieved, predictions must be made based on features that cannot discriminate between the training (source) and test (target) domains. The approach implements this idea in the context of neural network architectures that are trained on labeled data from the source domain and unlabeled data from the target domain (no labeled target-domain data is necessary). As the training progresses, the approach promotes the emergence of features that are (i) discriminative for the main learning task on the source domain and (ii) indiscriminate with respect to the shift between the domains. We show that this adaptation…

Domain-Adversarial Neural Networks

Hana Ajakan, Pascal Germain, Hugo Larochelle +2 more

stat.ML · December 2014 · cited 359× · arXiv ↗ · more like this ·

We introduce a new representation learning algorithm suited to the context of domain adaptation, in which data at training and test time come from similar but different distributions. Our algorithm is directly inspired by theory on domain adaptation suggesting that, for effective domain transfer to be achieved, predictions must be made based on a data representation that cannot discriminate between the training (source) and test (target) domains. We propose a training objective that implements this idea in the context of a neural network, whose hidden layer is trained to be predictive of the classification task, but uninformative as to the domain of the input. Our experiments on a sentiment analysis classification benchmark, where the target domain data available at training time is unlabeled, show that our neural network for domain adaption algorithm has better performance than either…

PAC-Bayesian learning of linear classifiers

Pascal Germain, Alexandre Lacasse, François Laviolette +1 more

Artificial Intelligence · 14 Jun 2009 · cited 177× · DOI ↗ · more like this ·

We present a general PAC-Bayes theorem from which all known PAC-Bayes risk bounds are obtained as particular cases. We also propose different learning algorithms for finding linear classifiers that minimize these bounds. These learning algorithms are generally competitive with both AdaBoost and the SVM.

Risk Bounds for the Majority Vote: From a PAC-Bayesian Analysis to a Learning Algorithm

Pascal Germain, Alexandre Lacasse, François Laviolette +2 more

stat.ML · March 2015 · cited 158× · arXiv ↗ · more like this ·

We propose an extensive analysis of the behavior of majority votes in binary classification. In particular, we introduce a risk bound for majority votes, called the C-bound, that takes into account the average quality of the voters and their average disagreement. We also propose an extensive PAC-Bayesian analysis that shows how the C-bound can be estimated from various observations contained in the training data. The analysis intends to be self-contained and can be used as introductory material to PAC-Bayesian statistical learning theory. It starts from a general PAC-Bayesian perspective and ends with uncommon PAC-Bayesian bounds. Some of these bounds contain no Kullback-Leibler divergence and others allow kernel functions to be used as voters (via the sample compression setting). Finally, out of the analysis, we propose the MinCq learning algorithm that basically minimizes the C-bound.…

A Convergence Theorem for Sequential Learning in Two-Layer Perceptrons

Mario Marchand, Mostefa Golea, P. Ruján

Artificial Intelligence · Europhysics Letters (EPL) · 15 Mar 1990 · cited 136× · DOI ↗ · more like this ·

We consider a Perceptron with Ni input units, one output and a yet unspecified number of hidden units. This Perceptron must be able to learn a given but arbitrary set of input-output examples. By sequential learning we mean that groups of patterns, pertaining to the same class, are sequentially separated from the rest by successively adding hidden units until the remaining patterns are all in the same class. We prove that the internal representations obtained by such procedures are linearly separable. Preliminary numerical tests of an algorithm implementing these ideas are presented and compare favourably with results of other growth algorithms.

The set covering machine

Mario Marchand, John Shawe‐Taylor

Artificial Intelligence · ePrints Soton (University of Southampton) · March 2003 · cited 112× · DOI ↗ · more like this ·

We extend the classical algorithms of Valiant and Haussler for learning compact conjunctions and disjunctions of Boolean attributes to allow features that are constructed from the data and to allow a trade-off between accuracy and complexity. The result is a general-purpose learning machine, suitable for practical learning tasks, that we call the set covering machine. We present a version of the set covering machine that uses data-dependent balls for its set of features and compare its performance with the support vector machine. By extending a technique pioneered by Littlestone and Warmuth, we bound its generalization error as a function of the amount of data compression it achieves during training. In experiments with real-world learning tasks, the bound is shown to be extremely tight and to provide an effective guide for model selection.

PAC-Bayes Bounds for the Risk of the Majority Vote and the Variance of the Gibbs Classifier

Alexandre Lacasse, François Laviolette, Mario Marchand +2 more

Statistics and Probability · The MIT Press eBooks · 7 Sep 2007 · cited 77× · DOI ↗ · more like this ·

We propose new PAC-Bayes bounds for the risk of the weighted majority vote that depend on the mean and variance of the error of its associated Gibbs classifier.We show that these bounds can be smaller than the risk of the Gibbs classifier and can be arbitrarily close to zero even if the risk of the Gibbs classifier is close to 1/2.Moreover, we show that these bounds can be uniformly estimated on the training data for all possible posteriors Q.Moreover, they can be improved by using a large sample of unlabelled data.

Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis

Qi Chen, Changjian Shui, Mario Marchand

cs.LG · September 2021 · cited 66× · arXiv ↗ · more like this ·

We derive a novel information-theoretic analysis of the generalization property of meta-learning algorithms. Concretely, our analysis proposes a generic understanding of both the conventional learning-to-learn framework and the modern model-agnostic meta-learning (MAML) algorithms. Moreover, we provide a data-dependent generalization bound for a stochastic variant of MAML, which is non-vacuous for deep few-shot learning. As compared to previous bounds that depend on the square norm of gradients, empirical validations on both simulated data and a well-known few-shot benchmark show that our bound is orders of magnitude tighter in most situations.

A Growth Algorithm for Neural Network Decision Trees

Mostefa Golea, Mario Marchand

Artificial Intelligence · Europhysics Letters (EPL) · June 1990 · cited 58× · DOI ↗ · more like this ·

This paper explores the application of neural network principles to the construction of decision trees from examples. We consider the problem of constructing a tree of perceptrons able to execute a given but arbitrary Boolean function defined on N i input bits. We apply a sequential (from one tree level to the next) and parallel (for neurons in the same level) learning procedure to add hidden units until the task in hand is performed. At each step, we use a perceptron-type algorithm over a suitable defined input space to minimise a classification error. The internal representations obtained in this way are linearly separable. Preliminary results of this algorithm are presented.

From PAC-Bayes Bounds to Quadratic Programs for Majority Votes

Jean-Francis Roy, Mario Marchand, Fran ois Laviolette

Artificial Intelligence · International Conference on Machine Learning · 28 Jun 2011 · cited 49× · OpenAlex ↗ · more like this ·

We propose to construct a weighted majority vote on a set of basis functions by minimizing a risk bound (called the C-bound) that depends on the first two moments of the margin of the Q-convex combination realized on the data. This bound minimization algorithm turns out to be a quadratic program that can be efficiently solved. A first version of the algorithm is designed for the supervised inductive setting and turns out to be very competitive with AdaBoost, MDBoost and the SVM. The second version is designed for the transductive setting. It competes well against TSVM. We also propose a new PAC-Bayes theorem that bounds the difference between the true value of the C-bound and its empirical estimate and that, unexpectedly, contains no KL-divergence.

Learning with Decision Lists of Data-Dependent Features

Mario Marchand, Marina Sokolova

Computer Vision and Pattern Recognition · Journal of Machine Learning Research · December 2005 · cited 47× · OpenAlex ↗ · more like this ·

We present a learning algorithm for decision lists which allows features that are constructed from the data and allows a trade-off between accuracy and complexity. We provide bounds on the generalization error of this learning algorithm in terms of the number of errors and the size of the classifier it finds on the training data. We also compare its performance on some natural data sets with the set covering machine and the support vector machine. Furthermore, we show that the proposed bounds on the generalization error provide effective guides for model selection.

On learning simple neural concepts: from halfspace intersections to neural decision lists

Mario Marchand, Mostefa Golea

Artificial Intelligence · Network Computation in Neural Systems · 1993 · cited 45× · DOI ↗ · more like this ·

In this paper, we take a close look at the problem of learning simple neural concepts under the uniform distribution of examples. By simple neural concepts the authors mean concepts that can be represented as simple combinations of perceptrons (halfspaces). One such class of concepts is the class of halfspace intersections. By formalizing the problem of learning halfspace intersections as a set-covering problem, we are led to consider the following sub-problem: given a set of nonlinearly separable examples, find the largest linearly separable subset of it. The authors give an approximation algorithm for this NP-hard sub-problem. Simulations, on both linearly and nonlinearly separable functions, show that this approximation algorithm works well under the uniform distribution, outperforming the pocket algorithm used by many constructive neural algorithms. Based on this approximation, we…

Sequential Model-Based Ensemble Optimization

Alexandre Lacoste, Hugo Larochelle, François Laviolette +1 more

cs.LG · 4 Feb 2014 · cited 38× · arXiv ↗ · more like this ·

One of the most tedious tasks in the application of machine learning is model selection, i.e. hyperparameter selection. Fortunately, recent progress has been made in the automation of this process, through the use of sequential model-based optimization (SMBO) methods. This can be used to optimize a cross-validation performance of a learning algorithm over the value of its hyperparameters. However, it is well known that ensembles of learned models almost consistently outperform a single model, even if properly selected. In this paper, we thus propose an extension of SMBO methods that automatically constructs such ensembles. This method builds on a recently proposed ensemble construction paradigm known as agnostic Bayesian learning. In experiments on 22 regression and 39 classification data sets, we confirm the success of this proposed approach, which is able to outperform model selection…

Adaptive Deep Kernel Learning

Prudencio Tossou, Basile Dura, Francois Laviolette +2 more

cs.LG · May 2019 · cited 31× · arXiv ↗ · more like this ·

Deep kernel learning provides an elegant and principled framework for combining the structural properties of deep learning algorithms with the flexibility of kernel methods. By means of a deep neural network, we learn a parametrized kernel operator that can be combined with a differentiable kernel algorithm during inference. While previous work within this framework has focused on learning a single kernel for large datasets, we learn a kernel family for a variety of few-shot regression tasks. Compared to single deep kernel learning, our algorithm enables the identification of the appropriate kernel for each task during inference. As such, it is well adapted for complex task distributions in a few-shot learning setting, which we demonstrate by comparing against existing state-of-the-art algorithms using real-world, few-shot regression tasks related to the field of drug discovery.

Bayesian Comparison of Machine Learning Algorithms on Single and Multiple Datasets

Alexandre Lacoste, Françcois Laviolette, Mario Marchand

Artificial Intelligence · International Conference on Artificial Intelligence and Statistics · 21 Mar 2012 · cited 29× · OpenAlex ↗ · more like this ·

We propose a new method for comparing learning algorithms on multiple tasks which is based on a novel non-parametric test that we call the Poisson binomial test. The key aspect of this work is that we provide a formal definition for what is meant to have an algorithm that is better than another. Also, we are able to take into account the dependencies induced when evaluating classifiers on the same test set. Finally we make optimal use (in the Bayesian sense) of all the testing data we have. We demonstrate empirically that our approach is more reliable than the sign test and the Wilcoxon signed rank test, the current state of the art for algorithm comparisons. 1

Agnostic Bayesian Learning of Ensembles

Alexandre Lacoste, Mario Marchand, Fran ois Laviolette +1 more

Artificial Intelligence · International Conference on Machine Learning · 21 Jun 2014 · cited 28× · OpenAlex ↗ · more like this ·

We propose a method for producing ensembles of predictors based on holdout estimations of their generalization performances. This approach uses a prior directly on the performance of predictors taken from a finite set of candidates and attempts to infer which one is best. Using Bayesian infer-ence, we can thus obtain a posterior that repre-sents our uncertainty about that choice and con-struct a weighted ensemble of predictors accord-ingly. This approach has the advantage of not re-quiring that the predictors be probabilistic them-selves, can deal with arbitrary measures of per-formance and does not assume that the data was actually generated from any of the predictors in the ensemble. Since the problem of finding the best (as opposed to the true) predictor among a class is known as agnostic PAC-learning, we re-fer to our method as agnostic Bayesian learning. We also propose a method to…

PAC-Bayes Risk Bounds for Stochastic Averages and Majority Votes of Sample-Compressed Classifiers

François Laviolette, Mario Marchand

Artificial Intelligence · Journal of Machine Learning Research · December 2007 · cited 28× · OpenAlex ↗ · more like this ·

We propose a PAC-Bayes theorem for the sample-compression setting where each classifier is described by a compression subset of the training data and a message string of additional information. This setting, which is the appropriate one to describe many learning algorithms, strictly generalizes the usual data-independent setting where classifiers are represented only by data-independent message strings (or parameters taken from a continuous set). The proposed PAC-Bayes theorem for the sample-compression setting reduces to the PAC-Bayes theorem of Seeger (2002) and Langford (2005) when the compression subset of each classifier vanishes. For posteriors having all their weights on a single sample-compressed classifier, the general risk bound reduces to a bound similar to the tight sample-compression bound proposed in Laviolette et al. (2005). Finally, we extend our results to the case…

Learning by Minimizing Resources in Neural Networks

P. Ruján, Mario Marchand

Artificial Intelligence · Complex Systems · 1989 · cited 26× · OpenAlex ↗ · more like this ·

We reformulate the problem of supervised learning in neu ral nets to include the search for a network with minimal resources . The information processing in feedforward networks is described in geometrical terms as the partitioning of the space of possible input configurations by hyperplanes corresponding to hidden units. Regu lar partitionings introduced here are a special class of partitionings. Corresponding architectures can represent any Boolean function using a single layer of hidden units whose number depends on the specific symmetries of the function. Accordin gly, a new class of pla ne-cutting algorithms is proposed that const ruct in polynomial time a custom made architecture implementing the desired set of inputj'ouput ex amples . We report the results of our experiments on the storage and rule-extraction abilities of three-layer perceptrons synthetized by a simple greedy…

Learning with the Set Covering Machine

Mario Marchand, John Shawe‐Taylor

Artificial Intelligence · ePrints Soton (University of Southampton) · 28 Jun 2001 · cited 23× · OpenAlex ↗ · more like this ·

We generalize the classical algorithms of Valiant and Haussler for learning conjunctions and disjunctions of Boolean attributes to the problem of learning these functions over arbitrary sets of features. The result is a general-purposed learning machine, suitable for practical learning tasks, that we call the Set Covering Machine. We present a version of the Set Covering Machine that uses generalized balls for its set of features and compare its performance to the famous Support Vector Machine. 1 Motivation We may attribute the eectiveness of Support Vector Machines [6] to the fact that they combine two very good ideas. First, they map the space of input vectors onto a very high-dimensional feature space in such a way that nonlinear decision functions on the input space can be constructed by using only hyperplanes on the feature space. Second, they construct the separating hyperplane on…

The Decision List Machine

Marina Sokolova, Mario Marchand, Nathalie Japkowicz +1 more

Artificial Intelligence · ePrints Soton (University of Southampton) · 2002 · cited 22× · OpenAlex ↗ · more like this ·

We introduce a new learning algorithm for decision lists to allow features that are constructed from the data and to allow a tradeoff between accuracy and complexity. We bound its generalization error in terms of the number of errors and the size of the classifier it finds on the training data. We also compare its performance on some natural data sets with the set covering machine and the support vector machine. 1

PAC-Bayes Learning of Conjunctions and Classification of Gene-Expression Data

Mario Marchand, Mohak Shah

Artificial Intelligence · Neural Information Processing Systems · December 2004 · cited 17× · OpenAlex ↗ · more like this ·

We propose a “soft greedy ” learning algorithm for building small conjunctions of simple threshold functions, called rays, defined on single real-valued attributes. We also propose a PAC-Bayes risk bound which is minimized for classifiers achieving a non-trivial tradeoff between sparsity (the number of rays used) and the mag-nitude of the separating margin of each ray. Finally, we test the soft greedy algorithm on four DNA micro-array data sets. 1

From PAC-Bayes Bounds to KL Regularization

Pascal Germain, Alexandre Lacasse, Mario Marchand +2 more

Artificial Intelligence · Neural Information Processing Systems · 7 Dec 2009 · cited 16× · OpenAlex ↗ · more like this ·

We show that convex KL-regularized objective functions are obtained from a PAC-Bayes risk bound when using convex loss functions for the stochastic Gibbs classifier that upper-bound the standard zero-one loss used for the weighted ma-jority vote. By restricting ourselves to a class of posteriors, that we call quasi uniform, we propose a simple coordinate descent learning algorithm to minimize the proposed KL-regularized cost function. We show that standard `p-regularized objective functions currently used, such as ridge regression and `p-regularized boosting, are obtained from a relaxation of the KL divergence between the quasi uniform posterior and the uniform prior. We present numerical experiments where the proposed learning algorithm generally outperforms ridge regression and Ada-Boost. 1

Comparing GPLVM approaches for dimensionality reduction in character animation

Sébastien Quirion, Chantale Duchesne, Denis Laurendeau +1 more

Artificial Intelligence · Digital Library (University of West Bohemia) · 2008 · cited 16× · OpenAlex ↗ · more like this ·

Gaussian Process Latent Variable Models (GPLVMs) have been found to allow dramatic dimensionality\nreduction in character animations, often yielding two-dimensional or three-dimensional spaces from which the\nanimation can be retrieved without perceptible alterations. Recently, many researchers have used this approach\nand improved on it for their purposes, thus creating a number of GPLVM-based approaches. The current paper\nintroduces the main concepts behind GPLVMs and introduces its most widely known variants. Each approach is\nthen compared based on various criteria pertaining to the task of dimensionality reduction in character\nanimation. In the light of our experiments, no single approach is preferred over all others in all respects.\nDepending whether dimensionality reduction is used for compression purposes, to interpolate new natural\nlooking poses or to synthesize entirely…

Computing the Bayes Kernel Classifier

P. Ruján, Mario Marchand

Artificial Intelligence · The MIT Press eBooks · 29 Sep 2000 · cited 16× · DOI ↗ · more like this ·

Introduction Support Vector Machines try to achieve good generalization by computing the maximum margin separating hyperplane in a high-dimensional feature space. This approach eectively combines two very good ideas. The rst idea is to map the space of input vectors into a very high-dimensional feature space in such a way that nonlinear decisions functions on the input space can be constructed by using only separating hyperplanes on the feature space. By making use of kernels, we can implicitly perform such mappings without explicitly using high-dimensional separating vectors(Boser et al., 1992). Since it is very likely that the training examples will be linearly separable in the high-dimensional feature space, this method oers an elegant alternative to network growth algorithms as in(Rujan and Marchand, 1989; Marchand et al., 1990) which try to construct nonlinear decision surfaces by…

A PAC-Bayes Sample-compression Approach to Kernel Methods

Pascal Germain, Alexandre Lacoste, Mario Marchand +2 more

Computer Vision and Pattern Recognition · International Conference on Machine Learning · 28 Jun 2011 · cited 15× · OpenAlex ↗ · more like this ·

We propose a PAC-Bayes sample compression approach to kernel methods that can accommodate any bounded similarity function and show that the support vector machine (SVM) classifier is a particular case of a more general class of data-dependent classifiers known as majority votes of samplecompressed classifiers. We provide novel risk bounds for these majority votes and learning algorithms that minimize these bounds. 1.