Author

Stuart Geman

34 papers · 1979–2018

1979peak 4 in 19822018

Frequent co-authors

Papers

showing 1–25 of 34 RSS

Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images*

Computer Vision and Pattern Recognition · Journal of Applied Statistics · 1993 · cited 17,980× · DOI ↗ · more like this ·

We make an analogy between images and statistical mechanics systems. Pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice-like physical system. The assignment of an energy function in the physical system determines its Gibbs distribution. Because of the Gibbs distribution, Markov random field (MRF) equivalence, this assignment also determines an MRF image model. The energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the MRF. For a range of degradation mechanisms, including blurring, non-linear deformations, and multiplicative or additive noise, the posterior distribution is an MRF with a structure akin to the image model. By the analogy, the posterior distribution defines another (imaginary) physical system. Gradual temperature reduction in the…

Neural Networks and the Bias/Variance Dilemma

Stuart Geman, Elie Bienenstock, René Doursat

Artificial Intelligence · Neural Computation · 1992 · cited 3,563× · DOI ↗ · more like this ·

Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.

A Limit Theorem for the Norm of Random Matrices

Stuart Geman

Statistics and Probability · The Annals of Probability · April 1980 · cited 451× · DOI ↗ · more like this ·

This paper establishes an almost sure limit for the operator norm of rectangular random matrices: Suppose $\{v_{ij}\}i = 1,2, \cdots, j = 1,2, \cdots$ are zero mean i.i.d. random variables satisfying the moment condition $E|\nu_{11}|^n \leqslant n^{\alpha n}$ for all $n \geqslant 2$, and some $\alpha$. Let $\sigma^2 = Ev^2_{11}$ and let $V_{pn}$ be the $p \times n$ matrix $\{v_{ij}\}_{1\leqslant i\leqslant p; 1\leqslant j\leqslant n}$. If $p_n$ is a sequence of integers such that $p_n/n \rightarrow y$ as $n \rightarrow \infty$, for some $0 < y < \infty$, then $1/n|V_{p_nn}V^T_{p_nn}| \rightarrow (1 + y^{\frac{1}{2}})^2\sigma^2$ almost surely, where $|A|$ denotes the operator ("induced") norm of $A$. Since $1/n|V_{p_nn}V^T_{p_nn}|$ is the maximum eigenvalue of $1/nV_{p_nn}V^T_{p_nn}$, the result relates to studies on the spectrum of symmetric random matrices.

Nonparametric Maximum Likelihood Estimation by the Method of Sieves

Stuart Geman, Chii-Ruey Hwang

Statistics and Probability · The Annals of Statistics · June 1982 · cited 368× · DOI ↗ · more like this ·

Maximum likelihood estimation often fails when the parameter takes values in an infinite dimensional space. For example, the maximum likelihood method cannot be applied to the completely nonparametric estimation of a density function from an $\operatorname{iid}$ sample; the maximum of the likelihood is not attained by any density. In this example, as in many other examples, the parameter space (positive functions with area one) is too big. But the likelihood method can often be salvaged if we first maximize over a constrained subspace of the parameter space and then relax the constraint as the sample size grows. This is Grenander's "method of sieves." Application of the method sometimes leads to new estimators for familiar problems, or to a new motivation for an already well-studied technique. We will establish some general consistency results for the method, and then we will focus on…

Visual Turing test for computer vision systems

Donald Geman, Stuart Geman, Neil Hallonquist +1 more

Computer Vision and Pattern Recognition · Proceedings of the National Academy of Sciences · 9 Mar 2015 · cited 339× · DOI ↗ · more like this ·

Today, computer vision systems are tested by their accuracy in detecting and localizing instances of objects. As an alternative, and motivated by the ability of humans to provide far richer descriptions and even tell a story about an image, we construct a "visual Turing test": an operator-assisted device that produces a stochastic sequence of binary questions from a given test image. The query engine proposes a question; the operator either provides the correct answer or rejects the question as ambiguous; the engine proposes the next question ("just-in-time truthing"). The test is then administered to the computer-vision system, one question at a time. After the system's answer is recorded, the system is provided the correct answer and the next question. Parsing is trivial and deterministic; the system being tested requires no natural language processing. The query engine employs…

Estimators for Stochastic ``Unification-Based'' Grammars

Mark Johnson, Stuart Geman, Stephen Canon +2 more

cs.CL · 25 Aug 2000 · cited 224× · arXiv ↗ · more like this ·

Log-linear models provide a statistically sound framework for Stochastic ``Unification-Based'' Grammars (SUBGs) and stochastic versions of other kinds of grammars. We describe two computationally-tractable ways of estimating the parameters of such grammars from a training corpus of syntactic analyses, and apply these to estimate a stochastic version of Lexical-Functional Grammar.

Context and Hierarchy in a Probabilistic Image Model

Ya Jin, Stuart Geman

Computer Vision and Pattern Recognition · 10 Jul 2006 · cited 171× · DOI ↗ · more like this ·

It is widely conjectured that the excellent ROC performance of biological vision systems is due in large part to the exploitation of context at each of many levels in a part/whole hierarchy. We propose a mathematical framework (a "composition machine") for constructing probabilistic hierarchical image models, designed to accommodate arbitrary contextual relationships, and we build a demonstration system for reading Massachusetts license plates in an image set collected at Logan Airport. The demonstration system detects and correctly reads more than 98% of the plates, with a negligible rate of false detection. Unlike a formal grammar, the architecture of a composition machine does not exclude the sharing of sub-parts among multiple entities, and does not limit interpretations to single trees (e.g. a scene can have multiple license plates, or no plates at all). In this sense, the…

Estimation of probabilistic context-free grammars

Zhiyi Chi, Stuart Geman

Artificial Intelligence · June 1998 · cited 144× · OpenAlex ↗ · more like this ·

The assignment of probabilities to the productions of a context-free grammar may generate an improper distribution: the probability of all finite parse trees is less than one. The condition for proper assignment is rather subtle. Production probabilities can be estimated from parsed or unparsed sentences, and the question arises as to whether or not an estimated system is automatically proper. We show here that estimated production probabilities always yield proper distributions.

Compositionality, MDL Priors, and Object Recognition

Elie Bienenstock, Stuart Geman, Daniel Potter

Computer Vision and Pattern Recognition · Neural Information Processing Systems · 3 Dec 1996 · cited 115× · OpenAlex ↗ · more like this ·

Images are ambiguous at each of many levels of a contextual hierarchy. Nevertheless, the high-level interpretation of most scenes is unambiguous, as evidenced by the superior performance of humans. This observation argues for global vision models, such as deformable templates. Unfortunately, such models are computationally intractable for unconstrained problems. We propose a compositional model in which primitives are recursively composed, subject to syntactic restrictions, to form tree-structured objects and object groupings. Ambiguity is propagated up the hierarchy in the form of multiple interpretations, which are later resolved by a Bayesian, equivalently minimum-description-length, cost functional. 1 Bayesian decision theory and compositionality In his Essay on Probability, Laplace (1812) devotes a short chapter---his &quot;Sixth Principle&quot;---to what we call today the…

Consistent Cross-Validated Density Estimation

Y. S. Chow, Stuart Geman, Lili Wu

Statistics and Probability · The Annals of Statistics · March 1983 · cited 109× · DOI ↗ · more like this ·

Application of nonparametric density estimators generally requires the specification of a "smoothing parameter." The kernel estimator, for example, is not fully defined until a window width, or scaling, for the kernels has been chosen. Many "data-driven" techniques have been suggested for the practical choice of smoothing parameter. Of these, the most widely studied is the method of cross-validation. Our own simulations, as well as those of many other investigators, indicate that cross-validated smoothing can be an extremely effective practical solution. However, many of the most basic properties of cross-validated estimators are unknown. Indeed, recent results show that cross-validated estimators can fail even to be consistent for seemingly well-behaved problems. In this paper we will review the application of cross-validation to the smoothing problem, and establish $L_1$ consistency…

Some Averaging and Stability Results for Random Differential Equations

Stuart Geman

Statistics and Probability · SIAM Journal on Applied Mathematics · February 1979 · cited 71× · DOI ↗ · more like this ·

This paper concerns differential equations which contain strongly mixing random processes (processes for which the “past” and the “future” are asymptotically independent). When the “rate” of mixing is rapid relative to the rate of change of the solution process, information about the behavior of the solution is obtained. Roughly, the results fall into three categories: 1. Quite generally, the solution process is well approximated by a deterministic trajectory, over a finite time interval. 2. For more restricted systems, this approximation extends to the infinite interval $[ {0,\infty } ),3$. Conditions for the asymptotic stability of $\dot x = AX$, where A is an $n \times n$ matrix-valued random process, are obtained.

Hidden Markov Random Fields

Hans R. Künsch, Stuart Geman, Athanasios Kehagias

Artificial Intelligence · The Annals of Applied Probability · August 1995 · cited 69× · DOI ↗ · more like this ·

A noninvertible function of a first-order Markov process or of a nearest-neighbor Markov random field is called a hidden Markov model. Hidden Markov models are generally not Markovian. In fact, they may have complex and long range interactions, which is largely the reason for their utility. Applications include signal and image processing, speech recognition and biological modeling. We show that hidden Markov models are dense among essentially all finite-state discrete-time stationary processes and finite-state lattice-based stationary random fields. This leads to a nearly universal parameterization of stationary processes and stationary random fields, and to a consistent nonparametric estimator. We show the results of attempts to fit simple speech and texture patterns.

The Spectral Radius of Large Random Matrices

Stuart Geman

Statistics and Probability · The Annals of Probability · October 1986 · cited 69× · DOI ↗ · more like this ·

Let $\{m_{ij}\}, i = 1,2,\ldots, j = 1,2,\ldots,$ be iid random variables with $Em_{11} = 0$ and $Em^2_{11} = \sigma^2$. For each $n$ define $M_n = \{m_{ij}\}_{1 \leq i, j \leq n}$, the $n \times n$ matrix whose $(i, j)$ component is $m_{ij}$. We show that $\lim \sup_{n \rightarrow \infty}\rho_n \leq \sigma$ a.s., where $\rho_n$ is the spectral radius of $M_n/\sqrt n$. Evidence from computer experiments indicates that in fact $\rho_n \rightarrow \sigma$ a.s.

Dynamic programming for parsing and estimation of stochastic unification-based grammars

Stuart Geman, Mark Johnson

Artificial Intelligence · 2001 · cited 54× · DOI ↗ · more like this ·

Stochastic unification-based grammars (SUBGs) define exponential distributions over the parses generated by a unification-based grammar (UBG). Existing algorithms for parsing and estimation require the enumeration of all of the parses of a string in order to determine the most likely one, or in order to calculate the statistics needed to estimate a grammar from a training corpus. This paper describes a graph-based dynamic programming algorithm for calculating these statistics from the packed UBG parse representations of Maxwell and Kaplan (1995) which does not require enumerating all parses. Like many graphical algorithms, the dynamic programming algorithm's complexity is worst-case exponential, but is often polynomial. The key observation is that by using Maxwell and Kaplan packed representations, the required statistics can be rewritten as either the max or the sum of a product of…

Dynamic programming and the graphical representation of error-correcting codes

Stuart Geman, Kevin Geoffrey Kochanek

Computer Networks and Communications · IEEE Transactions on Information Theory · 2001 · cited 38× · DOI ↗ · more like this ·

Graphical representations of codes facilitate the design of computationally efficient decoding algorithms. This is an example of a general connection between dependency graphs, as arise in the representations of Markov random fields, and the dynamic programming principle. We concentrate on two computational tasks: finding the maximum-likelihood codeword and finding its posterior probability, given a signal received through a noisy channel. These two computations lend themselves to a particularly elegant version of dynamic programming, whereby the decoding complexity is particularly transparent. We explore some codes and some graphical representations designed specifically to facilitate computation. We further explore a coarse-to-fine version of dynamic programming that can produce an exact maximum-likelihood decoding many orders of magnitude faster than ordinary dynamic programming.

Almost Sure Stable Oscillations in a Large System of Randomly Coupled Equations

Stuart Geman

Computational Theory and Mathematics · SIAM Journal on Applied Mathematics · August 1982 · cited 23× · DOI ↗ · more like this ·

This paper is about limiting (large system) behavior of a set of differential equations with random coefficients. Under certain conditions the behavior of the entire system is well described by a small number of “prototype” equations, and these can be derived, heuristically, by applying a law of large numbers to the original system. An application of this theorem is the specification of a small number of parameters which guarantee that sufficiently large versions of the systems studied will oscillate with a predicted period and wave form.

Neural Networks and the BiadVariance Dilemma

Stuart Geman, Elie Bienenstock

Artificial Intelligence · 1992 · cited 14× · OpenAlex ↗ · more like this ·

Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallelversus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.

Image warping using radial basis functions

Ting‐Li Chen, Stuart Geman

Computer Vision and Pattern Recognition · Journal of Applied Statistics · 18 Sep 2013 · cited 12× · DOI ↗ · more like this ·

Image warping is the process of deforming an image through a transformation of its domain, which is typically a subset of R2. Given the destination of a collection of points, the problem becomes one of finding a suitable smooth interpolation for the destinations of the remaining points of the domain. A common solution is to use the thin plate spline (TPS). We find that the TPS often introduces unintended distortions of image structures. In this paper, we will analyze interpolation by TPS, experiment with other radial basis functions, and suggest two alternative functions that provide better results.

Dynamic programming, tree-width and computation on graphical models

Stuart Geman, Brian Lucena

Artificial Intelligence · 2002 · cited 9× · OpenAlex ↗ · more like this ·

Computing on graphical models is a field of diverse interest today, due to the general applicability of these models. This thesis begins by giving background on the generalized Dynamic Programming (DP) method of performing inference computations (Chapter 1). We then (in Chapter 2) demonstrate explicity equivalences between different methods of computation and the importance of a parameter called tree-width. We go on to prove a novel method of bounding the tree-width of a graph by using maximum cardinality search. This gives a lower bound on the computational complexity of a graph with respect to standard methods. This bound can be quite weak or quite good. We provide experimental results demonstrating both cases. Chapter 3 is concerned with Coarse-to-Fine Dynamic Programming (CFDP), a method which can be faster than the standard methods, but requires special conditions. We prove…

On the Minimum Entropy of a Mixture of Unimodal and Symmetric Distributions

Ting‐Li Chen, Stuart Geman

Computer Vision and Pattern Recognition · IEEE Transactions on Information Theory · 16 Jun 2008 · cited 7× · DOI ↗ · more like this ·

Progressive encoding of a signal generally involves an estimation step, designed to reduce the entropy of the residual of an observation over the entropy of the observation itself. Oftentimes the conditional distributions of an observation, given already-encoded observations, are well fit within a class of symmetric and unimodal distributions (e.g., the two-sided geometric distributions in images of natural scenes, or symmetric Paretian distributions in models of financial data). It is common practice to choose an estimator that centers, or aligns, the modes of the conditional distributions, since it is common sense that this will minimize the entropy, and hence the coding cost of the residuals. But with the exception of a special case, there has been no rigorous proof. Here we prove that the entropy of an arbitrary mixture of symmetric and unimodal distributions is minimized by…

Maximum likelihood features for generative image models

Lo‐Bin Chang, Eran Borenstein, Wei Zhang +1 more

Computer Vision and Pattern Recognition · The Annals of Applied Statistics · September 2017 · cited 5× · DOI ↗ · more like this ·

Most approaches to computer vision can be thought of as lying somewhere on a continuum between generative and discriminative. Although each approach has had its successes, recent advances have favored discriminative methods, most notably the convolutional neural network. Still, there is some doubt about whether this approach will scale to a human-level performance given the numbers of samples that are needed to train state-of-the-art systems. Here, we focus on the generative or Bayesian approach, which is more model based and, in theory, more efficient. Challenges include latent-variable modeling, computationally efficient inference, and data modeling. We restrict ourselves to the problem of data modeling, which is possibly the most daunting, and specifically to the generative modeling of image patches. We formulate a new approach, which can be broadly characterized as an application of…

<title>Grammatical approach to mine detection</title>

Christopher Raphael, Stuart Geman

Artificial Intelligence · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 22 Jul 1997 · cited 5× · DOI ↗ · more like this ·

We demonstrate a three stage procedure for mine detection. The first phase locates plausible candidates through a series of hypothesis tests. The second state finds the globally optimal convex set containing each candidate using 'course-to-fine' dynamic programming. The third stage refines the mine shape estimates using the knowledge contained in a mine-shape library.

Dynamic programming algorithms for maximum likelihood decoding

Stuart Geman, Kevin Geoffrey Kochanek

Artificial Intelligence · 1998 · cited 3× · OpenAlex ↗ · more like this ·

The Viterbi algorithm is the traditional prototype dynamic programming algorithm for maximum likelihood decoding. Seen from the perspective of formal language theory, this algorithm recursively parses a trellis code's regular grammar. This thesis discusses generalized Viterbi algorithms for the maximum likelihood decoding of codes generated by context-free grammars and transmitted across either memoryless or Markov communications channels. Among the codes representable by context-free grammars are iterated squaring constructions--including the Reed-Muller codes. Two additional strategies are introduced for handling large Reed-Muller-like codes. First, by systematically discarding information bits, a code's grammatical and decoding complexities can be reduced to manageable levels without seriously reducing its information capacity. Second, a coarse-to-fine dynamic programming algorithm…

Statistical inference and probabilistic modeling in compositional vision

Stuart Geman, Wei Zhang

Artificial Intelligence · 2009 · cited 2× · OpenAlex ↗ · more like this ·

This thesis is a mathematical and computational study of compositional vision. Three topics are covered: (1) ROC performance in a compositional world; (2) the construction of a probabilistic model for compositional structure; and (3) the construction of probabilistic model of image gray levels for a given vocabulary of elementary parts. Chapter 1 introduces compositional vision and a probabilistic framework for modeling hierarchy, reusability, and conditional data models. Chapter 2 focuses on theoretical questions about the ROC performance of various approaches to recognition in hypothetical compositional worlds. The results suggest that even sub-optimal decisions within a hierarchical framework will substantially outperform a decision process that does not explicitly allow for part-based decomposition. Chapter 3 focuses on the first component of the Bayesian approach to…

VIEW Communicated by Lawrence Jackel Neural Networks and the BiadVariance Dilemma

Stuart Geman, Elie Bienenstock, Renc Doursat

Communication · 2 Sep 2015 · cited 0× · OpenAlex ↗ · more like this ·

Feedforward neural networks trained by error backpropagation are ex-amples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numer-als. In way of conclusion, we suggest that current-generation feed-forward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals. 1