Author

Alex Colburn

31 papers · 2000–2026

2000peak 5 in 20232026

Frequent co-authors

Papers

showing 1–25 of 31 RSS

Interactive digital photomontage

Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala +5 more

Computer Vision and Pattern Recognition · ACM Transactions on Graphics · August 2004 · cited 837× · DOI ↗ · more like this ·

We describe an interactive, computer-assisted framework for combining parts of a set of photographs into a single composite picture, a process we call "digital photomontage." Our framework makes use of two techniques primarily: graph-cut optimization, to choose good seams within the constituent images so that they can be combined as seamlessly as possible; and gradient-domain fusion, a process based on Poisson equations, to further reduce any remaining visible artifacts in the composite. Also central to the framework is a suite of interactive tools that allow the user to specify a variety of high-level image objectives, either globally across the image, or locally through a painting-style interface. Image objectives are applied independently at each pixel location and generally involve a function of the pixel values (such as "maximum contrast") drawn from that same location in the set…

Interactive video cutout

Jue Wang, Pravin Bhat, Alex Colburn +2 more

Computer Vision and Pattern Recognition · ACM Transactions on Graphics · July 2005 · cited 330× · DOI ↗ · more like this ·

We present an interactive system for efficiently extracting foreground objects from a video. We extend previous min-cut based image segmentation techniques to the domain of video with four new contributions. We provide a novel painting-based user interface that allows users to easily indicate the foreground object across space and time. We introduce a hierarchical mean-shift preprocess in order to minimize the number of nodes that min-cut must operate on. Within the min-cut we also define new local cost functions to augment the global costs defined in earlier work. Finally, we extend 2D alpha matting methods designed for images to work with 3D video volumes. We demonstrate that our matting approach preserves smoothness across both space and time. Our interactive video cutout system allows users to quickly extract foreground objects from video sequences for use in a variety of…

LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image

Chuhang Zou, Alex Colburn, Qi Shan +1 more

cs.CV · 23 Mar 2018 · cited 267× · arXiv ↗ · more like this ·

We propose an algorithm to predict room layout from a single image that generalizes across panoramas and perspective images, cuboid layouts and more general layouts (e.g. L-shape room). Our method operates directly on the panoramic image, rather than decomposing into perspective images as do recent works. Our network architecture is similar to that of RoomNet, but we show improvements due to aligning the image based on vanishing points, predicting multiple layout elements (corners, boundaries, size and translation), and fitting a constrained Manhattan layout to the resulting predictions. Our method compares well in speed and accuracy to other existing work on panoramas, achieves among the best accuracy for perspective images, and can handle both cuboid-shaped and more general Manhattan layouts.

Distributed meetings

Ross Cutler, Yong Rui, Anoop Gupta +7 more

Computer Vision and Pattern Recognition · December 2002 · cited 227× · DOI ↗ · more like this ·

The common meeting is an integral part of everyday life for most workgroups. However, due to travel, time, or other constraints, people are often not able to attend all the meetings they need to. Teleconferencing and recording of meetings can address this problem. In this paper we describe a system that provides these features, as well as a user study evaluation of the system. The system uses a variety of capture devices (a novel 360° camera, a whiteboard camera, an overview camera, and a microphone array) to provide a rich experience for people who want to participate in a meeting from a distance. The system is also combined with speaker clustering, spatial indexing, and time compression to provide a rich experience for people who miss a meeting and want to watch it afterward.

A Viewer-Centric Editor for 3D Movies

Sanjeev J. Koppal, C. Lawrence Zitnick, Michael Cohen +3 more

Computer Vision and Pattern Recognition · IEEE Computer Graphics and Applications · 19 Mar 2010 · cited 58× · DOI ↗ · more like this ·

A proposed mathematical framework is the basis for a viewer-centric digital editor for 3D movies that's driven by the audience's perception of the scene. The editing tool allows both shot planning and after-the-fact digital manipulation of the perceived scene shape.

Fast and Explicit Neural View Synthesis

Pengsheng Guo, Miguel Angel Bautista, Alex Colburn +4 more

cs.CV · July 2021 · cited 41× · arXiv ↗ · more like this ·

We study the problem of novel view synthesis from sparse source observations of a scene comprised of 3D objects. We propose a simple yet effective approach that is neither continuous nor implicit, challenging recent trends on view synthesis. Our approach explicitly encodes observations into a volumetric representation that enables amortized rendering. We demonstrate that although continuous radiance field representations have gained a lot of attention due to their expressive power, our simple approach obtains comparable or even better novel view reconstruction quality comparing with state-of-the-art baselines while increasing rendering speed by over 400x. Our model is trained in a category-agnostic manner and does not require scene-specific optimization. Therefore, it is able to generalize novel view synthesis to object categories not seen during training. In addition, we show that with…

FineRecon: Depth-aware Feed-forward Network for Detailed 3D Reconstruction

Noah Stier, Anurag Ranjan, Alex Colburn +4 more

cs.CV · April 2023 · cited 32× · arXiv ↗ · more like this ·

Recent works on 3D reconstruction from posed images have demonstrated that direct inference of scene-level 3D geometry without test-time optimization is feasible using deep neural networks, showing remarkable promise and high efficiency. However, the reconstructed geometry, typically represented as a 3D truncated signed distance function (TSDF), is often coarse without fine geometric details. To address this problem, we propose three effective solutions for improving the fidelity of inference-based 3D reconstructions. We first present a resolution-agnostic TSDF supervision strategy to provide the network with a more accurate learning signal during training, avoiding the pitfalls of TSDF interpolation seen in previous work. We then introduce a depth guidance strategy using multi-view depth estimates to enhance the scene representation and recover more accurate surfaces. Finally, we…

Manhattan Room Layout Reconstruction from a Single 360 image: A Comparative Study of State-of-the-art Methods

Chuhang Zou, Jheng-Wei Su, Chi-Han Peng +5 more

cs.CV · October 2019 · cited 31× · arXiv ↗ · more like this ·

Recent approaches for predicting layouts from 360 panoramas produce excellent results. These approaches build on a common framework consisting of three steps: a pre-processing step based on edge-based alignment, prediction of layout elements, and a post-processing step by fitting a 3D layout to the layout elements. Until now, it has been difficult to compare the methods due to multiple different design decisions, such as the encoding network (e.g. SegNet or ResNet), type of elements predicted (e.g. corners, wall/floor boundaries, or semantic segmentation), or method of fitting the 3D layout. To address this challenge, we summarize and describe the common framework, the variants, and the impact of the design decisions. For a complete evaluation, we also propose extended annotations for the Matterport3D dataset [3], and introduce two depth-based evaluation metrics.

Learning to Generate 3D Stylized Character Expressions from Humans

Deepali Aneja, Bindita Chaudhuri, Alex Colburn +3 more

Computer Vision and Pattern Recognition · March 2018 · cited 29× · DOI ↗ · more like this ·

We present ExprGen, a system to automatically generate 3D stylized character expressions from humans in a perceptually valid and geometrically consistent manner. Our multi-stage deep learning system utilizes the latent variables of human and character expression recognition convolutional neural networks to control a 3D animated character rig. This end-to-end system takes images of human faces and generates the character rig parameters that best match the human's facial expression. ExprGen generalizes to multiple characters, and allows expression transfer between characters in a semi-supervised manner. Qualitative and quantitative evaluation of our method based on Mechanical Turk tests show the high perceptual accuracy of our expression transfer results.

Parallax photography: creating 3D cinematic effects from stills

Ke Zheng, Alex Colburn, Aseem Agarwala +4 more

Computer Vision and Pattern Recognition · Graphics Interface · 25 May 2009 · cited 28× · OpenAlex ↗ · more like this ·

We present an approach to convert a small portion of a light field with extracted depth information into a cinematic effect with simulated, smooth camera motion that exhibits a sense of 3D parallax. We develop a taxonomy of the cinematic conventions of these effects, distilled from observations of documentary film footage and organized by the number of subjects of interest in the scene. We present an automatic, content-aware approach to apply these cinematic conventions to an input light field. A face detector identifies subjects of interest. We then optimize for a camera path that conforms to a cinematic convention, maximizes apparent parallax, and avoids missing information in the input. We describe a GPU-accelerated, temporally coherent rendering algorithm that allows users to create more complex camera moves interactively, while experimenting with effects such as focal length, depth…

AutoFocusFormer: Image Segmentation off the Grid

Chen Ziwen, Kaushik Patnaik, Shuangfei Zhai +5 more

cs.CV · April 2023 · cited 21× · arXiv ↗ · more like this ·

Real world images often have highly imbalanced content density. Some areas are very uniform, e.g., large patches of blue sky, while other areas are scattered with many small objects. Yet, the commonly used successive grid downsampling strategy in convolutional deep networks treats all areas equally. Hence, small objects are represented in very few spatial locations, leading to worse results in tasks such as segmentation. Intuitively, retaining more pixels representing small objects during downsampling helps to preserve important information. To achieve this, we propose AutoFocusFormer (AFF), a local-attention transformer image recognition backbone, which performs adaptive downsampling by learning to retain the most important pixels for the task. Since adaptive downsampling generates a set of pixels irregularly distributed on the image plane, we abandon the classic grid structure.…

Video Cubism

Michael Cohen, Alex Colburn, Adam Finkelstein +2 more

Computer Graphics and Computer-Aided Design · April 2001 · cited 21× · OpenAlex ↗ · more like this ·

We present new non-photorealistic (NPR) rendering tools for video. Inspired by the Cubist and Futurist art movements that questioned previous notions of space and time within paintings, we view an input video as a space-time cube of data, rather than a series of static frames. Our tools process the video as a whole to produce a set of stroke-solids , units for rendering that appear over multiple frames in the resulting NPR video. Depending on stylistic considerations and video content, each stroke solid is encoded with parameters such as location, size, curvature, orientation, color values, or other relevant information. A non-photorealistic video is constructed interactively by compositing slices of the stroke solids. These slices, or strokes , are rendered as multi-textured sprites. The textures for each stroke may be derived from a pre-defined texture atlas and/or drawn from the…

Image-Based Remodeling

Alex Colburn, A. Agarwala, Aaron Hertzmann +2 more

Computer Vision and Pattern Recognition · IEEE Transactions on Visualization and Computer Graphics · 11 Apr 2012 · cited 18× · DOI ↗ · more like this ·

Imagining what a proposed home remodel might look like without actually performing it is challenging. We present an image-based remodeling methodology that allows real-time photorealistic visualization during both the modeling and remodeling process of a home interior. Large-scale edits, like removing a wall or enlarging a window, are performed easily and in real time, with realistic results. Our interface supports the creation of concise, parameterized, and constrained geometry, as well as remodeling directly from within the photographs. Real-time texturing of modified geometry is made possible by precomputing view-dependent textures for all faces that are potentially visible to each original camera viewpoint, blending multiple viewpoints and hole-filling when necessary. The resulting textures are stored and accessed efficiently enabling intuitive real-time realistic visualization…

Image Stacks

Michael Cohen, Alex Colburn, Steven M. Drucker

Computer Vision and Pattern Recognition · July 2003 · cited 18× · OpenAlex ↗ · more like this ·

We present a simple but powerful Image Stack process for creating an enhanced image from a stack of registered images. This paradigm combines pixels using multi-image operations on a set of images of the same subject matter. We demonstrate how Image Stacks can help create group photographs, enhance high dynamic range images, combine images captured under different lighting conditions, remove unwanted objects from images, and combine images captured at different times and with different focal lengths.

Graphical Enhancements for Voice Only Conference Calls

Alex Colburn, Michael Cohen, Steven M. Drucker +3 more

Artificial Intelligence · October 2001 · cited 7× · OpenAlex ↗ · more like this ·

We present two very low bandwidth graphically enhanced interfaces for small group voice communications. One interface presents static images of the participants that highlight when one is speaking. The other interface utilizes three-dimensional avatars that can be quickly created. Eleven groups of 4 or 5 people were presented with each enhanced interface as well as conducting a live conversation and a voice only conversation. Experiments show that both graphically enhanced interfaces improve the understandability of conversations, particular with respect to impressions that others in the group could express themselves more easily, knowing who is talking, and when to speak. Little difference was found between the two graphical interfaces. Analysis of voice tracks also revealed differences between interfaces in the length and number of medium duration silences.

A Consistent Segmentation Approach to Image-based Rendering

Ke Zheng, Alex Colburn, Aseem Agarwala +4 more

Computer Vision and Pattern Recognition · 2009 · cited 2× · OpenAlex ↗ · more like this ·

This paper presents an approach to render novel views from input photographs, a task which is commonly referred to as image based rendering. We first compute dense view dependent depthmaps using consistent segmentation. This method jointly computes multiview stereo and segments input photographs while accounting for mixed pixels (matting). We take the images with depth as our input and then propose two rendering algorithms to render novel views using the segmentation, both realtime and off-line. We demonstrate the results of our approach on a wide variety of scenes.

MvFFN: Multi-view Floor plan Feed-forward Network for Unposed Wide-Baseline Panorama Layout Reconstruction

Yuguang Li, Yichuan Deng, Zixuan Liu +5 more

cs.CV · 14 Oct 2026 · cited 0× · arXiv ↗ · more like this ·

Reconstructing accurate camera poses and floor-plan layouts from sparse, unposed, wide-baseline RGB panoramas remains a challenging open problem. Direct multi-room floor-plan prediction is especially difficult in this setting: pose, geometry, semantics, and global layout structure are tightly coupled, yet a complete floor-plan is hard to learn as a single global output. Our key insight is to keep the model's predictions strictly local. Multi-view Ordered Wall Instance Segmentation (MOIS) is designed around this concept. Each panorama column predicts only the wall it observes plus the next two adjacent walls in clockwise order. The global wall partition, room layouts, and shared coordinates emerge from deterministic aggregation of overlapping local predictions. Building on MOIS, we propose Multi-view floor-plan Feed-Forward Network (MvFFN), which jointly predicts coarse camera poses…

StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D

Pengsheng Guo, Hans Hao, Adam Caccavale +7 more

cs.CV · 2 Dec 2023 · cited 0× · arXiv ↗ · more like this ·

In the realm of text-to-3D generation, utilizing 2D diffusion models through score distillation sampling (SDS) frequently leads to issues such as blurred appearances and multi-faced geometry, primarily due to the intrinsically noisy nature of the SDS loss. Our analysis identifies the core of these challenges as the interaction among noise levels in the 2D diffusion process, the architecture of the diffusion network, and the 3D model representation. To overcome these limitations, we present StableDreamer, a methodology incorporating three advances. First, inspired by InstructNeRF2NeRF, we formalize the equivalence of the SDS generative prior and a simple supervised L2 reconstruction loss. This finding provides a novel tool to debug SDS, which we use to show the impact of time-annealing noise levels on reducing multi-faced geometries. Second, our analysis shows that while image-space…

Pseudo-Generalized Dynamic View Synthesis from a Video

Xiaoming Zhao, Alex Colburn, Fangchang Ma +3 more

cs.CV · October 2023 · cited 0× · arXiv ↗ · more like this ·

Rendering scenes observed in a monocular video from novel viewpoints is a challenging problem. For static scenes the community has studied both scene-specific optimization techniques, which optimize on every test scene, and generalized techniques, which only run a deep net forward pass on a test scene. In contrast, for dynamic scenes, scene-specific optimization techniques exist, but, to our best knowledge, there is currently no generalized method for dynamic novel view synthesis from a given monocular video. To answer whether generalized dynamic novel view synthesis from monocular videos is possible today, we establish an analysis framework based on existing techniques and work toward the generalized approach. We find a pseudo-generalized process without scene-specific appearance optimization is possible, but geometrically and temporally consistent depth estimates are needed. Despite…

LivePose: Online 3D Reconstruction from Monocular Video with Dynamic Camera Poses

Noah Stier, Baptiste Angles, Liang Yang +3 more

cs.CV · April 2023 · cited 0× · arXiv ↗ · more like this ·

Dense 3D reconstruction from RGB images traditionally assumes static camera pose estimates. This assumption has endured, even as recent works have increasingly focused on real-time methods for mobile devices. However, the assumption of a fixed pose for each image does not hold for online execution: poses from real-time SLAM are dynamic and may be updated following events such as bundle adjustment and loop closure. This has been addressed in the RGB-D setting, by de-integrating past views and re-integrating them with updated poses, but it remains largely untreated in the RGB-only setting. We formalize this problem to define the new task of dense online reconstruction from dynamically-posed images. To support further research, we introduce a dataset called LivePose containing the dynamic poses from a SLAM system running on ScanNet. We select three recent reconstruction systems and apply a…

Generative Multiplane Images: Making a 2D GAN 3D-Aware

Xiaoming Zhao, Fangchang Ma, David Güera +3 more

cs.CV · 21 Jul 2022 · cited 0× · arXiv ↗ · more like this ·

What is really needed to make an existing 2D GAN 3D-aware? To answer this question, we modify a classical GAN, i.e., StyleGANv2, as little as possible. We find that only two modifications are absolutely necessary: 1) a multiplane image style generator branch which produces a set of alpha maps conditioned on their depth; 2) a pose-conditioned discriminator. We refer to the generated output as a 'generative multiplane image' (GMPI) and emphasize that its renderings are not only high-quality but also guaranteed to be view-consistent, which makes GMPIs different from many prior works. Importantly, the number of alpha maps can be dynamically adjusted and can differ between training and inference, alleviating memory concerns and enabling fast training of GMPIs in less than half a day at a resolution of $1024^2$. Our findings are consistent across three challenging and common high-resolution…

Learning Stylized Character Expressions from Humans

Deepali Aneja, Alex Colburn, Gary Faigin +2 more

cs.CV · 19 Nov 2019 · cited 0× · arXiv ↗ · more like this ·

We present DeepExpr, a novel expression transfer system from humans to multiple stylized characters via deep learning. We developed : 1) a data-driven perceptual model of facial expressions, 2) a novel stylized character data set with cardinal expression annotations : FERG (Facial Expression Research Group) - DB (added two new characters), and 3) . We evaluated our method on a set of retrieval tasks on our collected stylized character dataset of expressions. We have also shown that the ranking order predicted by the proposed features is highly correlated with the ranking order provided by a facial expression expert and Mechanical Turk (MT) experiments.

Abstract Image-Based Remodeling: A Framework for Creating, Visualizing, and Editing Image-Based Models

R. Alex Colburn, R. Alex Colburn, Dieter Fox +1 more

Education · 2014 · cited 0× · OpenAlex ↗ · more like this ·

Image-based models are geometric models created from photographs and textured with the photographs for realistic rendering. In recent years, it has become increasingly easy to capture many photographs of an object and use computer vision techniques to model the object with image-based geometry. However, it can be difficult to display and interact with these models in a manner that reproduces the visual fidelity of the original photographs. It is even more difficult to support the interactive manipulation and editing that one might expect from other types of geometric models. The overarching goal of this thesis is to create a framework for building, visualizing, and editing image-based models. Toward this goal, I first describe the related literature and outline the associated research challenges related to geometric representation, navigation, creation, and editing of image-based…

To appear in an IEEE VGTC sponsored conference proceedings Parallax Photography: Creating 3D Cinematic Effects from Stills

Ke Colin Zheng, Alex Colburn, Aseem Agarwala +4 more

Computer Vision and Pattern Recognition · 12 Aug 2013 · cited 0× · OpenAlex ↗ · more like this ·

We present an approach to convert a small portion of a light field with extracted depth information into a cinematic effect with simulated, smooth camera motion that exhibits a sense of 3D parallax. We develop a taxonomy of the cinematic conventions of these effects, distilled from observations of documentary film footage and organized by the number of subjects of interest in the scene. We present an automatic, content-aware approach to apply these cinematic conventions to an input light field. A face detector identifies subjects of interest. We then optimize for a camera path that conforms to a cinematic convention, maximizes apparent parallax, and avoids missing information in the input. We describe a GPUaccelerated, temporally coherent rendering algorithm that allows users to create more complex camera moves interactively, while experimenting with effects such as focal length, depth…

Adobe Systems, Inc.

Ke Colin Zheng, Alex Colburn, Aseem Agarwala +4 more

Computer Vision and Pattern Recognition · 28 Dec 2009 · cited 0× · OpenAlex ↗ · more like this ·