Author

Ruichi Yu

16 papers · 2014–2024

2014peak 5 in 20172024

Frequent co-authors

Papers

showing 1–16 of 16 RSS

NISP: Pruning Networks using Neuron Importance Score Propagation

Ruichi Yu, Ang Li, Chun-Fu Chen +6 more

cs.CV · November 2017 · cited 889× · arXiv ↗ · more like this ·

To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the second-to-last layer before classification, for a pruned network to retrain its predictive power. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, and formulate network pruning as a binary integer optimization problem and derive a closed-form solution to it for pruning…

VITON: An Image-based Virtual Try-on Network

Xintong Han, Zuxuan Wu, Zhe Wu +2 more

cs.CV · November 2017 · cited 785× · arXiv ↗ · more like this ·

We present an image-based VIirtual Try-On Network (VITON) without using 3D information in any form, which seamlessly transfers a desired clothing item onto the corresponding region of a person using a coarse-to-fine strategy. Conditioned upon a new clothing-agnostic yet descriptive person representation, our framework first generates a coarse synthesized image with the target clothing item overlaid on that same person in the same pose. We further enhance the initial blurry clothing area with a refinement network. The network is trained to learn how much detail to utilize from the target clothing item, and where to apply to the person in order to synthesize a photo-realistic image in which the target item deforms naturally with clear visual patterns. Experiments on our newly collected Zalando dataset demonstrate its promise in the image-based virtual try-on task over state-of-the-art…

Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation

Ruichi Yu, Ang Li, Vlad I. Morariu +1 more

cs.CV · July 2017 · cited 327× · arXiv ↗ · more like this ·

Understanding visual relationships involves identifying the subject, the object, and a predicate relating them. We leverage the strong correlations between the predicate and the (subj,obj) pair (both semantically and spatially) to predict the predicates conditioned on the subjects and the objects. Modeling the three entities jointly more accurately reflects their relationships, but complicates learning since the semantic space of visual relationships is huge and the training data is limited, especially for the long-tail relationships that have few instances. To overcome this, we use knowledge of linguistic statistics to regularize visual model learning. We obtain linguistic knowledge by mining from both training annotations (internal knowledge) and publicly available text, e.g., Wikipedia (external knowledge), computing the conditional probability distribution of a predicate given a…

Dynamic Zoom-in Network for Fast Object Detection in Large Images

Mingfei Gao, Ruichi Yu, Ang Li +2 more

cs.CV · November 2017 · cited 143× · arXiv ↗ · more like this ·

We introduce a generic framework that reduces the computational cost of object detection while retaining accuracy for scenarios where objects with varied sizes appear in high resolution images. Detection progresses in a coarse-to-fine manner, first on a down-sampled version of the image and then on a sequence of higher resolution regions identified as likely to improve the detection accuracy. Built upon reinforcement learning, our approach consists of a model (R-net) that uses coarse detection results to predict the potential accuracy gain for analyzing a region at a higher resolution and another model (Q-net) that sequentially selects regions to zoom in. Experiments on the Caltech Pedestrians dataset show that our approach reduces the number of processed pixels by over 50% without a drop in detection accuracy. The merits of our approach become more significant on a high resolution test…

The Role of Context Selection in Object Detection

Ruichi Yu, Xi Chen, Vlad I. Morariu +1 more

cs.CV · 9 Sep 2016 · cited 43× · arXiv ↗ · more like this ·

We investigate the reasons why context in object detection has limited utility by isolating and evaluating the predictive power of different context cues under ideal conditions in which context provided by an oracle. Based on this study, we propose a region-based context re-scoring method with dynamic context selection to remove noise and emphasize informative context. We introduce latent indicator variables to select (or ignore) potential contextual regions, and learn the selection strategy with latent-SVM. We conduct experiments to evaluate the performance of the proposed context selection method on the SUN RGB-D dataset. The method achieves a significant improvement in terms of mean average precision (mAP), compared with both appearance based detectors and a conventional context model without the selection scheme.

Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking

Longlong Jing, Ruichi Yu, Henrik Kretzschmar +11 more

cs.CV · 8 Jun 2022 · cited 29× · arXiv ↗ · more like this ·

Monocular image-based 3D perception has become an active research area in recent years owing to its applications in autonomous driving. Approaches to monocular 3D perception including detection and tracking, however, often yield inferior performance when compared to LiDAR-based techniques. Through systematic analysis, we identified that per-object depth estimation accuracy is a major factor bounding the performance. Motivated by this observation, we propose a multi-level fusion method that combines different representations (RGB and pseudo-LiDAR) and temporal information across multiple frames for objects (tracklets) to enhance per-object depth estimation. Our proposed fusion method achieves the state-of-the-art performance of per-object depth estimation on the Waymo Open Dataset, the KITTI detection dataset, and the KITTI MOT dataset. We further demonstrate that by simply replacing…

Generating Holistic 3D Scene Abstractions for Text-based Image Retrieval

Ang Li, Jin Sun, Joe Yue-Hei Ng +3 more

cs.CV · November 2016 · cited 25× · arXiv ↗ · more like this ·

Spatial relationships between objects provide important information for text-based image retrieval. As users are more likely to describe a scene from a real world perspective, using 3D spatial relationships rather than 2D relationships that assume a particular viewing direction, one of the main challenges is to infer the 3D structure that bridges images with users' text descriptions. However, direct inference of 3D structure from images requires learning from large scale annotated data. Since interactions between objects can be reduced to a limited set of atomic spatial relations in 3D, we study the possibility of inferring 3D structure from a text description rather than an image, applying physical relation models to synthesize holistic 3D abstract object layouts satisfying the spatial constraints present in a textual description. We present a generic framework for retrieving images…

ReMotENet: Efficient Relevant Motion Event Detection for Large-scale Home Surveillance Videos

Ruichi Yu, Hongcheng Wang, Larry S. Davis

cs.CV · 6 Jan 2018 · cited 23× · arXiv ↗ · more like this ·

This paper addresses the problem of detecting relevant motion caused by objects of interest (e.g., person and vehicles) in large scale home surveillance videos. The traditional method usually consists of two separate steps, i.e., detecting moving objects with background subtraction running on the camera, and filtering out nuisance motion events (e.g., trees, cloud, shadow, rain/snow, flag) with deep learning based object detection and tracking running on cloud. The method is extremely slow and therefore not cost effective, and does not fully leverage the spatial-temporal redundancies with a pre-trained off-the-shelf object detector. To dramatically speedup relevant motion event detection and improve its performance, we propose a novel network for relevant motion event detection, ReMotENet, which is a unified, end-to-end data-driven method using spatial-temporal attention-based 3D…

C-WSL: Count-guided Weakly Supervised Localization

Mingfei Gao, Ang Li, Ruichi Yu +2 more

cs.CV · November 2017 · cited 9× · arXiv ↗ · more like this ·

We introduce count-guided weakly supervised localization (C-WSL), an approach that uses per-class object count as a new form of supervision to improve weakly supervised localization (WSL). C-WSL uses a simple count-based region selection algorithm to select high-quality regions, each of which covers a single object instance during training, and improves existing WSL methods by training with the selected regions. To demonstrate the effectiveness of C-WSL, we integrate it into two WSL architectures and conduct extensive experiments on VOC2007 and VOC2012. Experimental results show that C-WSL leads to large improvements in WSL and that the proposed approach significantly outperforms the state-of-the-art methods. The results of annotation experiments on VOC2007 suggest that a modest extra time is needed to obtain per-class object counts compared to labeling only object categories in an…

SoDA: Multi-Object Tracking with Soft Data Association

Wei-Chih Hung, Henrik Kretzschmar, Tsung-Yi Lin +4 more

cs.CV · 19 Aug 2020 · cited 6× · arXiv ↗ · more like this ·

Robust multi-object tracking (MOT) is a prerequisite fora safe deployment of self-driving cars. Tracking objects, however, remains a highly challenging problem, especially in cluttered autonomous driving scenes in which objects tend to interact with each other in complex ways and frequently get occluded. We propose a novel approach to MOT that uses attention to compute track embeddings that encode the spatiotemporal dependencies between observed objects. This attention measurement encoding allows our model to relax hard data associations, which may lead to unrecoverable errors. Instead, our model aggregates information from all object detections via soft data associations. The resulting latent space representation allows our model to learn to reason about occlusions in a holistic data-driven way and maintain track estimates for objects even when they are occluded. Our experimental…

Mobile App Connecting People Based on Personality Detection and Image Perception Analysis

Ruichi Yu, Shuguan Yang, Guifan Li +3 more

Artificial Intelligence · December 2014 · cited 3× · DOI ↗ · more like this ·

We propose a personality detection algorithm based on human perceptions of images and created a mobile social application named Date gram to test the efficiency of our algorithm and connect people who share similar image perceptions. We detect user's Big Five personality based on their perceptual opinions on several images. Users in Date gram are matched to play visual-sentiment games, through which their connections based on personality similarity are further strengthened. Datagram utilizes techniques of image sentiment detection to analyze user's personality. Given the collected personal sentiment perception data, we proposed personality-mapping algorithm to categorize users into "Big Five" personality. After the Adjective-Noun-Pairs (ANPs) are extracted from each image, a personalized image-profile that consists of multiple pairs of ANPs and corresponding user decisions is generated.…

R4D: Utilizing Reference Objects for Long-Range Distance Estimation

Yingwei Li, Tiffany Chen, Maya Kabkab +4 more

cs.CV · 10 Jun 2022 · cited 2× · arXiv ↗ · more like this ·

Estimating the distance of objects is a safety-critical task for autonomous driving. Focusing on short-range objects, existing methods and datasets neglect the equally important long-range objects. In this paper, we introduce a challenging and under-explored task, which we refer to as Long-Range Distance Estimation, as well as two datasets to validate new methods developed for this task. We then proposeR4D, the first framework to accurately estimate the distance of long-range objects by using references with known distances in the scene. Drawing inspiration from human perception, R4D builds a graph by connecting a target object to all references. An edge in the graph encodes the relative distance information between a pair of target and reference objects. An attention module is then used to weigh the importance of reference objects and combine them into one target object distance…

Uncertainty Modeling of Contextual-Connections between Tracklets for Unconstrained Video-based Face Recognition

Jingxiao Zheng, Ruichi Yu, Jun-Cheng Chen +3 more

cs.CV · May 2019 · cited 1× · arXiv ↗ · more like this ·

Unconstrained video-based face recognition is a challenging problem due to significant within-video variations caused by pose, occlusion and blur. To tackle this problem, an effective idea is to propagate the identity from high-quality faces to low-quality ones through contextual connections, which are constructed based on context such as body appearance. However, previous methods have often propagated erroneous information due to lack of uncertainty modeling of the noisy contextual connections. In this paper, we propose the Uncertainty-Gated Graph (UGG), which conducts graph-based identity propagation between tracklets, which are represented by nodes in a graph. UGG explicitly models the uncertainty of the contextual connections by adaptively updating the weights of the edge gates according to the identity distributions of the nodes during inference. UGG is a generic graphical model…

Layout-induced Video Representation for Recognizing Agent-in-Place Actions

Ruichi Yu, Hongcheng Wang, Ang Li +3 more

cs.CV · April 2018 · cited 0× · arXiv ↗ · more like this ·

We address the recognition of agent-in-place actions, which are associated with agents who perform them and places where they occur, in the context of outdoor home surveillance. We introduce a representation of the geometry and topology of scene layouts so that a network can generalize from the layouts observed in the training set to unseen layouts in the test set. This Layout-Induced Video Representation (LIVR) abstracts away low-level appearance variance and encodes geometric and topological relationships of places in a specific scene layout. LIVR partitions the semantic features of a video clip into different places to force the network to learn place-based feature descriptions; to predict the confidence of each action, LIVR aggregates features from the place associated with an action and its adjacent places on the scene layout. We introduce the Agent-in-Place Action dataset to show…

STT: Stateful Tracking with Transformers for Autonomous Driving

Longlong Jing, Ruichi Yu, Xu Chen +19 more

cs.RO · 30 Apr 2024 · arXiv ↗ · more like this ·

Tracking objects in three-dimensional space is critical for autonomous driving. To ensure safety while driving, the tracker must be able to reliably track objects across frames and accurately estimate their states such as velocity and acceleration in the present. Existing works frequently focus on the association task while either neglecting the model performance on state estimation or deploying complex heuristics to predict the states. In this paper, we propose STT, a Stateful Tracking model built with Transformers, that can consistently track objects in the scenes while also predicting their states accurately. STT consumes rich appearance, geometry, and motion signals through long term history of detections and is jointly optimized for both data association and state estimation tasks. Since the standard tracking metrics like MOTA and MOTP do not capture the combined performance of the…

Modeling Local Geometric Structure of 3D Point Clouds using Geo-CNN

Shiyi Lan, Ruichi Yu, Gang Yu +1 more

cs.CV · 19 Nov 2018 · arXiv ↗ · more like this ·

Recent advances in deep convolutional neural networks (CNNs) have motivated researchers to adapt CNNs to directly model points in 3D point clouds. Modeling local structure has been proven to be important for the success of convolutional architectures, and researchers exploited the modeling of local point sets in the feature extraction hierarchy. However, limited attention has been paid to explicitly model the geometric structure amongst points in a local region. To address this problem, we propose Geo-CNN, which applies a generic convolution-like operation dubbed as GeoConv to each point and its local neighborhood. Local geometric relationships among points are captured when extracting edge features between the center and its neighboring points. We first decompose the edge feature extraction process onto three orthogonal bases, and then aggregate the extracted features based on the…