Author

Yang Fu

359 papers · 1994–2026

1994peak 23 in 20242026

Frequent co-authors

Papers

showing 1–25 of 359 RSS

SSD: Single Shot MultiBox Detector

Wei Liu, Dragomir Anguelov, Dumitru Erhan +4 more

cs.CV · December 2015 · cited 36,286× · arXiv ↗ · more like this ·

We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. Our SSD model is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stage and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require…

DSSD : Deconvolutional Single Shot Detector

Cheng-Yang Fu, Wei Liu, Ananth Ranga +2 more

cs.CV · 23 Jan 2017 · cited 2,082× · arXiv ↗ · more like this ·

The main contribution of this paper is an approach for introducing additional context into state-of-the-art general object detection. To achieve this we first combine a state-of-the-art classifier (Residual-101[14]) with a fast detection framework (SSD[18]). We then augment SSD+Residual-101 with deconvolution layers to introduce additional large-scale context in object detection and improve accuracy, especially for small objects, calling our resulting system DSSD for deconvolutional single shot detector. While these two contributions are easily described at a high-level, a naive implementation does not succeed. Instead we show that carefully adding additional stages of learned transformations, specifically a module for feed-forward connections in deconvolution and a new output module, enables this new approach and forms a potential way forward for further detection research. Results are…

Token Merging: Your ViT But Faster

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai +3 more

cs.CV · October 2022 · cited 1,322× · arXiv ↗ · more like this ·

We introduce Token Merging (ToMe), a simple method to increase the throughput of existing ViT models without needing to train. ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm that is as fast as pruning while being more accurate. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.2x the throughput of ViT-L on video with only a 0.2-0.3% accuracy drop in each case. ToMe can also easily be applied during training, improving in practice training speed up to 2x for MAE fine-tuning on video. Training with ToMe further minimizes accuracy drop, leading to 2x the throughput of ViT-B on audio for only a 0.4% mAP drop. Qualitatively, we find that ToMe merges object parts into one token, even over multiple frames of video. Overall, ToMe's accuracy and speed are competitive with…

HIN2Vec

Tao-Yang Fu, Wang-Chien Lee, Zhen Lei

Artificial Intelligence · 6 Nov 2017 · cited 618× · DOI ↗ · more like this ·

In this paper, we propose a novel representation learning framework, namely HIN2Vec, for heterogeneous information networks (HINs). The core of the proposed framework is a neural network model, also called HIN2Vec, designed to capture the rich semantics embedded in HINs by exploiting different types of relationships among nodes. Given a set of relationships specified in forms of meta-paths in an HIN, HIN2Vec carries out multiple prediction training tasks jointly based on a target set of relationships to learn latent vectors of nodes and meta-paths in the HIN. In addition to model design, several issues unique to HIN2Vec, including regularization of meta-path vectors, node type selection in negative sampling, and cycles in random walks, are examined. To validate our ideas, we learn latent vectors of nodes using four large-scale real HIN datasets, including Blogcatalog, Yelp, DBLP and…

Horizontal Pyramid Matching for Person Re-identification

Yang Fu, Yunchao Wei, Yuqian Zhou +5 more

cs.CV · April 2018 · cited 512× · arXiv ↗ · more like this ·

Despite the remarkable recent progress, person re-identification (Re-ID) approaches are still suffering from the failure cases where the discriminative body parts are missing. To mitigate such cases, we propose a simple yet effective Horizontal Pyramid Matching (HPM) approach to fully exploit various partial information of a given person, so that correct person candidates can be still identified even even some key parts are missing. Within the HPM, we make the following contributions to produce a more robust feature representation for the Re-ID task: 1) we learn to classify using partial feature representations at different horizontal pyramid scales, which successfully enhance the discriminative capabilities of various person parts; 2) we exploit average and max pooling strategies to account for person-specific discriminative information in a global-local manner. To validate the…

STA: Spatial-Temporal Attention for Large-Scale Video-based Person Re-Identification

Yang Fu, Xiaoyang Wang, Yunchao Wei +1 more

cs.CV · 9 Nov 2018 · cited 234× · arXiv ↗ · more like this ·

In this work, we propose a novel Spatial-Temporal Attention (STA) approach to tackle the large-scale person re-identification task in videos. Different from the most existing methods, which simply compute representations of video clips using frame-level aggregation (e.g. average pooling), the proposed STA adopts a more effective way for producing robust clip-level feature representation. Concretely, our STA fully exploits those discriminative parts of one target person in both spatial and temporal dimensions, which results in a 2-D attention score matrix via inter-frame regularization to measure the importances of spatial parts across different frames. Thus, a more robust clip-level feature representation can be generated according to a weighted sum operation guided by the mined 2-D attention score matrix. In this way, the challenging cases for video-based person re-identification such…

RetinaMask: Learning to predict masks improves state-of-the-art single-shot detection for free

Cheng-Yang Fu, Mykhailo Shvets, Alexander C. Berg

cs.CV · 10 Jan 2019 · cited 154× · arXiv ↗ · more like this ·

Recently two-stage detectors have surged ahead of single-shot detectors in the accuracy-vs-speed trade-off. Nevertheless single-shot detectors are immensely popular in embedded vision applications. This paper brings single-shot detectors up to the same level as current two-stage techniques. We do this by improving training for the state-of-the-art single-shot detector, RetinaNet, in three ways: integrating instance mask prediction for the first time, making the loss function adaptive and more stable, and including additional hard examples in training. We call the resulting augmented network RetinaMask. The detection component of RetinaMask has the same computational cost as the original RetinaNet, but is more accurate. COCO test-dev results are up to 41.4 mAP for RetinaMask-101 vs 39.1mAP for RetinaNet-101, while the runtime is the same during evaluation. Adding Group Normalization…

Fast Single Shot Detection and Pose Estimation

Patrick Poirson, Phil Ammirato, Cheng-Yang Fu +3 more

cs.CV · 19 Sep 2016 · cited 112× · arXiv ↗ · more like this ·

For applications in navigation and robotics, estimating the 3D pose of objects is as important as detection. Many approaches to pose estimation rely on detecting or tracking parts or keypoints [11, 21]. In this paper we build on a recent state-of-the-art convolutional network for slidingwindow detection [10] to provide detection and rough pose estimation in a single shot, without intermediate stages of detecting parts or initial bounding boxes. While not the first system to treat pose estimation as a categorization problem, this is the first attempt to combine detection and pose estimation at the same level using a deep learning approach. The key to the architecture is a deep convolutional network where scores for the presence of an object category, the offset for its location, and the approximate pose are all estimated on a regular grid of locations in the image. The resulting system…

Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset

Yang Fu, Xiaolong Wang

cs.CV · 30 Jun 2022 · cited 104× · arXiv ↗ · more like this ·

6D object pose estimation is one of the fundamental problems in computer vision and robotics research. While a lot of recent efforts have been made on generalizing pose estimation to novel object instances within the same category, namely category-level 6D pose estimation, it is still restricted in constrained environments given the limited number of annotated data. In this paper, we collect Wild6D, a new unlabeled RGBD object video dataset with diverse instances and backgrounds. We utilize this data to generalize category-level 6D object pose estimation in the wild with semi-supervised learning. We propose a new model, called Rendering for Pose estimation network RePoNet, that is jointly trained using the free ground-truths with the synthetic data, and a silhouette matching objective function on the real-world data. Without using any 3D annotations on real data, our method outperforms…

Perceptual digital watermark of images using wavelet transform

Zhihui Wei, Pengjie Qin, Yang Fu

Computer Vision and Pattern Recognition · IEEE Transactions on Consumer Electronics · 1998 · cited 94× · DOI ↗ · more like this ·

A perceptually based watermarking technique for an image is proposed. Based on the just-noticeable difference model in the wavelet domain, the watermark is inserted in the wavelet coefficients and its amplitudes are controlled by the wavelet coefficients so that watermark noise does not exceed the just-noticeable difference of each wavelet coefficient. Meanwhile, the order of inserting watermark noise in the wavelet coefficients is the same as the order of the visual significance of the wavelet coefficients. Hence, our approach is adapted with the image content and transparency, and the robustness of the digital watermark can be guaranteed. Experimental results show our approach has excellent performance in the quality of the watermarked images and the robustness of the watermark.

Trembr

Tao-Yang Fu, Wang-Chien Lee

Signal Processing · ACM Transactions on Intelligent Systems and Technology · 4 Feb 2020 · cited 93× · DOI ↗ · more like this ·

In this article, we propose a novel representation learning framework, namely TRajectory EMBedding via Road networks (Trembr) , to learn trajectory embeddings (low-dimensional feature vectors) for use in a variety of trajectory applications. The novelty of Trembr lies in (1) the design of a recurrent neural network--(RNN) based encoder--decoder model, namely Traj2Vec , that encodes spatial and temporal properties inherent in trajectories into trajectory embeddings by exploiting the underlying road networks to constrain the learning process in accordance with the matched road segments obtained using road network matching techniques (e.g., Barefoot [24, 27]), and (2) the design of a neural network--based model, namely Road2Vec , to learn road segment embeddings in road networks that captures various relationships amongst road segments in preparation for trajectory representation learning.…

FACET: Fairness in Computer Vision Evaluation Benchmark

Laura Gustafson, Chloe Rolland, Nikhila Ravi +5 more

cs.CV · 31 Aug 2023 · cited 89× · arXiv ↗ · more like this ·

Computer vision models have known performance disparities across attributes such as gender and skin tone. This means during tasks such as classification and detection, model performance differs for certain classes based on the demographics of the people in the image. These disparities have been shown to exist, but until now there has not been a unified approach to measure these differences for common use-cases of computer vision models. We present a new benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large, publicly available evaluation set of 32k images for some of the most common vision tasks - image classification, object detection and segmentation. For every image in FACET, we hired expert reviewers to manually annotate person-related attributes such as perceived skin tone and hair type, manually draw bounding boxes and label fine-grained person-related classes…

Learning Based Energy Efficient Task Offloading for Vehicular Collaborative Edge Computing

Peng Qin, Yang Fu, Guoming Tang +2 more

Computer Networks and Communications · IEEE Transactions on Vehicular Technology · 29 Apr 2022 · cited 89× · DOI ↗ · more like this ·

Extensive delay-sensitive and computation-intensive tasks are involved in emerging vehicular applications. These tasks can hardly be all processed by the resource constrained vehicle alone, nor fully offloaded to edge facilities (like road side units) due to their incomplete coverage. To this end, we refer to the new paradigm of vehicular collaborative edge computing (VCEC) and make the best use of vehicles’ idle and redundant resources for energy consumption reduction within the VCEC system. To realize this target, we are faced with several nontrivial challenges, including short-term decision making coupled with long-term queue delay constraints, information uncertainty, and task offloading conflicts. Accordingly, we apply Lyapunov optimization to decouple the original problem into three sub-problems and then tackle them one by one: the first sub-problem is resolved by Lagrange…

CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation

Yang Fu, Linjie Yang, Ding Liu +2 more

cs.CV · 7 Dec 2020 · cited 80× · arXiv ↗ · more like this ·

Video instance segmentation is a complex task in which we need to detect, segment, and track each object for any given video. Previous approaches only utilize single-frame features for the detection, segmentation, and tracking of objects and they suffer in the video scenario due to several distinct challenges such as motion blur and drastic appearance change. To eliminate ambiguities introduced by only using single-frame features, we propose a novel comprehensive feature aggregation approach (CompFeat) to refine features at both frame-level and object-level with temporal and spatial context information. The aggregation process is carefully designed with a new attention mechanism which significantly increases the discriminative power of the learned features. We further improve the tracking capability of our model through a siamese design by incorporating both feature similarities and…

Semi-Supervised Multi-Modal Multi-Instance Multi-Label Deep Network with Optimal Transport

Yang Yang, Zhao-Yang Fu, De-Chuan Zhan +2 more

cs.LG · 17 Apr 2021 · cited 71× · arXiv ↗ · more like this ·

Complex objects are usually with multiple labels, and can be represented by multiple modal representations, e.g., the complex articles contain text and image information as well as multiple annotations. Previous methods assume that the homogeneous multi-modal data are consistent, while in real applications, the raw data are disordered, e.g., the article constitutes with variable number of inconsistent text and image instances. Therefore, Multi-modal Multi-instance Multi-label (M3) learning provides a framework for handling such task and has exhibited excellent performance. However, M3 learning is facing two main challenges: 1) how to effectively utilize label correlation; 2) how to take advantage of multi-modal learning to process unlabeled instances. To solve these problems, we first propose a novel Multi-modal Multi-instance Multi-label Deep Network (M3DN), which considers M3 learning…

Attacking the One-Out-Of-m Multicore Problem by Combining Hardware Management with Mixed-Criticality Provisioning

Namhoon Kim, Bryan C. Ward, Micaiah Chisholm +3 more

Hardware and Architecture · April 2016 · cited 62× · DOI ↗ · more like this ·

The multicore revolution is having limited impact in safety-critical application domains. A key reason is the "one-out-of-m" problem: when validating real-time constraints on an m-core platform, excessive analysis pessimism can effectively negate the processing capacity of the additional m-1 cores so that only "one core's worth" of capacity is available. Two approaches have been investigated previously to address this problem: mixed-criticality allocation techniques, which provision less-critical software components less pessimistically, and hardware-management techniques, which make the underlying platform itself more predictable. A better way forward may be to combine both approaches, but to show this, fundamentally new criticality-cognizant hardware-management tradeoffs must be explored. Such tradeoffs are investigated herein in the context of a large-scale, overhead-aware…

DeepIST: Deep Image-based Spatio-Temporal Network for Travel Time Estimation

Tao-yang Fu, Wang-Chien Lee

cs.CV · 5 Sep 2019 · cited 54× · arXiv ↗ · more like this ·

Estimating the travel time for a given path is a fundamental problem in many urban transportation systems. However, prior works fail to well capture moving behaviors embedded in paths and thus do not estimate the travel time accurately. To fill in this gap, in this work, we propose a novel neural network framework, namely {\em Deep Image-based Spatio-Temporal network (DeepIST)}, for travel time estimation of a given path. The novelty of DeepIST lies in the following aspects: 1) we propose to plot a path as a sequence of "generalized images" which include sub-paths along with additional information, such as traffic conditions, road network and traffic signals, in order to harness the power of convolutional neural network model (CNN) on image processing; 2) we design a novel two-dimensional CNN, namely {\em PathCNN}, to extract spatial patterns for lines in images by regularization and…

DETERMINANTS OF INNOVATIVE ACTIVITIES: EVIDENCE FROM EUROPE AND CENTRAL ASIA REGION

Chi Keung Marco Lau, Yang Fu, Zhe Zhang +1 more

Sociology and Political Science · The Singapore Economic Review · 26 Feb 2015 · cited 50× · DOI ↗ · more like this ·

Recent studies in the innovation literature show that Foreign Direct Investment (FDI) enhances innovations in recipient countries through spill-over effects. In this paper we extend the existing literature by incorporating the corruption index in the estimation procedure. Using a cross-country analysis from the Europe and Central Asia (ECA) region, covering 57 countries over the period of 1995–2010, we find no evidence of FDI spill-over effects on innovations, when corruption is endogenously modelled in the regression. Interestingly, we find that corruption and expenditure on education sector are positively related to the number of patents applications, suggesting anti-corruption programs encourage innovations that promote economic growth. Our study shed light on the national innovations and anti-corruption programs.

A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose

Kaiwen Jiang, Yang Fu, Mukund Varma T +4 more

cs.CV · May 2024 · cited 49× · arXiv ↗ · more like this ·

Novel view synthesis from a sparse set of input images is a challenging problem of great practical interest, especially when camera poses are absent or inaccurate. Direct optimization of camera poses and usage of estimated depths in neural radiance field algorithms usually do not produce good results because of the coupling between poses and depths, and inaccuracies in monocular depth estimation. In this paper, we leverage the recent 3D Gaussian splatting method to develop a novel construct-and-optimize method for sparse view synthesis without camera poses. Specifically, we construct a solution progressively by using monocular depth and projecting pixels back into the 3D world. During construction, we optimize the solution by detecting 2D correspondences between training views and the corresponding rendered images. We develop a unified differentiable pipeline for camera registration and…

Parallelizing Itinerary-Based KNN Query Processing in Wireless Sensor Networks

Tao-Yang Fu, Wen-Chih Peng, Wang-Chien Lee

Computer Networks and Communications · IEEE Transactions on Knowledge and Data Engineering · 25 Jun 2009 · cited 46× · DOI ↗ · more like this ·

Wireless sensor networks have been proposed for facilitating various monitoring applications (e.g., environmental monitoring and military surveillance) over a wide geographical region. In these applications, spatial queries that collect data from wireless sensor networks play an important role. One such query is the K-Nearest Neighbor (KNN) query that facilitates collection of sensor data samples based on a given query location and the number of samples specified (i.e., K). Recently, itinerary-based KNN query processing techniques, which propagate queries and collect data along a predetermined itinerary, have been developed. Prior studies demonstrate that itinerary-based KNN query processing algorithms are able to achieve better energy efficiency than other existing algorithms developed upon tree-based network infrastructures. However, how to derive itineraries for KNN query based on…

HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data

Mengqi Zhang, Yang Fu, Zheng Ding +3 more

cs.CV · 18 Mar 2024 · cited 45× · arXiv ↗ · more like this ·

3D hand-object interaction data is scarce due to the hardware constraints in scaling up the data collection process. In this paper, we propose HOIDiffusion for generating realistic and diverse 3D hand-object interaction data. Our model is a conditional diffusion model that takes both the 3D hand-object geometric structure and text description as inputs for image synthesis. This offers a more controllable and realistic synthesis as we can specify the structure and style inputs in a disentangled manner. HOIDiffusion is trained by leveraging a diffusion model pre-trained on large-scale natural images and a few 3D human demonstrations. Beyond controllable image synthesis, we adopt the generated 3D data for learning 6D object pose estimation and show its effectiveness in improving perception systems. Project page: https://mq-zhang1.github.io/HOIDiffusion

Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation

Tsu-Jui Fu, Licheng Yu, Ning Zhang +4 more

cs.CV · November 2022 · cited 44× · arXiv ↗ · more like this ·

Generating a video given the first several static frames is challenging as it anticipates reasonable future frames with temporal coherence. Besides video prediction, the ability to rewind from the last frame or infilling between the head and tail is also crucial, but they have rarely been explored for video completion. Since there could be different outcomes from the hints of just a few frames, a system that can follow natural language to perform video completion may significantly improve controllability. Inspired by this, we introduce a novel task, text-guided video completion (TVC), which requests the model to generate a video from partial frames guided by an instruction. We then propose Multimodal Masked Video Generation (MMVG) to address this TVC task. During training, MMVG discretizes the video frames into visual tokens and masks most of them to perform video completion from any…

Fostering Voluntary Compliance in the COVID-19 Pandemic: An Analytical Framework of Information Disclosure

Yang Fu, Weihong Ma, Jinjin Wu

Sociology and Political Science · The American Review of Public Administration · 17 Jul 2020 · cited 44× · DOI ↗ · more like this ·

Although there have been studies investigating the relationship between information disclosure and voluntary compliance behaviors, the terrain of such research is largely fragmented and has been rarely tested empirically in the pandemic contexts. This article reviewed the intervention and control of the pandemic from the perspective of information disclosure with reflections on the experience in China. Furthermore, the authors propose a comprehensive framework demonstrating the overall landscape of information disclosure and voluntary compliance behaviors with highlights on (a) the tensions between privacy and information transparency; (b) the trade-offs between policy rigorousness and compliance behaviors; (c) different sources of information and how they influence public behaviors differently; and most importantly, (d) how the variegated configurations and contextualization of factors…

Video Highlight Prediction Using Audience Chat Reactions

Cheng-Yang Fu, Joon Lee, Mohit Bansal +1 more

cs.CL · 26 Jul 2017 · cited 39× · arXiv ↗ · more like this ·

Sports channel video portals offer an exciting domain for research on multimodal, multilingual analysis. We present methods addressing the problem of automatic video highlight prediction based on joint visual features and textual analysis of the real-world audience discourse with complex slang, in both English and traditional Chinese. We present a novel dataset based on League of Legends championships recorded from North American and Taiwanese Twitch.tv channels (will be released for further research), and demonstrate strong results on these using multimodal, character-level CNN-RNN model architectures.

Sustainable Urban Community Development: A Case Study from the Perspective of Self-Governance and Public Participation

Yang Fu, Weihong Ma

Public Administration · Sustainability · 14 Jan 2020 · cited 36× · DOI ↗ · more like this ·

(1) Background: There is a global trend to stimulate sustainable urbanization by updating the hardware of the built environment with green technologies. However, simply greening the city hardware does not ensure a sustainable urban system. In reality, urban communities, as cells of the city, play a crucial role in the sustainable development of the entire city. (2) Methods: This paper conducts a case study by investigating a community in Taipei with semi-structured interviews and other first-hand data. It examines how self-organization, voluntary groups, and the public participation of community members has successfully institutionalized a governing system for the sustainable development of communities; (3) Results: This paper identifies the major actors and mechanisms underpinning the sustainable development of urban communities with a case study in Taipei. The establishment of this…