arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Shang-Fu Chen

17 papers · 2017–2026

2017: 12018: 02019: 12020: 12021: 12022: 42023: 12024: 22025: 22026: 4
2017peak 4 in 20222026

Frequent co-authors

Papers

showing 1–17 of 17 RSS

Order-Free RNN with Visual Attention for Multi-Label Classification

Shang-Fu Chen, Yi-Chen Chen, Chih-Kuan Yeh +1 more

cs.CV · · cited 161× · arXiv ↗ · more like this ·

In this paper, we propose the joint learning attention and recurrent neural network (RNN) models for multi-label classification. While approaches based on the use of either model exist (e.g., for the task of image captioning), training such existing network architectures typically require pre-defined label sequences. For multi-label classification, it would be desirable to have a robust inference process, so that the prediction error would not propagate and thus affect the performance. Our proposed model uniquely integrates attention and Long Short Term Memory (LSTM) models, which not only addresses the above problem but also allows one to identify visual objects of interests with varying sizes without the prior knowledge of particular label ordering. More importantly, label co-occurrence information can be jointly exploited by our LSTM model. Finally, by advancing the technique of beam…

Learning Hierarchical Self-Attention for Video Summarization

Yen‐Ting Liu, Yu-Jhe Li, Fu-En Yang +2 more

Computer Vision and Pattern Recognition · · cited 60× · DOI ↗ · more like this ·

Video summarization still remains a challenging task. Due to sufficient video data on the Internet, such task draws significant attention in the vision community and benefits a wide range of applications, e.g., video retrieval, search, etc. To effectively perform video summarization by deriving the keyframes which represent the given input video, we propose a novel framework named Hierarchical Multi-Attention Network (H-MAN) which comprises the shot-level reconstruction model and multi-head attention model. While our designed attention model is two-stage hierarchical structure for producing various attention maps, we are among the first to utilize the multi-attention mechanism in the video summarization task, which brings improved performance. The quantitative and qualitative results demonstrate the effectiveness of our model, which performs favorably against state-of-the-art approaches.

Controllable Model Compression for Roadside Camera Depth Estimation

Jose Jaena Mari Ople, Shang‐Fu Chen, Yung-Yao Chen +4 more

Computer Vision and Pattern Recognition · IEEE Transactions on Intelligent Transportation Systems · · cited 11× · DOI ↗ · more like this ·

In the Cooperative Intelligent Transportation System (C-ITS) paradigm, vehicles could communicate with roadside units to augment their traffic knowledge. Smart roadside units could provide second-order information (e.g., vehicle count) from raw first-order data (e.g., visual feed, point clouds), and this “smart” feature is usually provided using deep neural network models. However, implementing these useful models implies a cost for computational complexity that could hinder the future deployment of smart roadside units needed for sustainability in transportation systems. In this paper, we propose to use model compression on deep image processing models to promote its feasibility for usage in smart sensors. We formulated a controllable convolutional model compression (CCMC) algorithm that can perform filter-wise evolutionary pruning on image processing networks, along with a predefined…

Domain-Generalized Textured Surface Anomaly Detection

Shang-Fu Chen, Yu-Min Liu, Chia-Ching Lin +2 more

cs.CV · · cited 7× · arXiv ↗ · more like this ·

Anomaly detection aims to identify abnormal data that deviates from the normal ones, while typically requiring a sufficient amount of normal data to train the model for performing this task. Despite the success of recent anomaly detection methods, performing anomaly detection in an unseen domain remain a challenging task. In this paper, we address the task of domain-generalized textured surface anomaly detection. By observing normal and abnormal surface data across multiple source domains, our model is expected to be generalized to an unseen textured surface of interest, in which only a small number of normal data can be observed during testing. Although with only image-level labels observed in the training data, our patch-based meta-learning model exhibits promising generalization ability: not only can it generalize to unseen image domains, but it can also localize abnormal regions in…

Learning Facial Liveness Representation for Domain Generalized Face Anti-spoofing

Zih-Ching Chen, Lin-Hsi Tsao, Chin-Lun Fu +2 more

cs.CV · · cited 4× · arXiv ↗ · more like this ·

Face anti-spoofing (FAS) aims at distinguishing face spoof attacks from the authentic ones, which is typically approached by learning proper models for performing the associated classification task. In practice, one would expect such models to be generalized to FAS in different image domains. Moreover, it is not practical to assume that the type of spoof attacks would be known in advance. In this paper, we propose a deep learning model for addressing the aforementioned domain-generalized face anti-spoofing task. In particular, our proposed network is able to disentangle facial liveness representation from the irrelevant ones (i.e., facial content and image domain features). The resulting liveness representation exhibits sufficient domain invariant properties, and thus it can be applied for performing domain-generalized FAS. In our experiments, we conduct experiments on five benchmark…

Robust Image Outpainting With Learnable Image Margins

Cheng-Yo Tan, Chiao-An Yang, Shang‐Fu Chen +2 more

Computer Vision and Pattern Recognition · · cited 4× · DOI ↗ · more like this ·

Given a partial image input, image outpainting is to produce the desirable output by recovering or extending the surrounding image regions. While existing image outpainting methods achieve impressive results based on the recent advances of deep learning, they either lack the ability to extend image regions in arbitrary directions or require the filling image margins to be given in advance. To address this challenging task, we propose a unique deep learning framework for robust image outpainting, which consists of a margin prediction network and a teacher-student-based network for producing outpainted images. Our proposed model does not require image filling margins to be known beforehand, while both image appearance and perceptual feature consistencies can be jointly enforced. Our experiments quantitatively and qualitatively verify the effectiveness of our method, which is shown to…

Diffusion Model-Augmented Behavioral Cloning

Shang-Fu Chen, Hsiang-Chun Wang, Ming-Hao Hsu +2 more

cs.LG · · cited 3× · arXiv ↗ · more like this ·

Imitation learning addresses the challenge of learning by observing an expert's demonstrations without access to reward signals from environments. Most existing imitation learning methods that do not require interacting with environments either model the expert distribution as the conditional probability p(a|s) (e.g., behavioral cloning, BC) or the joint probability p(s, a). Despite the simplicity of modeling the conditional probability with BC, it usually struggles with generalization. While modeling the joint probability can improve generalization performance, the inference procedure is often time-consuming, and the model can suffer from manifold overfitting. This work proposes an imitation learning framework that benefits from modeling both the conditional and joint probability of the expert distribution. Our proposed Diffusion Model-Augmented Behavioral Cloning (DBC) employs a…

See More, Detect Less? Taming Information Leakage in Multi-View Anomaly Detection

Shang-Fu Chen, Kuan-Chuan Peng, Jhih-Ciang Wu +2 more

cs.CV · · cited 1× · arXiv ↗ · more like this ·

In multi-view anomaly detection, more cross-view information can actually hurt. When multiple inspection views are naively fused in a reconstruction-based pipeline, normal cues from intact views propagate to the decoder, which faithfully reconstructs anomalous regions, collapsing the reconstruction gap the detector depends on. We call this failure mode \emph{cross-view information leakage} and show that effective multi-view fusion must explicitly restrict the information reaching the decoder. Building on this insight, we present GLAD(Global-Local Attention Driven framework), the first framework combining vision foundation model features with local and global cross-view fusion for multi-view anomaly detection. The Multi-view Merging Attention (MMA) module performs local cross-view fusion at linear complexity with learnable view importance weighting and token-wise gating, letting each…

Representation and Boundary Enhancement for Action Segmentation Using Transformer

Shang‐Fu Chen, Cheng-Xun Wen, Wen-Huang Cheng +1 more

Computer Vision and Pattern Recognition · · cited 1× · DOI ↗ · more like this ·

In the task of action segmentation, the goal is to partition a lengthy, untrimmed video into a series of action segments. Recently, Transformer-based methods have outperformed the previous temporal convolutional networks (TCNs) in terms of overall performance. However, both TCNs and Transformers encounter the challenge of over-segmentation. Prior approaches often relied on post-processing techniques to address this issue, but these methods are not universally applicable to every model and may sometimes result in performance degradation. Therefore, in this paper, we propose a set of loss functions to enhance representation learning and employ a multi-task learning approach to strengthen the model’s ability to identify action boundaries. Through extensive experiments, we validate that our method demonstrates significant improvements, particularly in addressing the challenge of…

Representation Decomposition for Image Manipulation and Beyond

Shang-Fu Chen, Jia-Wei Yan, Ya-Fan Su +1 more

cs.CV · · cited 1× · arXiv ↗ · more like this ·

Representation disentanglement aims at learning interpretable features, so that the output can be recovered or manipulated accordingly. While existing works like infoGAN and AC-GAN exist, they choose to derive disjoint attribute code for feature disentanglement, which is not applicable for existing/trained generative models. In this paper, we propose a decomposition-GAN (dec-GAN), which is able to achieve the decomposition of an existing latent representation into content and attribute features. Guided by the classifier pre-trained on the attributes of interest, our dec-GAN decomposes the attributes of interest from the latent representation, while data recovery and feature consistency objectives enforce the learning of our proposed method. Our experiments on multiple image datasets confirm the effectiveness and robustness of our dec-GAN over recent representation disentanglement models.

HAMMER: Harmonic-Aware Parallel Context Modeling and Discriminator-Free Perceptual Optimization for Speech Enhancement

Shang-Fu Chen, Szu-Wei Fu, Sung-Feng Huang +3 more

eess.AS · · cited 0× · arXiv ↗ · more like this ·

Recent speech enhancement systems combine self-attention and Mamba to capture global interactions and long-range dependencies. Yet these hybrids usually operate as sequence mixers and do not explicitly exploit harmonic periodicity, a strong cue for preserving voiced speech under noise. Perceptual optimization poses another challenge. PESQ is non-differentiable, so many methods train auxiliary metric discriminators that increase complexity and introduce adversarial instability. We propose \ours, a harmonic-aware and discriminator-free speech enhancer built around two components. (i) The Time-Frequency Harmonic-aware Attention-Mamba (TF-HAM) block runs self-attention and bidirectional Mamba in parallel along both spectrogram axes, then applies a speech-adapted autocorrelation feed-forward network to encode local periodic structure. (ii) Metric-explicit perceptual refinement (MEPR)…

GeoMAD: Geometry-Aware Multi-View Anomaly Detection via Deformable Fusion and Distributional Alignment

Shang-Fu Chen, Jhih-Ciang Wu, Kuan-Chuan Peng +2 more

cs.CV · · cited 0× · arXiv ↗ · more like this ·

Multi-view anomaly detection (MvAD) detects defects by exploiting complementary observations from multiple camera viewpoints. The central challenge is to fuse views with sufficient geometric awareness while remaining scalable to multi-class industrial settings. Existing methods typically fall into two extremes: voxel-based fusion provides explicit geometric alignment but requires costly 3D construction and class-specific assumptions, whereas lightweight patch-based fusion is efficient but relies on discrete candidate matching and lacks continuous cross-view correspondence. In this paper, we propose GeoMAD, a unified multi-view, multi-class AD framework that addresses both geometric correspondence deficiency and distributional inconsistency. Our \textit{Cross-view Deformable Fusion Module} (CDFM) learns content-adaptive, view-pair-specific sampling offsets directly on 2D feature maps and…

A novel simulation tool for low-coverage whole-genome sequencing using multivariate Gaussian mixture models

Kaiyu Chen, Shang‐Fu Chen, Raquel Dias +1 more

Artificial Intelligence · BMC Bioinformatics · · cited 0× · DOI ↗ · more like this ·

Low-coverage whole-genome sequencing (lcWGS) has emerged as a cost-effective and robust approach for population genomic studies. Despite its advantages, publicly available resources for large-scale lcWGS datasets remain limited. To our knowledge, there is yet a bioinformatics tool capable of directly simulating lcWGS datasets from variant call format (VCF) files. To address this gap, we developed a tool called lcSimVCF, which leverages multivariate Gaussian mixture models (MGMMs) to simulate lcWGS genotype likelihood distributions directly from high-coverage whole-genome sequencing (hcWGS) VCF files. In this study, we introduced a tool called lcSimVCF that aim to use 30× hcWGS VCF files to simulate 1× genotype likelihood distributions as a demonstration. We trained an MGMM framework for three genotypes: homozygous reference, heterozygous, and homozygous non-reference, each simulating…

Restoring Noisy Demonstration for Imitation Learning With Diffusion Models

Shang-Fu Chen, Co Yong, Shao-Hua Sun

cs.RO · · cited 0× · arXiv ↗ · more like this ·

Imitation learning (IL) aims to learn a policy from expert demonstrations and has been applied to various applications. By learning from the expert policy, IL methods do not require environmental interactions or reward signals. However, most existing imitation learning algorithms assume perfect expert demonstrations, but expert demonstrations often contain imperfections caused by errors from human experts or sensor/control system inaccuracies. To address the above problems, this work proposes a filter-and-restore framework to best leverage expert demonstrations with inherent noise. Our proposed method first filters clean samples from the demonstrations and then learns conditional diffusion models to recover the noisy ones. We evaluate our proposed framework and existing methods in various domains, including robot arm manipulation, dexterous manipulation, and locomotion. The experiment…

A Data Hiding Scheme Based On Absolute Moment Block Truncation Coding and Lookup Table

Ting-Kai Yang, Shang‐Fu Chen, Tai-Ming Huang +3 more

Computer Vision and Pattern Recognition · 2022 IEEE International Conference on Consumer Electronics - Taiwan · · cited 0× · DOI ↗ · more like this ·

This research proposes a data hiding scheme which uses absolute moment block truncation coding (AMBTC) for image compression, and generates a reference lookup table which has a one-to-many relationship between the hidden data and the candidate bitmaps of the compressed image. When embedding data, the candidate blocks that match the hidden data are filtered through mean square error (MSE) function to select the optimal block. From the preliminary experiments, we prove that the proposed method is superior to other data hiding method in terms of data hiding ability and image visual quality.

DirectDrag: High-Fidelity, Mask-Free, Prompt-Free Drag-based Image Editing via Readout-Guided Feature Alignment

Sheng-Hao Liao, Shang-Fu Chen, Tai-Ming Huang +2 more

cs.CV · · arXiv ↗ · more like this ·

Drag-based image editing using generative models provides intuitive control over image structures. However, existing methods rely heavily on manually provided masks and textual prompts to preserve semantic fidelity and motion precision. Removing these constraints creates a fundamental trade-off: visual artifacts without masks and poor spatial control without prompts. To address these limitations, we propose DirectDrag, a novel mask- and prompt-free editing framework. DirectDrag enables precise and efficient manipulation with minimal user input while maintaining high image fidelity and accurate point alignment. DirectDrag introduces two key innovations. First, we design an Auto Soft Mask Generation module that intelligently infers editable regions from point displacement, automatically localizing deformation along movement paths while preserving contextual integrity through the…

HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning

Ayano Hiranaka, Shang-Fu Chen, Chieh-Hsin Lai +6 more

cs.LG · · arXiv ↗ · more like this ·

Controllable generation through Stable Diffusion (SD) fine-tuning aims to improve fidelity, safety, and alignment with human guidance. Existing reinforcement learning from human feedback methods usually rely on predefined heuristic reward functions or pretrained reward models built on large-scale datasets, limiting their applicability to scenarios where collecting such data is costly or difficult. To effectively and efficiently utilize human feedback, we develop a framework, HERO, which leverages online human feedback collected on the fly during model learning. Specifically, HERO features two key mechanisms: (1) Feedback-Aligned Representation Learning, an online training method that captures human feedback and provides informative learning signals for fine-tuning, and (2) Feedback-Guided Image Generation, which involves generating images from SD's refined initialization samples…