arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Ning Yu

295 papers · 1973–2026

1973: 11974: 01975: 01976: 01977: 01978: 01979: 01980: 01981: 01982: 01983: 01984: 01985: 01986: 01987: 01988: 01989: 01990: 01991: 01992: 01993: 01994: 01995: 01996: 11997: 01998: 01999: 12000: 02001: 02002: 22003: 12004: 42005: 52006: 82007: 122008: 92009: 52010: 82011: 112012: 92013: 162014: 72015: 42016: 132017: 52018: 102019: 62020: 182021: 162022: 252023: 262024: 352025: 252026: 12
1973peak 35 in 20242026

Frequent co-authors

Papers

showing 1–25 of 295 RSS

Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data

Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi +1 more

cs.CR · · cited 331× · arXiv ↗ · more like this ·

Photorealistic image generation has reached a new level of quality due to the breakthroughs of generative adversarial networks (GANs). Yet, the dark side of such deepfakes, the malicious use of generated media, raises concerns about visual misinformation. While existing research work on deepfake detection demonstrates high accuracy, it is subject to advances in generation techniques and adversarial iterations on detection countermeasure techniques. Thus, we seek a proactive and sustainable solution on deepfake detection, that is agnostic to the evolution of generative models, by introducing artificial fingerprints into the models. Our approach is simple and effective. We first embed artificial fingerprints into training data, then validate a surprising discovery on the transferability of such fingerprints from training data to generative models, which in turn appears in the generated…

DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models

Zeyang Sha, Zheng Li, Ning Yu +1 more

cs.CR · · cited 287× · arXiv ↗ · more like this ·

Text-to-image generation models that generate images based on prompt descriptions have attracted an increasing amount of attention during the past few months. Despite their encouraging performance, these models raise concerns about the misuse of their generated fake images. To tackle this problem, we pioneer a systematic study on the detection and attribution of fake images generated by text-to-image generation models. Concretely, we first build a machine learning classifier to detect the fake images generated by various text-to-image generation models. We then attribute these fake images to their source models, such that model owners can be held responsible for their models' misuse. We further investigate how prompts that generate fake images affect detection and attribution. We conduct extensive experiments on four popular text-to-image generation models, including DALL$\cdot$E 2…

ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding

Le Xue, Ning Yu, Shu Zhang +8 more

cs.CV · · cited 283× · arXiv ↗ · more like this ·

Recent advancements in multimodal pre-training have shown promising efficacy in 3D representation learning by aligning multimodal features across 3D shapes, their 2D counterparts, and language descriptions. However, the methods used by existing frameworks to curate such multimodal data, in particular language descriptions for 3D shapes, are not scalable, and the collected language descriptions are not diverse. To address this, we introduce ULIP-2, a simple yet effective tri-modal pre-training framework that leverages large multimodal models to automatically generate holistic language descriptions for 3D shapes. It only needs 3D data as input, eliminating the need for any manual 3D annotations, and is therefore scalable to large datasets. ULIP-2 is also equipped with scaled-up backbones for better multimodal representation learning. We conduct experiments on two large-scale 3D datasets…

UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild

Can Qin, Shu Zhang, Ning Yu +10 more

cs.CV · · cited 247× · arXiv ↗ · more like this ·

Achieving machine autonomy and human control often represent divergent objectives in the design of interactive AI systems. Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages. However, they often fall short in generating images with spatial, structural, or geometric controls. The integration of such controls, which can accommodate various visual conditions in a single unified model, remains an unaddressed challenge. In response, we introduce UniControl, a new generative foundation model that consolidates a wide array of controllable condition-to-image (C2I) tasks within a singular framework, while still allowing for arbitrary language prompts. UniControl enables pixel-level-precise image generation, where visual conditions primarily influence the generated structures and language prompts…

HIVE: Harnessing Human Feedback for Instructional Visual Editing

Shu Zhang, Xinyi Yang, Yihao Feng +9 more

cs.CV · · cited 200× · arXiv ↗ · more like this ·

Incorporating human feedback has been shown to be crucial to align text generated by large language models to human preferences. We hypothesize that state-of-the-art instructional image editing models, where outputs are generated based on an input image and an editing instruction, could similarly benefit from human feedback, as their outputs may not adhere to the correct instructions and preferences of users. In this paper, we present a novel framework to harness human feedback for instructional visual editing (HIVE). Specifically, we collect human feedback on the edited images and learn a reward function to capture the underlying user preferences. We then introduce scalable diffusion model fine-tuning methods that can incorporate human preferences based on the estimated reward. Besides, to mitigate the bias brought by the limitation of data, we contribute a new 1M training dataset, a…

Applying big data based deep learning system to intrusion detection

Wei Zhong, Ning Yu, Chunyu Ai

Computer Networks and Communications · Big Data Mining and Analytics · · cited 161× · DOI ↗ · more like this ·

With vast amounts of data being generated daily and the ever increasing interconnectivity of the world's internet infrastructures, a machine learning based Intrusion Detection Systems (IDS) has become a vital component to protect our economic and national security. Previous shallow learning and deep learning strategies adopt the single learning model approach for intrusion detection. The single learning model approach may experience problems to understand increasingly complicated data distribution of intrusion patterns. Particularly, the single deep learning model may not be effective to capture unique patterns from intrusive attacks having a small number of samples. In order to further enhance the performance of machine learning based IDS, we propose the Big Data based Hierarchical Deep Learning System (BDHDLS). BDHDLS utilizes behavioral features and content features to understand…

Responsible Disclosure of Generative Models Using Scalable Fingerprinting

Ning Yu, Vladislav Skripniuk, Dingfan Chen +2 more

cs.CR · · cited 125× · arXiv ↗ · more like this ·

Over the past years, deep generative models have achieved a new level of performance. Generated data has become difficult, if not impossible, to be distinguished from real data. While there are plenty of use cases that benefit from this technology, there are also strong concerns on how this new technology can be misused to generate deep fakes and enable misinformation at scale. Unfortunately, current deep fake detection methods are not sustainable, as the gap between real and fake continues to close. In contrast, our work enables a responsible disclosure of such state-of-the-art generative models, that allows model inventors to fingerprint their models, so that the generated samples containing a fingerprint can be accurately detected and attributed to a source. Our technique achieves this by an efficient and scalable ad-hoc generation of a large population of models with distinct…

RoSteALS: Robust Steganography using Autoencoder Latent Space

Tu Bui, Shruti Agarwal, Ning Yu +1 more

cs.CV · · cited 117× · arXiv ↗ · more like this ·

Data hiding such as steganography and invisible watermarking has important applications in copyright protection, privacy-preserved communication and content provenance. Existing works often fall short in either preserving image quality, or robustness against perturbations or are too complex to train. We propose RoSteALS, a practical steganography technique leveraging frozen pretrained autoencoders to free the payload embedding from learning the distribution of cover images. RoSteALS has a light-weight secret encoder of just 300k parameters, is easy to train, has perfect secret recovery performance and comparable image quality on three benchmarks. Additionally, RoSteALS can be adapted for novel cover-less steganography applications in which the cover image can be sampled from noise or conditioned on text prompts via a denoising diffusion process. Our model and code are available at…

Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis

Yang He, Ning Yu, Margret Keuper +1 more

cs.CV · · cited 101× · arXiv ↗ · more like this ·

The rapid advances in deep generative models over the past years have led to highly {realistic media, known as deepfakes,} that are commonly indistinguishable from real to human eyes. These advances make assessing the authenticity of visual data increasingly difficult and pose a misinformation threat to the trustworthiness of visual content in general. Although recent work has shown strong detection accuracy of such deepfakes, the success largely relies on identifying frequency artifacts in the generated images, which will not yield a sustainable detection approach as generative models continue evolving and closing the gap to real images. In order to overcome this issue, we propose a novel fake detection that is designed to re-synthesize testing images and extract visual cues for detection. The re-synthesis procedure is flexible, allowing us to incorporate a series of visual tasks - we…

Membership Inference Attacks Against Text-to-image Generation Models

Yixin Wu, Ning Yu, Zheng Li +2 more

cs.CR · · cited 89× · arXiv ↗ · more like this ·

Text-to-image generation models have recently attracted unprecedented attention as they unlatch imaginative applications in all areas of life. However, developing such models requires huge amounts of data that might contain privacy-sensitive information, e.g., face identity. While privacy risks have been extensively demonstrated in the image classification and GAN generation domains, privacy risks in the text-to-image generation domain are largely unexplored. In this paper, we perform the first privacy analysis of text-to-image generation models through the lens of membership inference. Specifically, we propose three key intuitions about membership information and design four attack methodologies accordingly. We conduct comprehensive evaluations on two mainstream text-to-image generation models including sequence-to-sequence modeling and diffusion-based modeling. The empirical results…

Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand

Jitesh Jain, Yuqian Zhou, Ning Yu +1 more

cs.CV · · cited 74× · arXiv ↗ · more like this ·

Deep image inpainting has made impressive progress with recent advances in image generation and processing algorithms. We claim that the performance of inpainting algorithms can be better judged by the generated structures and textures. Structures refer to the generated object boundary or novel geometric structures within the hole, while texture refers to high-frequency details, especially man-made repeating patterns filled inside the structural regions. We believe that better structures are usually obtained from a coarse-to-fine GAN-based generator network while repeating patterns nowadays can be better modeled using state-of-the-art high-frequency fast fourier convolutional layers. In this paper, we propose a novel inpainting network combining the advantages of the two designs. Therefore, our model achieves a remarkable visual quality to match state-of-the-art performance in both…

RelaxLoss: Defending Membership Inference Attacks without Losing Utility

Dingfan Chen, Ning Yu, Mario Fritz

cs.LG · · cited 65× · arXiv ↗ · more like this ·

As a long-term threat to the privacy of training data, membership inference attacks (MIAs) emerge ubiquitously in machine learning models. Existing works evidence strong connection between the distinguishability of the training and testing loss distributions and the model's vulnerability to MIAs. Motivated by existing results, we propose a novel training framework based on a relaxed loss with a more achievable learning target, which leads to narrowed generalization gap and reduced privacy leakage. RelaxLoss is applicable to any classification model with added benefits of easy implementation and negligible overhead. Through extensive evaluations on five datasets with diverse modalities (images, medical data, transaction records), our approach consistently outperforms state-of-the-art defense mechanisms in terms of resilience against MIAs as well as model utility. Our defense is the first…

Development of an automatic parking system for vehicle

Tsung-Hua Hsu, Jingfu Liu, Pen-Ning Yu +2 more

Computer Vision and Pattern Recognition · · cited 61× · DOI ↗ · more like this ·

This research proposes an automatic parking system (APS) with a good maneuver performance for vehicle. The proposed configuration of the APS includes sensors information fusion, position estimation, path planning, and tracking algorithm. The on-vehicle verification demonstrated by means of the control of an electric power steering system for tracking the planned path. Parking space is measured up as the vehicle drives past with the aid of an ultrasonic sensor. And drivers can use a user interactive interface to select the parking space if it is big enough. Then steering movements take place on the basis of the pure pursuit tracking algorithm and the inertial navigation method. Simulation and experiment results show good performance in the verification of our automatic parking system under reverse parking and parallel parking cases, and the parking space requires just 1.5 times longer…

Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture

Ning Yu, Connelly Barnes, Eli Shechtman +2 more

cs.CV · · cited 55× · arXiv ↗ · more like this ·

This paper addresses the problem of interpolating visual textures. We formulate this problem by requiring (1) by-example controllability and (2) realistic and smooth interpolation among an arbitrary number of texture samples. To solve it we propose a neural network trained simultaneously on a reconstruction task and a generation task, which can project texture examples onto a latent space where they can be linearly interpolated and projected back onto the image domain, thus ensuring both intuitive control and realistic results. We show our method outperforms a number of baselines according to a comprehensive suite of metrics as well as a user study. We further show several applications based on our technique, which include texture brush, texture dissolve, and animal hybridization.

GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models

Dingfan Chen, Ning Yu, Yang Zhang +1 more

cs.LG · · cited 53× · arXiv ↗ · more like this ·

Deep learning has achieved overwhelming success, spanning from discriminative models to generative models. In particular, deep generative models have facilitated a new level of performance in a myriad of areas, ranging from media manipulation to sanitized dataset generation. Despite the great success, the potential risks of privacy breach caused by generative models have not been analyzed systematically. In this paper, we focus on membership inference attack against deep generative models that reveals information about the training data used for victim models. Specifically, we present the first taxonomy of membership inference attacks, encompassing not only existing attacks but also our novel ones. In addition, we propose the first generic attack model that can be instantiated in a large range of settings and is applicable to various kinds of deep generative models. Moreover, we provide…

DifFRelight: Diffusion-Based Facial Performance Relighting

Mingming He, Pascal Clausen, Ahmet Levent Taşel +8 more

cs.CV · · cited 50× · arXiv ↗ · more like this ·

We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. Leveraging a subject-specific dataset containing diverse facial expressions captured under various lighting conditions, including flat-lit and one-light-at-a-time (OLAT) scenarios, we train a diffusion model for precise lighting control, enabling high-fidelity relit facial images from flat-lit inputs. Our framework includes spatially-aligned conditioning of flat-lit captures and random noise, along with integrated lighting information for global control, utilizing prior knowledge from the pre-trained Stable Diffusion model. This model is then applied to dynamic facial performances captured in a consistent flat-lit environment and reconstructed for novel-view synthesis using a scalable dynamic 3D Gaussian Splatting method to maintain quality and consistency in…

Can't Steal? Cont-Steal! Contrastive Stealing Attacks Against Image Encoders

Zeyang Sha, Xinlei He, Ning Yu +2 more

cs.CR · · cited 50× · arXiv ↗ · more like this ·

Self-supervised representation learning techniques have been developing rapidly to make full use of unlabeled images. They encode images into rich features that are oblivious to downstream tasks. Behind their revolutionary representation power, the requirements for dedicated model designs and a massive amount of computation resources expose image encoders to the risks of potential model stealing attacks - a cheap way to mimic the well-trained encoder performance while circumventing the demanding requirements. Yet conventional attacks only target supervised classifiers given their predicted labels and/or posteriors, which leaves the vulnerability of unsupervised encoders unexplored. In this paper, we first instantiate the conventional stealing attacks against encoders and demonstrate their severer vulnerability compared with downstream classifiers. To better leverage the rich…

Auditing Membership Leakages of Multi-Exit Networks

Zheng Li, Yiyong Liu, Xinlei He +3 more

cs.CR · · cited 49× · arXiv ↗ · more like this ·

Relying on the fact that not all inputs require the same amount of computation to yield a confident prediction, multi-exit networks are gaining attention as a prominent approach for pushing the limits of efficient deployment. Multi-exit networks endow a backbone model with early exits, allowing to obtain predictions at intermediate layers of the model and thus save computation time and/or energy. However, current various designs of multi-exit networks are only considered to achieve the best trade-off between resource usage efficiency and prediction accuracy, the privacy risks stemming from them have never been explored. This prompts the need for a comprehensive investigation of privacy risks in multi-exit networks. In this paper, we perform the first privacy analysis of multi-exit networks through the lens of membership leakages. In particular, we first leverage the existing attack…

Fast Query Point Movement Techniques for Large CBIR Systems

Danzhou Liu, K.A. Hua, Khanh Vu +1 more

Computer Vision and Pattern Recognition · IEEE Transactions on Knowledge and Data Engineering · · cited 49× · DOI ↗ · more like this ·

Target search in content-based image retrieval (CBIR) systems refers to finding a specific (target) image such as a particular registered logo or a specific historical photograph. Existing techniques, designed around query refinement based on relevance feedback, suffer from slow convergence, and do not guarantee to find intended targets. To address these limitations, we propose several efficient query point movement methods. We prove that our approach is able to reach any given target image with fewer iterations in the worst and average cases. We propose a new index structure and query processing technique to improve retrieval effectiveness and efficiency. We also consider strategies to minimize the effects of users' inaccurate relevance feedback. Extensive experiments in simulated and realistic environments show that our approach significantly reduces the number of required iterations…

A Credible Bayesian-Based Trust Management Scheme for Wireless Sensor Networks

Renjian Feng, Xiaona Han, Qiang Liu +1 more

Computer Networks and Communications · International Journal of Distributed Sensor Networks · · cited 46× · DOI ↗ · more like this ·

With the rapid development of wireless sensor networks (WSNs), the security of WSNs is an important issue in this field because of their vulnerabilities to attacks. Designing a reasonable trust management scheme that can evaluate the trust relationships among sensor nodes accurately is a challenging but meaningful task. In this paper, a credible Bayesian-based trust management scheme (BTMS) is proposed. The overall trust value is aggregated by both direct and indirect trust information. The former is calculated by a modified Bayesian equation and updated by a sliding window. The latter is computed by recommendations from a third party. Moreover, the indirect trust computation is invoked conditionally according to the uncertainty of direct trust calculated via Entropy Theory and malicious feedbacks are excluded. Meanwhile, different recommendations are appropriately weighted in light of…

A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports

Madhumita Sushil, Travis Zack, Divneet Mandair +6 more

Artificial Intelligence · Journal of the American Medical Informatics Association · · cited 41× · DOI ↗ · more like this ·

OBJECTIVE: Although supervised machine learning is popular for information extraction from clinical notes, creating large annotated datasets requires extensive domain expertise and is time-consuming. Meanwhile, large language models (LLMs) have demonstrated promising transfer learning capability. In this study, we explored whether recent LLMs could reduce the need for large-scale data annotations. MATERIALS AND METHODS: We curated a dataset of 769 breast cancer pathology reports, manually labeled with 12 categories, to compare zero-shot classification capability of the following LLMs: GPT-4, GPT-3.5, Starling, and ClinicalCamel, with task-specific supervised classification performance of 3 models: random forests, long short-term memory networks with attention (LSTM-Att), and the UCSF-BERT model. RESULTS: Across all 12 tasks, the GPT-4 model performed either significantly better than or…

Trust Management Scheme Based on D-S Evidence Theory for Wireless Sensor Networks

Renjian Feng, Shenyun Che, Xiao Wang +1 more

Computer Networks and Communications · International Journal of Distributed Sensor Networks · · cited 40× · DOI ↗ · more like this ·

Trust management scheme has been regarded as a powerful tool to defend against the wide set of security attacks and identify malicious nodes. In this paper, we propose a trust management scheme based on revised Dempster-Shafer (D-S) evidence theory. D-S theory is preponderant in tackling both random and subjective uncertainty in the trust mechanism. A trust propagation mechanism including conditional trust transitivity and dynamic recommendation aggregation is developed for obtaining the recommended trust values from third part nodes. We adopt a flexible synthesis method that uses recommended trust only when no direct trust exists to keep a good trust-energy consumption balance. We also consider on-off attack and bad mouthing attack in our simulation. The simulation results and analysis show that the proposed method has excellent ability to deal with typical network attacks, better…

Cross‐cultural comparisons between the earthquake preparedness models of Taiwan and New Zealand

Li-ju Jang, Jieh‐Jiuh Wang, Douglas Paton +1 more

Sociology and Political Science · Disasters · · cited 38× · DOI ↗ · more like this ·

Taiwan and New Zealand are both located in the Pacific Rim where 81 per cent of the world's largest earthquakes occur. Effective programmes for increasing people's preparedness for these hazards are essential. This paper tests the applicability of the community engagement theory of hazard preparedness in two distinct cultural contexts. Structural equation modelling analysis provides support for this theory. The paper suggests that the close fit between theory and data that is achieved by excluding trust supports the theoretical prediction that familiarity with a hazard negates the need to trust external sources. The results demonstrate that the hazard preparedness theory is applicable to communities that have previously experienced earthquakes and are therefore familiar with the associated hazards and the need for earthquake preparedness. The paper also argues that cross-cultural…

Query Decomposition: A Multiple Neighborhood Approach to Relevance Feedback Processing in Content-based Image Retrieval

K.A. Hua, Ning Yu, Danzhou Liu

Computer Vision and Pattern Recognition · Journal of International Crisis and Risk Communication Research · · cited 36× · DOI ↗ · more like this ·

Today’s Content-Based Image Retrieval (CBIR) techniques are based on the "k-nearest neighbors" (k- NN) model. They retrieve images from a single neighborhood using low-level visual features. In this model, semantically similar images are assumed to be clustered in the high-dimensional feature space. Unfortunately, no visual-based feature vector is sufficient to facilitate perfect semantic clustering; and semantically similar images with different appearances are always clustered into distinct neighborhoods in the feature space. Confinement of the search results to a single neighborhood is an inherent limitation of the k-NN techniques. In this paper we consider a new image retrieval paradigm — the Query Decomposition model - that facilitates retrieval of semantically similar images from multiple neighborhoods in the feature space. The retrieval results are the k most similar images from…

GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation

Can Qin, Ning Yu, Chen Xing +6 more

cs.CV · · cited 32× · arXiv ↗ · more like this ·

Text-to-image (T2I) models based on diffusion processes have achieved remarkable success in controllable image generation using user-provided captions. However, the tight coupling between the current text encoder and image decoder in T2I models makes it challenging to replace or upgrade. Such changes often require massive fine-tuning or even training from scratch with the prohibitive expense. To address this problem, we propose GlueGen, which applies a newly proposed GlueNet model to align features from single-modal or multi-modal encoders with the latent space of an existing T2I model. The approach introduces a new training objective that leverages parallel corpora to align the representation spaces of different encoders. Empirical results show that GlueNet can be trained efficiently and enables various capabilities beyond previous state-of-the-art models: 1) multilingual language…