Author

Zhou Yu

946 papers · 1989–2026

1989peak 60 in 20212026

Frequent co-authors

Papers

showing 1–25 of 946 RSS

Deep Modular Co-Attention Networks for Visual Question Answering

Zhou Yu, Jun Yu, Yuhao Cui +2 more

cs.CV · 25 Jun 2019 · cited 1,020× · arXiv ↗ · more like this ·

Visual Question Answering (VQA) requires a fine-grained and simultaneous understanding of both the visual content of images and the textual content of questions. Therefore, designing an effective `co-attention' model to associate key words in questions with key objects in images is central to VQA performance. So far, most successful attempts at co-attention learning have been achieved by using shallow models, and deep co-attention models show little improvement over their shallow counterparts. In this paper, we propose a deep Modular Co-Attention Network (MCAN) that consists of Modular Co-Attention (MCA) layers cascaded in depth. Each MCA layer models the self-attention of questions and images, as well as the guided-attention of images jointly using a modular composition of two basic attention units. We quantitatively and qualitatively evaluate MCAN on the benchmark VQA-v2 dataset and…

ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering

Zhou Yu, Dejing Xu, Jun Yu +4 more

cs.CV · 6 Jun 2019 · cited 788× · arXiv ↗ · more like this ·

Recent developments in modeling language and vision have been successfully applied to image question answering. It is both crucial and natural to extend this research direction to the video domain for video question answering (VideoQA). Compared to the image domain where large scale and fully annotated benchmark datasets exists, VideoQA datasets are limited to small scale and are automatically generated, etc. These limitations restrict their applicability in practice. Here we introduce ActivityNet-QA, a fully annotated and large scale VideoQA dataset. The dataset consists of 58,000 QA pairs on 5,800 complex web videos derived from the popular ActivityNet dataset. We present a statistical analysis of our ActivityNet-QA dataset and conduct extensive experiments on it by comparing existing VideoQA baselines. Moreover, we explore various video representation strategies to improve VideoQA…

Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering

Zhou Yu, Jun Yu, Jianping Fan +1 more

cs.CV · 4 Aug 2017 · cited 755× · arXiv ↗ · more like this ·

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both the visual content of images and the textual content of questions. The approaches used to represent the images and questions in a fine-grained manner and questions and to fuse these multi-modal features play key roles in performance. Bilinear pooling based models have been shown to outperform traditional linear models for VQA, but their high-dimensional representations and high computational complexity may seriously limit their applicability in practice. For multi-modal feature fusion, here we develop a Multi-modal Factorized Bilinear (MFB) pooling approach to efficiently and effectively combine multi-modal features, which results in superior performance for VQA compared with other bilinear pooling approaches. For fine-grained image and question representation, we develop a…

Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback

Baolin Peng, Michel Galley, Pengcheng He +8 more

cs.CL · February 2023 · cited 549× · arXiv ↗ · more like this ·

Large language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e.g., task-oriented dialog and question answering. However, applying LLMs to real-world, mission-critical applications remains challenging mainly due to their tendency to generate hallucinations and their inability to use external knowledge. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules. Our system makes the LLM generate responses grounded in external knowledge, e.g., stored in task-specific databases. It also iteratively revises LLM prompts to improve model responses using feedback generated by utility functions, e.g., the factuality score of a LLM-generated response. The effectiveness of LLM-Augmenter is empirically validated on two types of scenarios, task-oriented dialog and open-domain question…

Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering

Zhou Yu, Jun Yu, Chenchao Xiang +2 more

cs.CV · August 2017 · cited 527× · arXiv ↗ · more like this ·

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multi-modal feature fusion that is able to capture the complex interactions between multi-modal features; 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a `co-attention' mechanism is developed by using a deep neural network architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and…

Towards Emotional Support Dialog Systems

Siyang Liu, Chujie Zheng, Orianna Demasi +5 more

cs.CL · 2 Jun 2021 · cited 526× · arXiv ↗ · more like this ·

Emotional support is a crucial ability for many conversation scenarios, including social interactions, mental health support, and customer service chats. Following reasonable procedures and using various support skills can help to effectively provide support. However, due to the lack of a well-designed task and corpora of effective emotional support conversations, research on building emotional support into dialog systems remains untouched. In this paper, we define the Emotional Support Conversation (ESC) task and propose an ESC Framework, which is grounded on the Helping Skills Theory. We construct an Emotion Support Conversation dataset (ESConv) with rich annotation (especially support strategy) in a help-seeker and supporter mode. To ensure a corpus of high-quality conversations that provide examples of effective emotional support, we take extensive effort to design training…

Multimodal Transformer with Multi-View Visual Representation for Image Captioning

Jun Yu, Jing Li, Zhou Yu +1 more

cs.CV · 20 May 2019 · cited 454× · arXiv ↗ · more like this ·

Image captioning aims to automatically generate a natural language description of a given image, and most state-of-the-art models have adopted an encoder-decoder framework. The framework consists of a convolution neural network (CNN)-based image encoder that extracts region-based visual features from the input image, and an recurrent neural network (RNN)-based caption decoder that generates the output caption words based on the visual features with the attention mechanism. Despite the success of existing studies, current methods only model the co-attention that characterizes the inter-modal interactions while neglecting the self-attention that characterizes the intra-modal interactions. Inspired by the success of the Transformer model in machine translation, here we extend it to a Multimodal Transformer (MT) model for image captioning. Compared to existing image captioning approaches…

Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good

Xuewei Wang, Weiyan Shi, Richard Kim +4 more

cs.CL · June 2019 · cited 391× · arXiv ↗ · more like this ·

Developing intelligent persuasive conversational agents to change people's opinions and actions for social good is the frontier in advancing the ethical development of automated dialogue systems. To do so, the first step is to understand the intricate organization of strategic disclosures and appeals employed in human persuasion conversations. We designed an online persuasion task where one participant was asked to persuade the other to donate to a specific charity. We collected a large dataset with 1,017 dialogues and annotated emerging persuasion strategies from a subset. Based on the annotation, we built a baseline classifier with context information and sentence-level features to predict the 10 persuasion strategies used in the corpus. Furthermore, to develop an understanding of personalized persuasion processes, we analyzed the relationships between individuals' demographic and…

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

Pan Lu, Liang Qiu, Jiaqi Chen +6 more

cs.CV · October 2021 · cited 327× · arXiv ↗ · more like this ·

Current visual question answering (VQA) tasks mainly consider answering human-annotated questions for natural images. However, aside from natural images, abstract diagrams with semantic richness are still understudied in visual understanding and reasoning research. In this work, we introduce a new challenge of Icon Question Answering (IconQA) with the goal of answering a question in an icon image context. We release IconQA, a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. The IconQA dataset is inspired by real-world diagram word problems that highlight the importance of abstract diagram understanding and comprehensive cognitive reasoning. Thus, IconQA requires not only perception skills like object recognition and text understanding, but also diverse cognitive reasoning skills, such as…

INSPIRED: Toward Sociable Recommendation Dialog Systems

Shirley Anugrah Hayati, Dongyeop Kang, Qingxiaoyang Zhu +2 more

cs.CL · September 2020 · cited 216× · arXiv ↗ · more like this ·

In recommendation dialogs, humans commonly disclose their preference and make recommendations in a friendly manner. However, this is a challenge when developing a sociable recommendation dialog system, due to the lack of dialog dataset annotated with such sociable strategies. Therefore, we present INSPIRED, a new dataset of 1,001 human-human dialogs for movie recommendation with measures for successful recommendations. To better understand how humans make recommendations in communication, we design an annotation scheme related to recommendation strategies based on social science theories and annotate these dialogs. Our analysis shows that sociable recommendation strategies, such as sharing personal opinions or communicating with encouragement, more frequently lead to successful recommendations. Based on our dataset, we train end-to-end recommendation dialog systems with and without our…

Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context

Yichi Zhang, Zhijian Ou, Zhou Yu

cs.CL · November 2019 · cited 201× · arXiv ↗ · more like this ·

Conversations have an intrinsic one-to-many property, which means that multiple responses can be appropriate for the same dialog context. In task-oriented dialogs, this property leads to different valid dialog policies towards task completion. However, none of the existing task-oriented dialog generation approaches takes this property into account. We propose a Multi-Action Data Augmentation (MADA) framework to utilize the one-to-many property to generate diverse appropriate dialog responses. Specifically, we first use dialog states to summarize the dialog history, and then discover all possible mappings from every dialog state to its different valid system actions. During dialog system training, we enable the current dialog state to map to all valid system actions discovered in the previous process to create additional state-action pairs. By incorporating these additional pairs, the…

CARBON STORAGE AND BUDGET OF MAJOR CHINESE FOREST TYPES

Zhou Yu, Zhen Yu

Education · ACTA PHYTOECOLOGICA SINICA · 2000 · cited 173× · OpenAlex ↗ · more like this ·

The regional characteristics of carbon storage and carbon dioxide fluxes of major Chinese forest ecosystems were studied from the points of internal biological cycle, based on published data regarding forest biomass, productivity, the organic carbon content of soil profile, stand and annual weight of the litter, soil respiration etc. The results are as follows: the average carbon density of Chinese forest ecosystem is 258. 83 t .hm-2, showing a generally increasing trend with increasing latitude; carbon density of the vegetation, soil and litter is 57. 07 t ·hm-2, 193. 55 t .hm-2, and 8. 21 t.hm-2, respectively ; the characteristics of the carbon density of these three fractions (vegetation, soil, lit-ter) were also analyzed ; from the recent areal data provided by the Ministry of Forestry of China in 1989-1993 the total carbon storage of Chinese forests was estimated to be 281. 16 X…

Diffusion-Based Motion Planning for a Nonholonomic Flexible Needle Model

Wooram Park, Jin Seob Kim, Zhou Yu +3 more

Modeling and Simulation · 18 Jan 2006 · cited 166× · DOI ↗ · more like this ·

Fine needles facilitate diagnosis and therapy because they enable minimally invasive surgical interventions. This paper formulates the problem of steering a very flexible needle through firm tissue as a nonholonomic kinematics problem, and demonstrates how planning can be accomplished using diffusion-based motion planning on the Euclidean group, SE(3). In the present formulation, the tissue is treated as isotropic and no obstacles are present. The bevel tip of the needle is treated as a nonholonomic constraint that can be viewed as a 3D extension of the standard kinematic cart or unicycle. A deterministic model is used as the starting point, and reachability criteria are established. A stochastic differential equation and its corresponding Fokker-Planck equation are derived. The Euler-Maruyama method is used to generate the ensemble of reachable states of the needle tip. Inverse…

Continual Learning in Task-Oriented Dialogue Systems

Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou +6 more

cs.CL · 31 Dec 2020 · cited 151× · arXiv ↗ · more like this ·

Continual learning in task-oriented dialogue systems can allow us to add new domains and functionalities through time without incurring the high cost of a whole system retraining. In this paper, we propose a continual learning benchmark for task-oriented dialogue systems with 37 domains to be learned continuously in four settings, such as intent recognition, state tracking, natural language generation, and end-to-end. Moreover, we implement and compare multiple existing continual learning baselines, and we propose a simple yet effective architectural method based on residual adapters. Our experiments demonstrate that the proposed architectural method and a simple replay-based strategy perform comparably well but they both achieve inferior performance to the multi-task learning baseline, in where all the data are shown at once, showing that continual learning in task-oriented dialogue…

Rethinking Diversified and Discriminative Proposal Generation for Visual Grounding

Zhou Yu, Jun Yu, Chenchao Xiang +3 more

cs.CV · 9 May 2018 · cited 150× · arXiv ↗ · more like this ·

Visual grounding aims to localize an object in an image referred to by a textual query phrase. Various visual grounding approaches have been proposed, and the problem can be modularized into a general framework: proposal generation, multi-modal feature representation, and proposal ranking. Of these three modules, most existing approaches focus on the latter two, with the importance of proposal generation generally neglected. In this paper, we rethink the problem of what properties make a good proposal generator. We introduce the diversity and discrimination simultaneously when generating proposals, and in doing so propose Diversified and Discriminative Proposal Networks model (DDPN). Based on the proposals generated by DDPN, we propose a high performance baseline model for visual grounding and evaluate it on four benchmark datasets. Experimental results demonstrate that our model…

Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension

Ying Xu, Dakuo Wang, Mo Yu +15 more

cs.CL · 26 Mar 2022 · cited 145× · arXiv ↗ · more like this ·

Question answering (QA) is a fundamental means to facilitate assessment and training of narrative comprehension skills for both machines and young children, yet there is scarcity of high-quality QA datasets carefully designed to serve this purpose. In particular, existing datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. Drawing on the reading education research, we introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. Generated by educational experts based on an evidence-based theoretical framework, FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. Our dataset is valuable in two folds: First, we ran existing QA models on our dataset and confirmed that…

Domain Adaptive Dialog Generation via Meta Learning

Kun Qian, Zhou Yu

cs.CL · June 2019 · cited 134× · arXiv ↗ · more like this ·

Domain adaptation is an essential task in dialog system building because there are so many new dialog tasks created for different needs every day. Collecting and annotating training data for these new tasks is costly since it involves real user interactions. We propose a domain adaptive dialog generation method based on meta-learning (DAML). DAML is an end-to-end trainable dialog system model that learns from multiple rich-resource tasks and then adapts to new domains with minimal training samples. We train a dialog system model using multiple rich-resource single-domain dialog data by applying the model-agnostic meta-learning algorithm to dialog domain. The model is capable of learning a competitive dialog system on a new domain with only a few training examples in an efficient manner. The two-step gradient updates in DAML enable the model to learn general features across multiple…

Deep Multimodal Neural Architecture Search

Zhou Yu, Yuhao Cui, Jun Yu +3 more

cs.CV · April 2020 · cited 114× · arXiv ↗ · more like this ·

Designing effective neural networks is fundamentally important in deep multimodal learning. Most existing works focus on a single task and design neural architectures manually, which are highly task-specific and hard to generalize to different tasks. In this paper, we devise a generalized deep multimodal neural architecture search (MMnas) framework for various multimodal learning tasks. Given multimodal input, we first define a set of primitive operations, and then construct a deep encoder-decoder based unified backbone, where each encoder or decoder block corresponds to an operation searched from a predefined operation pool. On top of the unified backbone, we attach task-specific heads to tackle different multimodal learning tasks. By using a gradient-based NAS algorithm, the optimal architectures for different tasks are learned efficiently. Extensive ablation studies, comprehensive…

Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies

Weiyan Shi, Xuewei Wang, Yoo Jung Oh +3 more

cs.HC · 18 Jan 2020 · cited 114× · arXiv ↗ · more like this ·

Intelligent conversational agents, or chatbots, can take on various identities and are increasingly engaging in more human-centered conversations with persuasive goals. However, little is known about how identities and inquiry strategies influence the conversation's effectiveness. We conducted an online study involving 790 participants to be persuaded by a chatbot for charity donation. We designed a two by four factorial experiment (two chatbot identities and four inquiry strategies) where participants were randomly assigned to different conditions. Findings showed that the perceived identity of the chatbot had significant effects on the persuasion outcome (i.e., donation) and interpersonal perceptions (i.e., competence, confidence, warmth, and sincerity). Further, we identified interaction effects among perceived identities and inquiry strategies. We discuss the findings for…

PLACES: Prompting Language Models for Social Conversation Synthesis

Maximillian Chen, Alexandros Papangelis, Chenyang Tao +5 more

cs.CL · 17 Feb 2023 · cited 113× · arXiv ↗ · more like this ·

Collecting high quality conversational data can be very expensive for most applications and infeasible for others due to privacy, ethical, or similar concerns. A promising direction to tackle this problem is to generate synthetic dialogues by prompting large language models. In this work, we use a small set of expert-written conversations as in-context examples to synthesize a social conversation dataset using prompting. We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations. This includes various dimensions of conversation quality with human evaluation directly on the synthesized conversations, and interactive human evaluation of chatbots fine-tuned on the synthetically generated dataset. We additionally demonstrate that this prompting approach is generalizable to multi-party conversations, providing potential to create new…

On the Relation between Sensitivity and Accuracy in In-context Learning

Yanda Chen, Chen Zhao, Zhou Yu +2 more

cs.CL · September 2022 · cited 107× · arXiv ↗ · more like this ·

In-context learning (ICL) suffers from oversensitivity to the prompt, making it unreliable in real-world scenarios. We study the sensitivity of ICL with respect to multiple perturbation types. First, we find that label bias obscures the true sensitivity, and therefore prior work may have significantly underestimated ICL sensitivity. Second, we observe a strong negative correlation between ICL sensitivity and accuracy: predictions sensitive to perturbations are less likely to be correct. Motivated by these findings, we propose \textsc{SenSel}, a few-shot selective prediction method that abstains from sensitive predictions. Experiments on ten classification datasets show that \textsc{SenSel} consistently outperforms two commonly used confidence-based and entropy-based baselines on abstention decisions.

Selective Differential Privacy for Language Modeling

Weiyan Shi, Aiqi Cui, Evan Li +2 more

cs.CL · August 2021 · cited 100× · arXiv ↗ · more like this ·

With the increasing applications of language models, it has become crucial to protect these models from leaking private information. Previous work has attempted to tackle this challenge by training RNN-based language models with differential privacy guarantees. However, applying classical differential privacy to language models leads to poor model performance as the underlying privacy notion is over-pessimistic and provides undifferentiated protection for all tokens in the data. Given that the private information in natural language is sparse (for example, the bulk of an email might not carry personally identifiable information), we propose a new privacy notion, selective differential privacy, to provide rigorous privacy guarantees on the sensitive portion of the data to improve model utility. To realize such a new notion, we develop a corresponding privacy mechanism, Selective-DPSGD…

Deep learning based RF fingerprinting for device identification and wireless security

Qingyang Wu, Carlos Feres, Daniel Kuzmenko +6 more

Artificial Intelligence · Electronics Letters · 9 Oct 2018 · cited 97× · DOI ↗ · more like this ·

RF fingerprinting is an emerging technology for identifying hardware‐specific features of wireless transmitters and may find important applications in wireless security. In this study, the authors present a new RF fingerprinting scheme using deep neural networks. In particular, a long short‐term memory based recurrent neural network is proposed and used for automatically identifying hardware‐specific features and classifying transmitters. Experimental studies using identical RF transmitters showed very high detection accuracy in the presence of strong noise (signal‐to‐noise ratio as low as dB) and demonstrated the effectiveness of the proposed scheme.

On Reinforcement Learning for Full-length Game of StarCraft

Zhen-Jia Pang, Ruo-Ze Liu, Zhou-Yu Meng +3 more

cs.LG · September 2018 · cited 93× · arXiv ↗ · more like this ·

StarCraft II poses a grand challenge for reinforcement learning. The main difficulties of it include huge state and action space and a long-time horizon. In this paper, we investigate a hierarchical reinforcement learning approach for StarCraft II. The hierarchy involves two levels of abstraction. One is the macro-action automatically extracted from expert's trajectories, which reduces the action space in an order of magnitude yet remains effective. The other is a two-layer hierarchical architecture which is modular and easy to scale, enabling a curriculum transferring from simpler tasks to more complex tasks. The reinforcement training algorithm for this architecture is also investigated. On a 64x64 map and using restrictive units, we achieve a winning rate of more than 99\% against the difficulty level-1 built-in AI. Through the curriculum transfer learning algorithm and a mixture of…

Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems

Derek Chen, Howard Chen, Yi Yang +2 more

cs.CL · April 2021 · cited 92× · arXiv ↗ · more like this ·

Existing goal-oriented dialogue datasets focus mainly on identifying slots and values. However, customer support interactions in reality often involve agents following multi-step procedures derived from explicitly-defined company policies as well. To study customer service dialogue systems in more realistic settings, we introduce the Action-Based Conversations Dataset (ABCD), a fully-labeled dataset with over 10K human-to-human dialogues containing 55 distinct user intents requiring unique sequences of actions constrained by policies to achieve task success. We propose two additional dialog tasks, Action State Tracking and Cascading Dialogue Success, and establish a series of baselines involving large-scale, pre-trained language models on this dataset. Empirical results demonstrate that while more sophisticated networks outperform simpler models, a considerable gap (50.8% absolute…