arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Jianfei Yu

55 papers · 2014–2026

2014: 12015: 12016: 02017: 32018: 42019: 22020: 92021: 72022: 52023: 92024: 82025: 52026: 1
2014peak 9 in 20202026

Frequent co-authors

Papers

showing 1–25 of 55 RSS

Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer

Jianfei Yu, Jing Jiang, Yang Li +1 more

Artificial Intelligence · Institutional Knowledge (InK) - Institutional Knowledge at Singapore Management University (Singapore Management University) · · cited 267× · DOI ↗ · more like this ·

In this paper, we study Multimodal Named Entity Recognition (MNER) for social media posts. Existing approaches for MNER mainly suffer from two drawbacks: (1) despite generating word-aware visual representations, their word representations are insensitive to the visual context; (2) most of them ignore the bias brought by the visual context. To tackle the first issue, we propose a multimodal interaction module to obtain both image-aware word representations and word-aware visual representations. To alleviate the visual bias, we further propose to leverage purely text-based entity span detection as an auxiliary module, and design a Unified Multimodal Transformer to guide the final predictions with the entity span predictions. Experiments show that our unified approach achieves the new state-of-the-art performance on two benchmark datasets.

Adapting BERT for Target-Oriented Multimodal Sentiment Classification

Jianfei Yu, Jing Jiang

Artificial Intelligence · Institutional Knowledge (InK) - Institutional Knowledge at Singapore Management University (Singapore Management University) · · cited 261× · DOI ↗ · more like this ·

As an important task in Sentiment Analysis, Target-oriented Sentiment Classification (TSC) aims to identify sentiment polarities over each opinion target in a sentence. However, existing approaches to this task primarily rely on the textual content, but ignoring the other increasingly popular multimodal data sources (e.g., images), which can enhance the robustness of these text-based models. Motivated by this observation and inspired by the recently proposed BERT architecture, we study Target-oriented Multimodal Sentiment Classification (TMSC) and propose a multimodal BERT architecture. To model intra-modality dynamics, we first apply BERT to obtain target-sensitive textual representations. We then borrow the idea from self-attention and design a target attention mechanism to perform target-image matching to derive target-sensitive visual representations. To model inter-modality…

Entity-Sensitive Attention and Fusion Network for Entity-Level Multimodal Sentiment Classification

Jianfei Yu, Jing Jiang, Rui Xia

Artificial Intelligence · IEEE/ACM Transactions on Audio Speech and Language Processing · · cited 218× · DOI ↗ · more like this ·

Entity-level (aka target-dependent) sentiment analysis of social media posts has recently attracted increasing attention, and its goal is to predict the sentiment orientations over individual target entities mentioned in users' posts. Most existing approaches to this task primarily rely on the textual content, but fail to consider the other important data sources (e.g., images, videos, and user profiles), which can potentially enhance these text-based approaches. Motivated by the observation, we study entity-level multimodal sentiment classification in this article, and aim to explore the usefulness of images for entity-level sentiment detection in social media posts. Specifically, we propose an Entity-Sensitive Attention and Fusion Network (ESAFN) for this task. First, to capture the intra-modality dynamics, ESAFN leverages an effective attention mechanism to generate entity-sensitive…

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Yan Ling, Jianfei Yu, Rui Xia

cs.CV · · cited 136× · arXiv ↗ · more like this ·

As an important task in sentiment analysis, Multimodal Aspect-Based Sentiment Analysis (MABSA) has attracted increasing attention in recent years. However, previous approaches either (i) use separately pre-trained visual and textual models, which ignore the crossmodal alignment or (ii) use vision-language models pre-trained with general pre-training tasks, which are inadequate to identify finegrained aspects, opinions, and their alignments across modalities. To tackle these limitations, we propose a task-specific Vision-Language Pre-training framework for MABSA (VLPMABSA), which is a unified multimodal encoder-decoder architecture for all the pretraining and downstream tasks. We further design three types of task-specific pre-training tasks from the language, vision, and multimodal modalities, respectively. Experimental results show that our approach generally outperforms the…

ECPE-2D: Emotion-Cause Pair Extraction based on Joint Two-Dimensional Representation, Interaction and Prediction

Zixiang Ding, Rui Xia, Jianfei Yu

Artificial Intelligence · · cited 134× · DOI ↗ · more like this ·

In recent years, a new interesting task, called emotion-cause pair extraction (ECPE), has emerged in the area of text emotion analysis. It aims at extracting the potential pairs of emotions and their corresponding causes in a document. To solve this task, the existing research employed a two-step framework, which first extracts individual emotion set and cause set, and then pair the corresponding emotions and causes. However, such a pipeline of two steps contains some inherent flaws: 1) the modeling does not aim at extracting the final emotion-cause pair directly; 2) the errors from the first step will affect the performance of the second step. To address these shortcomings, in this paper we propose a new end-toend approach, called ECPE-Two-Dimensional (ECPE-2D), to represent the emotion-cause pairs by a 2D representation scheme. A 2D transformer module and two variants…

Hierarchical Interactive Multimodal Transformer for Aspect-Based Multimodal Sentiment Analysis

Jianfei Yu, Kai Chen, Rui Xia

Artificial Intelligence · IEEE Transactions on Affective Computing · · cited 128× · DOI ↗ · more like this ·

Aspect-based multimodal sentiment analysis (ABMSA) aims to determine the sentiment polarities of each aspect or entity mentioned in a multimodal post or review. Previous studies to ABMSA can be summarized into two subtasks: aspect-term based multimodal sentiment classification (ATMSC) and aspect-category based multimodal sentiment classification (ACMSC). However, these existing studies have three shortcomings: (1) ignoring the object-level semantics in images; (2) primarily focusing on aspect-text and aspect-image interactions; (3) failing to consider the semantic gap between text and image representations. To tackle these issues, we propose a general Hierarchical Interactive Multimodal Transformer (HIMT) model for ABMSA. Specifically, we extract salient features with semantic concepts from images via an object detection method, and then propose a hierarchical interaction module to…

Recurrent Neural Networks with Auxiliary Labels for Cross-Domain Opinion Target Extraction

Ying Ding, Jianfei Yu, Jing Jiang

Artificial Intelligence · Proceedings of the AAAI Conference on Artificial Intelligence · · cited 120× · DOI ↗ · more like this ·

Opinion target extraction is a fundamental task in opinion mining. In recent years, neural network based supervised learning methods have achieved competitive performance on this task. However, as with any supervised learning method, neural network based methods for this task cannot work well when the training data comes from a different domain than the test data. On the other hand, some rule-based unsupervised methods have shown to be robust when applied to different domains. In this work, we use rule-based unsupervised methods to create auxiliary labels and use neural network models to learn a hidden representation that works well for different domains. When this hidden representation is used for opinion target extraction, we find that it can outperform a number of strong baselines with a large margin.

Global Inference for Aspect and Opinion Terms Co-Extraction Based on Multi-Task Neural Networks

Jianfei Yu, Jing Jiang, Rui Xia

Artificial Intelligence · IEEE/ACM Transactions on Audio Speech and Language Processing · · cited 108× · DOI ↗ · more like this ·

Extracting aspect terms and opinion terms are two fundamental tasks in opinion mining. The recent success of deep learning has inspired various neural network architectures, which have been shown to achieve highly competitive performance in these two tasks. However, most existing methods fail to explicitly consider the syntactic relations among aspect terms and opinion terms, which may lead to the inconsistencies between the model predictions and the syntactic constraints. To this end, we first apply a multi-task learning framework to implicitly capture the relations between the two tasks, and then propose a global inference method by explicitly modelling several syntactic constraints among aspect term extraction and opinion term extraction to uncover their intra-task and inter-task relationship, which seeks an optimal solution over the neural predictions for both tasks. Extensive…

Modelling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce

Jianfei Yu, Minghui Qiu, Jing Jiang +4 more

cs.CL · · cited 105× · arXiv ↗ · more like this ·

In this paper, we study transfer learning for the PI and NLI problems, aiming to propose a general framework, which can effectively and efficiently adapt the shared knowledge learned from a resource-rich source domain to a resource- poor target domain. Specifically, since most existing transfer learning methods only focus on learning a shared feature space across domains while ignoring the relationship between the source and target domains, we propose to simultaneously learn shared representations and domain relationships in a unified framework. Furthermore, we propose an efficient and effective hybrid model by combining a sentence encoding- based method and a sentence interaction-based method as our base model. Extensive experiments on both paraphrase identification and natural language inference demonstrate that our base model is efficient and has promising performance compared to the…

Aspect-Category based Sentiment Analysis with Hierarchical Graph Convolutional Network

Hongjie Cai, Yaofeng Tu, Xiangsheng Zhou +2 more

Artificial Intelligence · · cited 102× · DOI ↗ · more like this ·

Most of the aspect based sentiment analysis research aims at identifying the sentiment polarities toward some explicit aspect terms while ignores implicit aspects in text. To capture both explicit and implicit aspects, we focus on aspect-category based sentiment analysis, which involves joint aspect category detection and category-oriented sentiment classification. However, currently only a few simple studies have focused on this problem. The shortcomings in the way they defined the task make their approaches difficult to effectively learn the inner-relations between categories and the inter-relations between categories and sentiments. In this work, we re-formalize the task as a category-sentiment hierarchy prediction problem, which contains a hierarchy output structure to first identify multiple aspect categories in a review sentence, and jointly predict the sentiment for each of the…

Multimodal Emotion-Cause Pair Extraction in Conversations

Fanfan Wang, Zixiang Ding, Rui Xia +2 more

cs.CL · · cited 92× · arXiv ↗ · more like this ·

Emotion cause analysis has received considerable attention in recent years. Previous studies primarily focused on emotion cause extraction from texts in news articles or microblogs. It is also interesting to discover emotions and their causes in conversations. As conversation in its natural form is multimodal, a large number of studies have been carried out on multimodal emotion recognition in conversations, but there is still a lack of work on multimodal emotion cause analysis. In this work, we introduce a new task named Multimodal Emotion-Cause Pair Extraction in Conversations, aiming to jointly extract emotions and their associated causes from conversations reflected in multiple modalities (text, audio and video). We accordingly construct a multimodal conversational emotion cause dataset, Emotion-Cause-in-Friends, which contains 9,272 multimodal emotion-cause pairs annotated on…

End-to-End Emotion-Cause Pair Extraction based on Sliding Window Multi-Label Learning

Zixiang Ding, Rui Xia, Jianfei Yu

Artificial Intelligence · · cited 85× · DOI ↗ · more like this ·

Emotion-cause pair extraction (ECPE) is a new task that aims to extract the potential pairs of emotions and their corresponding causes in a document. The existing methods first perform emotion extraction and cause extraction independently, and then perform emotion-cause pairing and filtering. However, the above methods ignore the fact that the cause and the emotion it triggers are inseparable, and the extraction of the cause without specifying the emotion is pathological, which greatly limits the performance of the above methods in the first step. To tackle these shortcomings, we propose two joint frameworks for ECPE: 1) multi-label learning for the extraction of the cause clauses corresponding to the specified emotion clause (CMLL) and 2) multi-label learning for the extraction of the emotion clauses corresponding to the specified cause clause (EMLL). The window of multi-label learning…

Improving Multi-label Emotion Classification via Sentiment Classification with Dual Attention Transfer Network

Jianfei Yu, Luís Marujo, Jing Jiang +2 more

Artificial Intelligence · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · · cited 74× · DOI ↗ · more like this ·

In this paper, we target at improving the performance of multi-label emotion classification with the help of sentiment classification. Specifically, we propose a new transfer learning architecture to divide the sentence representation into two different feature spaces, which are expected to respectively capture the general sentiment words and the other important emotion-specific words via a dual attention mechanism. Extensive experimental results demonstrate that our transfer learning approach can outperform several strong baselines and achieve the state-of-the-art performance on two benchmark datasets.

Unified Feature and Instance Based Domain Adaptation for Aspect-Based Sentiment Analysis

Chenggong Gong, Jianfei Yu, Rui Xia

Artificial Intelligence · · cited 67× · DOI ↗ · more like this ·

The supervised models for aspect-based sentiment analysis (ABSA) rely heavily on labeled data. However, fine-grained labeled data are scarce for the ABSA task. To alleviate the dependence on labeled data, prior works mainly focused on feature-based adaptation, which used the domain-shared knowledge to construct auxiliary tasks or domain adversarial learning to bridge the gap between domains, while ignored the attribute of instance-based adaptation. To resolve this limitation, we propose an end-to-end framework to jointly perform feature and instance based adaptation for the ABSA task in this paper. Based on BERT, we learn domain-invariant feature representations by using part-of-speech features and syntactic dependency relations to construct auxiliary tasks, and jointly perform word-level instance weighting in the framework of sequence labeling. Experiment results on four benchmarks…

Targeted Multimodal Sentiment Classification based on Coarse-to-Fine Grained Image-Target Matching

Jianfei Yu, Jieming Wang, Rui Xia +1 more

Artificial Intelligence · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · · cited 51× · DOI ↗ · more like this ·

Targeted Multimodal Sentiment Classification (TMSC) aims to identify the sentiment polarities over each target mentioned in a pair of sentence and image. Existing methods to TMSC failed to explicitly capture both coarse-grained and fine-grained image-target matching, including 1) the relevance between the image and the target and 2) the alignment between visual objects and the target. To tackle this issue, we propose a new multi-task learning architecture named coarse-to-fine grained Image-Target Matching network (ITM), which jointly performs image-target relevance classification, object-target alignment, and targeted sentiment classification. We further construct an Image-Target Matching dataset by manually annotating the image-target relevance and the visual object aligned with the input target. Experiments on two benchmark TMSC datasets show that our model consistently outperforms…

Coupled Hierarchical Transformer for Stance-Aware Rumor Verification in Social Media Conversations

Jianfei Yu, Jing Jiang, Ling Min Serena Khoo +2 more

Sociology and Political Science · Institutional Knowledge (InK) - Institutional Knowledge at Singapore Management University (Singapore Management University) · · cited 49× · DOI ↗ · more like this ·

The prevalent use of social media enables rapid spread of rumors on a massive scale, which leads to the emerging need of automatic rumor verification (RV). A number of previous studies focus on leveraging stance classification to enhance RV with multi-task learning (MTL) methods. However, most of these methods failed to employ pre-trained contextualized embeddings such as BERT, and did not exploit inter-task dependencies by using predicted stance labels to improve the RV task. Therefore, in this paper, to extend BERT to obtain thread representations, we first propose a Hierarchical Transformer 1 , which divides each long thread into shorter subthreads, and employs BERT to separately represent each subthread, followed by a global Transformer layer to encode all the subthreads. We further propose a Coupled Transformer Module to capture the inter-task interactions and a Post-Level…

Cross-Domain Review Generation for Aspect-Based Sentiment Analysis

Jianfei Yu, Chenggong Gong, Rui Xia

Artificial Intelligence · · cited 42× · DOI ↗ · more like this ·

Supervised learning methods have proven to be effective for Aspect-Based Sentiment Analysis (ABSA). However, the lack of finegrained labeled data hinders their effectiveness in many domains. To address this issue, unsupervised domain adaptation methods are desired to transfer knowledge from a labeled source domain to any unlabeled target domain. In this paper, we propose a new domain adaptation paradigm called cross-domain review generation (CDRG), which aims to generate target-domain reviews with fine-grained annotation based on the source-domain labeled reviews. To achieve this goal, we propose a two-step approach as a concrete realization of CDRG. It first converts a sourcedomain review to a domain-independent review by masking its source-specific attributes, and then converts the domain-independent review to a target-domain review with a masked language model pre-trained in the…

A State-independent and Time-evolving Network for Early Rumor Detection in Social Media

Rui Xia, Kaizhou Xuan, Jianfei Yu

Sociology and Political Science · · cited 38× · DOI ↗ · more like this ·

In this paper, we study automatic rumor detection for in social media at the event level where an event consists of a sequence of posts organized according to the posting time. It is common that the state of an event is dynamically evolving. However, most of the existing methods to this task ignored this problem, and established a global representation based on all the posts in the event's life cycle. Such coarse-grained methods failed to capture the event's unique features in different states. To address this limitation, we propose a state-independent and time-evolving Network (STN) for rumor detection based on fine-grained event state detection and segmentation. Given an event composed of a sequence of posts, STN first predicts the corresponding sequence of states and segments the event into several state-independent sub-events. For each sub-event, STN independently trains an encoder…

Grounded Multimodal Named Entity Recognition on Social Media

Jianfei Yu, Ziyan Li, Jieming Wang +1 more

Artificial Intelligence · · cited 33× · DOI ↗ · more like this ·

In recent years, Multimodal Named Entity Recognition (MNER) on social media has attracted considerable attention. However, existing MNER studies only extract entity-type pairs in text, which is useless for multimodal knowledge graph construction and insufficient for entity disambiguation. To solve these issues, in this work, we introduce a Grounded Multimodal Named Entity Recognition (GMNER) task. Given a text-image social post, GMNER aims to identify the named entities in text, their entity types, and their bounding box groundings in image (i.e. visual regions). To tackle the GMNER task, we construct a Twitter dataset based on two existing MNER datasets. Moreover, we extend four well-known MNER methods to establish a number of baseline systems and further propose a Hierarchical Index generation framework named H-Index, which generates the entity-type-region triples in a hierarchical…

Instance-based Domain Adaptation via Multiclustering Logistic Approximation

Feng Xu, Jianfei Yu, Rui Xia

Artificial Intelligence · IEEE Intelligent Systems · · cited 33× · DOI ↗ · more like this ·

With the explosive growth of the Internet online texts, we could nowadays easily collect a large amount of labeled training data from different source domains. However, a basic assumption in building statistical machine learning models for sentiment analysis is that the training and test data must be drawn from the same distribution. Directly training a statistical model usually results in poor performance, when the training and test data have different distributions. Faced with the massive labeled data from different domains, it is therefore important to identify the source-domain training instances that are closely relevant to the target domain, and make better use of them. In this work, we propose a new approach, called multiclustering logistic approximation (MLA), to address this problem. In MLA, we adapt the source-domain training data to the target domain via a framework of…

Generative Cross-Domain Data Augmentation for Aspect and Opinion Co-Extraction

Junjie Li, Jianfei Yu, Rui Xia

Artificial Intelligence · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · · cited 26× · DOI ↗ · more like this ·

As a fundamental task in opinion mining, aspect and opinion co-extraction aims to identify the aspect terms and opinion terms in reviews. However, due to the lack of fine-grained annotated resources, it is hard to train a robust model for many domains. To alleviate this issue, unsupervised domain adaptation is proposed to transfer knowledge from a labeled source domain to an unlabeled target domain. In this paper, we propose a new Generative Cross-Domain Data Augmentation framework for unsupervised domain adaptation. The proposed framework is aimed to generate targetdomain data with fine-grained annotation by exploiting the labeled data in the source domain. Specifically, we remove the domain-specific segments in a source-domain labeled sentence, and then use this as input to a pre-trained sequence-to-sequence model BART to simultaneously generate a target-domain sentence and predict…

Cross-Domain Data Augmentation with Domain-Adaptive Language Modeling for Aspect-Based Sentiment Analysis

Jianfei Yu, Qiankun Zhao, Rui Xia

Artificial Intelligence · · cited 19× · DOI ↗ · more like this ·

Cross-domain Aspect-Based Sentiment Analysis (ABSA) aims to leverage the useful knowledge from a source domain to identify aspect-sentiment pairs in sentences from a target domain. To tackle the task, several recent works explore a new unsupervised domain adaptation framework, i.e., Cross-Domain Data Augmentation (CDDA), aiming to directly generate much labeled target-domain data based on the labeled source-domain data. However, these CDDA methods still suffer from several issues: 1) preserving many source-specific attributes such as syntactic structures; 2) lack of fluency and coherence; 3) limiting the diversity of generated data. To address these issues, we propose a new cross-domain Data Augmentation approach based on Domain-Adaptive Language Modeling named DA2LM, which contains three stages: 1) assigning pseudo labels to unlabeled target-domain data; 2) unifying the process of…

Reinforced Counterfactual Data Augmentation for Dual Sentiment Classification

Hao Chen, Rui Xia, Jianfei Yu

Artificial Intelligence · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · · cited 19× · DOI ↗ · more like this ·

Data augmentation and adversarial perturbation approaches have recently achieved promising results in solving the over-fitting problem in many natural language processing (NLP) tasks including sentiment classification. However, existing studies aimed to improve the generalization ability by augmenting the training data with synonymous examples or adding random noises to word embeddings, which cannot address the spurious association problem. In this work, we propose an end-toend reinforcement learning framework, which jointly performs counterfactual data generation and dual sentiment classification. Our approach has three characteristics: 1) the generator automatically generates massive and diverse antonymous sentences; 2) the discriminator contains a original-side sentiment predictor and an antonymous-side sentiment predictor, which jointly evaluate the quality of the generated sample…

Instance-Based Domain Adaptation in NLP via In-Target-Domain Logistic Approximation

Rui Xia, Jianfei Yu, Feng Xu +1 more

Artificial Intelligence · Proceedings of the AAAI Conference on Artificial Intelligence · · cited 18× · DOI ↗ · more like this ·

In the field of NLP, most of the existing domain adaptation studies belong to the feature-based adaptation, while the research of instance-based adaptation is very scarce. In this work, we propose a new instance-based adaptation model, called in-target-domain logistic approximation (ILA). In ILA, we adapt the source-domain data to the target domain by a logistic approximation. The normalized in-target-domain probability is assigned as an instance weight to each of the source-domain training data. An instance-weighted classification model is trained finally for the cross-domain classification problem. Compared to the previous techniques, ILA conducts instance adaptation in a dimensionality-reduced linear feature space to ensure efficiency in high-dimensional NLP tasks. The instance weights in ILA are learnt by leveraging the criteria of both maximum likelihood and minimum statistical…