Author

Han Lu

233 papers · 2001–2026

2001peak 34 in 20262026

Frequent co-authors

Papers

showing 1–25 of 233 RSS

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Gemini Team, Petko Georgiev, Ving Ian Lei +1134 more

cs.CL · March 2024 · cited 4,033× · arXiv ↗ · more like this ·

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's…

Gemini: A Family of Highly Capable Multimodal Models

Gemini Team, Rohan Anil, Sebastian Borgeaud +1348 more

cs.CL · December 2023 · cited 816× · arXiv ↗ · more like this ·

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models…

Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss

Qian Zhang, Han Lu, Hasim Sak +4 more

eess.AS · 14 Feb 2020 · cited 544× · arXiv ↗ · more like this ·

In this paper we present an end-to-end speech recognition model with Transformer encoders that can be used in a streaming speech recognition system. Transformer computation blocks based on self-attention are used to encode both audio and label sequences independently. The activations from both audio and label encoders are combined with a feed-forward layer to compute a probability distribution over the label space for every combination of acoustic frame position and label history. This is similar to the Recurrent Neural Network Transducer (RNN-T) model, which uses RNNs for information encoding instead of Transformer encoders. The model is trained with the RNN-T loss well-suited to streaming decoding. We present results on the LibriSpeech dataset showing that limiting the left context for self-attention in the Transformer layers makes decoding computationally tractable for streaming…

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

Haiguang Wen, Junxing Shi, Yizhen Zhang +3 more

q-bio.NC · August 2016 · cited 333× · arXiv ↗ · more like this ·

Convolutional neural network (CNN) driven by image recognition has been shown to be able to explain cortical responses to static pictures at ventral-stream areas. Here, we further showed that such CNN could reliably predict and decode functional magnetic resonance imaging data from humans watching natural movies, despite its lack of any mechanism to account for temporal dynamics or feedback processing. Using separate data, encoding and decoding models were developed and evaluated for describing the bi-directional relationships be-tween the CNN and the brain. Through the encoding models, the CNN-predicted areas covered not only the ventral stream, but also the dorsal stream, albe-it to a lesser degree; single-voxel response was visualized as the specific pixel pattern that drove the response, revealing the distinct representation of individual cortical location; cortical activation was…

Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering

Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu +2 more

cs.CL · July 2023 · cited 224× · arXiv ↗ · more like this ·

Retriever-augmented instruction-following models are attractive alternatives to fine-tuned approaches for information-seeking tasks such as question answering (QA). By simply prepending retrieved documents in its input along with an instruction, these models can be adapted to various information domains and tasks without additional fine-tuning. While the model responses tend to be natural and fluent, the additional verbosity makes traditional QA evaluation metrics such as exact match (EM) and F1 unreliable for accurately quantifying model performance. In this work, we investigate the performance of instruction-following models across three information-seeking QA tasks. We use both automatic and human evaluation to evaluate these models along two dimensions: 1) how well they satisfy the user's information need (correctness), and 2) whether they produce a response based on the provided…

MMTEB: Massive Multilingual Text Embedding Benchmark

Kenneth Enevoldsen, Isaac Chung, Imene Kerboua +83 more

cs.CL · February 2025 · cited 186× · arXiv ↗ · more like this ·

Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain…

Detailed 2D-3D Joint Representation for Human-Object Interaction

Yong-Lu Li, Xinpeng Liu, Han Lu +4 more

cs.CV · April 2020 · cited 157× · arXiv ↗ · more like this ·

Human-Object Interaction (HOI) detection lies at the core of action understanding. Besides 2D information such as human/object appearance and locations, 3D pose is also usually utilized in HOI learning since its view-independence. However, rough 3D body joints just carry sparse body information and are not sufficient to understand complex interactions. Thus, we need detailed 3D body shape to go further. Meanwhile, the interacted object in 3D is also not fully studied in HOI learning. In light of these, we propose a detailed 2D-3D joint representation learning method. First, we utilize the single-view human body capture method to obtain detailed 3D body, face and hand shapes. Next, we estimate the 3D object location and size with reference to the 2D human-object spatial configuration and object category priors. Finally, a joint learning framework and cross-modal consistency tasks are…

Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech

Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang +12 more

eess.AS · September 2023 · cited 129× · arXiv ↗ · more like this ·

Text language models have shown remarkable zero-shot capability in generalizing to unseen tasks when provided with well-formulated instructions. However, existing studies in speech processing primarily focus on limited or specific tasks. Moreover, the lack of standardized benchmarks hinders a fair comparison across different approaches. Thus, we present Dynamic-SUPERB, a benchmark designed for building universal speech models capable of leveraging instruction tuning to perform multiple tasks in a zero-shot fashion. To achieve comprehensive coverage of diverse speech tasks and harness instruction tuning, we invite the community to collaborate and contribute, facilitating the dynamic growth of the benchmark. To initiate, Dynamic-SUPERB features 55 evaluation instances by combining 33 tasks and 22 datasets. This spans a broad spectrum of dimensions, providing a comprehensive platform for…

Hacker types, motivations and strategies: A comprehensive framework

Samuel Chng, Han Lu, Ayush Kumar +1 more

Information Systems · Computers in Human Behavior Reports · 19 Jan 2022 · cited 78× · DOI ↗ · more like this ·

Understanding and predicting cyber malfeasance is an emerging area of research with the increase in cybercrimes and heightened awareness about cybersecurity in recent years. The nature of cybercrimes is also becoming increasingly complex as hackers are more proficient and well-financed than earlier. Thus, our understanding of the various types of hackers and their motivations needs to be consistently updated. We identified and reviewed 11 classifications and typologies of hackers and their motivations published over three decades to consolidate our understanding in this area and summarize the state of the art. Following that, we present a unified framework of 13 hacker types and 7 unique motivations. In addition, we detail the strategies each hacker type typically employs. This framework, applicable to various domains, allows readers to map various motivations to each hacker type and…

Inferring multimodal latent topics from electronic health records

Yue Li, Pratheeksha Nair, Xing Han Lù +9 more

Artificial Intelligence · Nature Communications · 21 May 2020 · cited 76× · DOI ↗ · more like this ·

Electronic health records (EHR) are rich heterogeneous collections of patient health information, whose broad adoption provides clinicians and researchers unprecedented opportunities for health informatics, disease-risk prediction, actionable clinical recommendations, and precision medicine. However, EHRs present several modeling challenges, including highly sparse data matrices, noisy irregular clinical notes, arbitrary biases in billing code assignment, diagnosis-driven lab tests, and heterogeneous data types. To address these challenges, we present MixEHR, a multi-view Bayesian topic model. We demonstrate MixEHR on MIMIC-III, Mayo Clinic Bipolar Disorder, and Quebec Congenital Heart Disease EHR datasets. Qualitatively, MixEHR disease topics reveal meaningful combinations of clinical features across heterogeneous data types. Quantitatively, we observe superior prediction accuracy of…

Learning Character-level Compositionality with Visual Features

Frederick Liu, Han Lu, Chieh Lo +1 more

cs.CL · April 2017 · cited 66× · arXiv ↗ · more like this ·

Previous work has modeled the compositionality of words by creating character-level models of meaning, reducing problems of sparsity for rare words. However, in many writing systems compositionality has an effect even on the character-level: the meaning of a character is derived by the sum of its parts. In this paper, we model this effect by creating embeddings for characters based on their visual characteristics, creating an image for the character and running it through a convolutional neural network to produce a visual character embedding. Experiments on a text classification task demonstrate that such model allows for better processing of instances with rare characters in languages such as Chinese, Japanese, and Korean. Additionally, qualitative analyses demonstrate that our proposed model learns to focus on the parts of characters that carry semantic content, resulting in…

Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection

Wei Xia, Han Lu, Quan Wang +4 more

eess.AS · September 2021 · cited 64× · arXiv ↗ · more like this ·

In this paper, we present a novel speaker diarization system for streaming on-device applications. In this system, we use a transformer transducer to detect the speaker turns, represent each speaker turn by a speaker embedding, then cluster these embeddings with constraints from the detected speaker turns. Compared with conventional clustering-based diarization systems, our system largely reduces the computational cost of clustering due to the sparsity of speaker turns. Unlike other supervised speaker diarization systems which require annotations of time-stamped speaker labels for training, our system only requires including speaker turn tokens during the transcribing process, which largely reduces the human efforts involved in data collection.

Handling Homographs in Neural Machine Translation

Frederick Liu, Han Lu, Graham Neubig

cs.CL · August 2017 · cited 63× · arXiv ↗ · more like this ·

Homographs, words with different meanings but the same surface form, have long caused difficulty for machine translation systems, as it is difficult to select the correct translation based on the context. However, with the advent of neural machine translation (NMT) systems, which can theoretically take into account global sentential context, one may hypothesize that this problem has been alleviated. In this paper, we first provide empirical evidence that existing NMT systems in fact still have significant problems in properly translating ambiguous words. We then proceed to describe methods, inspired by the word sense disambiguation literature, that model the context of the input word with context-aware word embeddings that help to differentiate the word sense be- fore feeding it into the encoder. Experiments on three language pairs demonstrate that such models improve the performance of…

Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm

Yichen Xie, Han Lu, Junchi Yan +3 more

cs.CV · 25 Mar 2023 · cited 56× · arXiv ↗ · more like this ·

Given the large-scale data and the high annotation cost, pretraining-finetuning becomes a popular paradigm in multiple computer vision tasks. Previous research has covered both the unsupervised pretraining and supervised finetuning in this paradigm, while little attention is paid to exploiting the annotation budget for finetuning. To fill in this gap, we formally define this new active finetuning task focusing on the selection of samples for annotation in the pretraining-finetuning paradigm. We propose a novel method called ActiveFT for active finetuning task to select a subset of data distributing similarly with the entire unlabeled pool and maintaining enough diversity by optimizing a parametric model in the continuous space. We prove that the Earth Mover's distance between the distributions of the selected subset and the entire data pool is also reduced in this process. Extensive…

Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition

Anshuman Tripathi, Jaeyoung Kim, Qian Zhang +2 more

cs.SD · 7 Oct 2020 · cited 49× · arXiv ↗ · more like this ·

In this paper we present a Transformer-Transducer model architecture and a training technique to unify streaming and non-streaming speech recognition models into one model. The model is composed of a stack of transformer layers for audio encoding with no lookahead or right context and an additional stack of transformer layers on top trained with variable right context. In inference time, the context length for the variable context layers can be changed to trade off the latency and the accuracy of the model. We also show that we can run this model in a Y-model architecture with the top layers running in parallel in low latency and high latency modes. This allows us to have streaming speech recognition results with limited latency and delayed speech recognition results with large improvements in accuracy (20% relative improvement for voice-search task). We show that with limited right…

MeDAL: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretraining

Zhi Wen, Xing Han Lu, Siva Reddy

cs.CL · 27 Dec 2020 · cited 44× · arXiv ↗ · more like this ·

One of the biggest challenges that prohibit the use of many current NLP methods in clinical settings is the availability of public datasets. In this work, we present MeDAL, a large medical text dataset curated for abbreviation disambiguation, designed for natural language understanding pre-training in the medical domain. We pre-trained several models of common architectures on this dataset and empirically showed that such pre-training leads to improved performance and convergence speed when fine-tuning on downstream medical tasks.

Exploring the roles of vlogger characteristics and video attributes on followers’ value perceptions and behavioral intention

Hsiao-Han Lu, Ching‐Fu Chen, Yuhong Tai

Sociology and Political Science · Journal of Retailing and Consumer Services · 2024 · cited 42× · DOI ↗ · more like this ·

YouTube has emerged as an effective social media platform for brands to reach consumers through YouTube vloggers. However, little is known about how vlogger characteristics and video attributes affect consumers' perceived value and behavioral intention. Drawing on the stimulus‒organism‒response (SOR) framework, this research examines how video attributes (i.e., content quality and visual aesthetics) and vlogger characteristics (i.e., authenticity, interactivity, physical attractiveness, and expertise) influence followers' utilitarian and hedonic values, which consequently lead to continuance intention and purchase intention. The survey data collected from 1014 followers of the vloggers (i.e., beauty and technology) were analyzed by structural equation modeling (SEM) and fuzzy-set qualitative comparative analysis (fsQCA). The results show that visual aesthetics, authenticity, and…

Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment

Zichao Li, Prakhar Sharma, Xing Han Lu +2 more

cs.CL · 6 Apr 2022 · cited 36× · arXiv ↗ · more like this ·

Most research on question answering focuses on the pre-deployment stage; i.e., building an accurate model for deployment. In this paper, we ask the question: Can we improve QA systems further \emph{post-}deployment based on user interactions? We focus on two kinds of improvements: 1) improving the QA system's performance itself, and 2) providing the model with the ability to explain the correctness or incorrectness of an answer. We collect a retrieval-based QA dataset, FeedbackQA, which contains interactive feedback from users. We collect this dataset by deploying a base QA system to crowdworkers who then engage with the system and provide feedback on the quality of its answers. The feedback contains both structured ratings and unstructured natural language explanations. We train a neural model with this feedback data that can generate explanations and re-score answer candidates. We…

The Efficiency of Cooperative Learning in Physical Education on the Learning of Action Skills and Learning Motivation

Cairu Yang, Rongli Chen, Xiaozhong Chen +1 more

Education · Frontiers in Psychology · 20 Oct 2021 · cited 36× · DOI ↗ · more like this ·

This paper proposes a cooperative learning method for use in physical education, involving two different grouping methods: S-type heterogeneous grouping and "free" grouping. Cooperative learning was found to enhance the effectiveness of basketball skills learning and learning motivation. A comparison was made of the differences between action skills grouping (the control group) and "free" grouping (the experimental group). The ARCS Motivation Scale and Basketball Action Skills Test were used to measure results, and SPSS statistical analysis software was used for relevant statistical processing (with α set to.05). The results showed that overall skills, dribbling and passing among the action skills groups and "free" groupings significantly improved, but results for shooting were not significant; motivation levels for the two grouping methods significantly improved overall, and no…

How do influencers’ characteristics affect followers’ stickiness and well-being in the social media context?

Hsiao-Han Lu, Ching‐Fu Chen

Sociology and Political Science · Journal of Services Marketing · 5 Jun 2023 · cited 34× · DOI ↗ · more like this ·

Purpose Drawing on source credibility theory, this study aims to explore the relationship between influencers’ characteristics and followers’ well-being through followers’ stickiness to influencers’ channels or posts. Design/methodology/approach The authors gathered data through an online survey of 450 followers. The proposed hypotheses were examined using structural equation modeling. Findings Results reveal that trustworthiness and physical attractiveness positively affect followers’ stickiness to influencers, while the influence of expertise is not evident. Stickiness has positive impacts on both hedonic and eudaimonic well-being. The findings also show that stickiness mediates the relationships between physical attractiveness and trustworthiness toward both hedonic and eudaimonic well-being. Originality/value To the best of the authors’ knowledge, this study is among the first to…

Recurrent disease progression networks for modelling risk trajectory of heart failure

Xing Han Lù, Aihua Liu, Shih-Chieh Fuh +5 more

Artificial Intelligence · PLoS ONE · 6 Jan 2021 · cited 33× · DOI ↗ · more like this ·

MOTIVATION: Recurrent neural networks (RNN) are powerful frameworks to model medical time series records. Recent studies showed improved accuracy of predicting future medical events (e.g., readmission, mortality) by leveraging large amount of high-dimensional data. However, very few studies have explored the ability of RNN in predicting long-term trajectories of recurrent events, which is more informative than predicting one single event in directing medical intervention. METHODS: In this study, we focus on heart failure (HF) which is the leading cause of death among cardiovascular diseases. We present a novel RNN framework named Deep Heart-failure Trajectory Model (DHTM) for modelling the long-term trajectories of recurrent HF. DHTM auto-regressively predicts the future HF onsets of each patient and uses the predicted HF as input to predict the HF event at the next time point.…

Non-Autoregressive ASR Modeling Using Pre-Trained Language Models for Chinese Speech Recognition

Fu-Hao Yu, Kuan-Yu Chen, Ke-Han Lu

Artificial Intelligence · IEEE/ACM Transactions on Audio Speech and Language Processing · 2022 · cited 27× · DOI ↗ · more like this ·

Transformer-based models have led to significant innovation in various classic and practical subjects, including speech processing, natural language processing, and computer vision. On top of the Transformer, attention-based end-to-end automatic speech recognition (ASR) models have become a popular fashion in recent years. Specifically, an emergent research topic is non-autoregressive modeling, which can achieve fast inference speed and obtain competitive performance when compared with conventional autoregressive methods. In addition, in the context of natural language processing, the bidirectional encoder representations from Transformers (BERT) model and its variants have received widespread attention, partially due to their ability to infer contextualized word representations and obtain superior performances of downstream tasks through simple fine-tuning. However, to our knowledge…

A context-aware knowledge transferring strategy for CTC-based ASR

Ke-Han Lu, Kuan-Yu Chen

cs.CL · 12 Oct 2022 · cited 26× · arXiv ↗ · more like this ·

Non-autoregressive automatic speech recognition (ASR) modeling has received increasing attention recently because of its fast decoding speed and superior performance. Among representatives, methods based on the connectionist temporal classification (CTC) are still a dominating stream. However, the theoretically inherent flaw, the assumption of independence between tokens, creates a performance barrier for the school of works. To mitigate the challenge, we propose a context-aware knowledge transferring strategy, consisting of a knowledge transferring module and a context-aware training strategy, for CTC-based ASR. The former is designed to distill linguistic information from a pre-trained language model, and the latter is framed to modulate the limitations caused by the conditional independence assumption. As a result, a knowledge-injected context-aware CTC-based ASR built upon the…

Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision

Chih-Kai Yang, Kuan-Po Huang, Ke-Han Lu +3 more

eess.AS · 30 Dec 2023 · cited 19× · arXiv ↗ · more like this ·

This work evaluated several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora. We found that self-supervised models can achieve performances close to the supervised model, indicating the effectiveness of multilingual self-supervised pre-training. We also observed that these models still have room for improvement as they kept making similar mistakes and had unsatisfactory performances on modeling intra-sentential code-switching. In addition, the validity of several variants of Whisper was explored, and we concluded that they remained effective in a code-switching scenario, and similar techniques for self-supervised models are worth studying to boost the performance of code-switched tasks.

Contrastive Siamese Network for Semi-supervised Speech Recognition

Soheil Khorram, Jaeyoung Kim, Anshuman Tripathi +3 more

cs.LG · 27 May 2022 · cited 17× · arXiv ↗ · more like this ·

This paper introduces contrastive siamese (c-siam) network, an architecture for leveraging unlabeled acoustic data in speech recognition. c-siam is the first network that extracts high-level linguistic information from speech by matching outputs of two identical transformer encoders. It contains augmented and target branches which are trained by: (1) masking inputs and matching outputs with a contrastive loss, (2) incorporating a stop gradient operation on the target branch, (3) using an extra learnable transformation on the augmented branch, (4) introducing new temporal augment functions to prevent the shortcut learning problem. We use the Libri-light 60k unsupervised data and the LibriSpeech 100hrs/960hrs supervised data to compare c-siam and other best-performing systems. Our experiments show that c-siam provides 20% relative word error rate improvement over wav2vec baselines. A…