arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Linjun Yang

94 papers · 2005–2026

2005: 12006: 02007: 62008: 62009: 122010: 102011: 102012: 132013: 52014: 22015: 42016: 32017: 12018: 12019: 02020: 12021: 12022: 62023: 32024: 72025: 12026: 1
2005peak 13 in 20122026

Frequent co-authors

Papers

showing 1–25 of 94 RSS

Text Embeddings by Weakly-Supervised Contrastive Pre-training

Liang Wang, Nan Yang, Xiaolong Huang +5 more

cs.CL · · cited 1,699× · arXiv ↗ · more like this ·

This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. We conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data. When fine-tuned, E5 obtains the best results on the MTEB benchmark, beating existing embedding models with 40x more parameters.

Multilingual E5 Text Embeddings: A Technical Report

Liang Wang, Nan Yang, Xiaolong Huang +3 more

cs.CL · · cited 637× · arXiv ↗ · more like this ·

This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a balance between the inference efficiency and embedding quality. The training procedure adheres to the English E5 model recipe, involving contrastive pre-training on 1 billion multilingual text pairs, followed by fine-tuning on a combination of labeled datasets. Additionally, we introduce a new instruction-tuned embedding model, whose performance is on par with state-of-the-art, English-only models of similar sizes. Information regarding the model release can be found at https://github.com/microsoft/unilm/tree/master/e5 .

CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise

Kuang-Huei Lee, Xiaodong He, Lei Zhang +1 more

cs.CV · · cited 522× · arXiv ↗ · more like this ·

In this paper, we study the problem of learning image classification models with label noise. Existing approaches depending on human supervision are generally not scalable as manually identifying correct or incorrect labels is time-consuming, whereas approaches not relying on human supervision are scalable but less effective. To reduce the amount of human supervision for label noise cleaning, we introduce CleanNet, a joint neural embedding network, which only requires a fraction of the classes being manually verified to provide the knowledge of label noise that can be transferred to other classes. We further integrate CleanNet and conventional convolutional neural network classifier into one framework for image classification learning. We demonstrate the effectiveness of the proposed algorithm on both of the label noise detection task and the image classification on noisy data task on…

Tag ranking

Dong Liu, Xian‐Sheng Hua, Linjun Yang +2 more

Computer Vision and Pattern Recognition · · cited 482× · DOI ↗ · more like this ·

Social media sharing web sites like Flickr allow users to annotate images with free tags, which significantly facilitate Web image search and organization. However, the tags associated with an image generally are in a random order without any importance or relevance information, which limits the effectiveness of these tags in search and other applications. In this paper, we propose a tag ranking scheme, aiming to automatically rank the tags associated with a given image according to their relevance to the image content. We first estimate initial relevance scores for the tags based on probability density estimation, and then perform a random walk over a tag similarity graph to refine the relevance scores. Experimental results on a 50, 000 Flickr photo collection

Improving Text Embeddings with Large Language Models

Liang Wang, Nan Yang, Xiaolong Huang +3 more

cs.CL · · cited 450× · arXiv ↗ · more like this ·

In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data.…

Embedding-based Retrieval in Facebook Search

Jui-Ting Huang, Ashish Sharma, Shuying Sun +6 more

cs.IR · · cited 433× · arXiv ↗ · more like this ·

Search in social networks such as Facebook poses different challenges than in classical web search: besides the query text, it is important to take into account the searcher's context to provide relevant results. Their social graph is an integral part of this context and is a unique aspect of Facebook search. While embedding-based retrieval (EBR) has been applied in eb search engines for years, Facebook search was still mainly based on a Boolean matching model. In this paper, we discuss the techniques for applying EBR to a Facebook Search system. We introduce the unified embedding framework developed to model semantic embeddings for personalized search, and the system to serve embedding-based retrieval in a typical search system based on an inverted index. We discuss various tricks and experiences on end-to-end optimization of the whole system, including ANN parameter tuning and…

Ensemble manifold regularization

Bo Geng, Chao Xu, Dacheng Tao +2 more

Computer Vision and Pattern Recognition · 2009 IEEE Conference on Computer Vision and Pattern Recognition · · cited 231× · DOI ↗ · more like this ·

We propose an automatic approximation of the intrinsic manifold for general semi-supervised learning problems. Unfortunately, it is not trivial to define an optimization function to obtain optimal hyperparameters. Usually, pure cross-validation is considered but it does not necessarily scale up. A second problem derives from the suboptimality incurred by discrete grid search and overfitting problems. As a consequence, we developed an ensemble manifold regularization (EMR) framework to approximate the intrinsic manifold by combining several initial guesses. Algorithmically, we designed EMR very carefully so that it (a) learns both the composite manifold and the semi-supervised classifier jointly; (b) is fully automatic for learning the intrinsic manifold hyperparameters implicitly; (c) is conditionally optimal for intrinsic manifold approximation under a mild and reasonable assumption…

Learning to tag

Lei Wu, Linjun Yang, Nenghai Yu +1 more

Computer Vision and Pattern Recognition · · cited 186× · DOI ↗ · more like this ·

Social tagging provides valuable and crucial information for large-scale web image retrieval. It is ontology-free and easy to obtain; however, irrelevant tags frequently appear, and users typically will not tag all semantic objects in the image, which is also called semantic loss. To avoid noises and compensate for the semantic loss, tag recommendation is proposed in literature. However, current recommendation simply ranks the related tags based on the single modality of tag co-occurrence on the whole dataset, which ignores other modalities, such as visual correlation. This paper proposes a multi-modality recommendation based on both tag and visual correlation, and formulates the tag recommendation as a learning problem. Each modality is used to generate a ranking feature, and Rankboost algorithm is applied to learn an optimal combination of these ranking features from different…

Visual query suggestion

Zheng-Jun Zha, Linjun Yang, Tao Mei +4 more

Computer Vision and Pattern Recognition · ACM Transactions on Multimedia Computing Communications and Applications · · cited 178× · DOI ↗ · more like this ·

Query suggestion is an effective approach to bridge the Intention Gap between the users' search intents and queries. Most existing search engines are able to automatically suggest a list of textual query terms based on users' current query input, which can be called Textual Query Suggestion. This article proposes a new query suggestion scheme named Visual Query Suggestion (VQS) which is dedicated to image search. VQS provides a more effective query interface to help users to precisely express their search intents by joint text and image suggestions. When a user submits a textual query, VQS first provides a list of suggestions, each containing a keyword and a collection of representative images in a dropdown menu. Once the user selects one of the suggestions, the corresponding keyword will be added to complement the initial query as the new textual query, while the image collection will…

SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

Liang Wang, Nan Yang, Xiaolong Huang +5 more

cs.IR · · cited 149× · arXiv ↗ · more like this ·

In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA, to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings. Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more…

Online video recommendation based on multimodal fusion and relevance feedback

Bo Yang, Tao Mei, Xian‐Sheng Hua +3 more

Computer Vision and Pattern Recognition · · cited 146× · DOI ↗ · more like this ·

With Internet delivery of video content surging to an un-precedented level, video recommendation has become a very popular online service. The capability of recommending relevant videos to targeted users can alleviate users' efforts on finding the most relevant content according to their current viewings or preferences. This paper presents a novel online video recommendation system based on multimodal fusion and relevance feedback. Given an online video document, which usually consists of video content and related information (such as query, title, tags, and surroundings), video recommendation is formulated as finding a list of the most relevant videos in terms of multimodal relevance. We express the multimodal relevance between two video documents as the combination of textual, visual, and aural relevance. Furthermore, since different video documents have different weights of the…

Bayesian video search reranking

Xinmei Tian, Linjun Yang, Jingdong Wang +3 more

Computer Vision and Pattern Recognition · · cited 134× · DOI ↗ · more like this ·

Content-based video search reranking can be regarded as a process that uses visual content to recover the "true" ranking list from the noisy one generated based on textual information. This paper explicitly formulates this problem in the Bayesian framework, i.e., maximizing the ranking score consistency among visually similar video shots while minimizing the ranking distance, which represents the disagreement between the objective ranking list and the initial text-based. Different from existing point-wise ranking distance measures, which compute the distance in terms of the individual scores, two new methods are proposed in this paper to measure the ranking distance based on the disagreement in terms of pair-wise orders. Specifically, hinge distance penalizes the pairs with reversed order according to the degree of the reverse, while preference strength distance further considers the…

Real-time large scale near-duplicate web video retrieval

Lifeng Shang, Linjun Yang, Fei Wang +2 more

Computer Vision and Pattern Recognition · The HKU Scholars Hub (University of Hong Kong) · · cited 125× · DOI ↗ · more like this ·

Near-duplicate video retrieval is becoming more and more important with the exponential growth of the Web. Though various approaches have been proposed to address this problem, they are mainly focusing on the retrieval accuracy while infeasible to query on Web scale video database in real time. This paper proposes a novel method to address the efficiency and scalability issues for near-duplicate We video retrieval. We introduce a compact spatiotemporal feature to represent videos and construct an efficient data structure to index the feature to achieve real-time retrieving performance. This novel feature leverages relative gray-level intensity distribution within a frame and temporal structure of videos along frame sequence. The new index structure is proposed based on inverted file to allow for fast histogram intersection computation between videos. To demonstrate the effectiveness and…

VideoSense

Tao Mei, Xian‐Sheng Hua, Linjun Yang +1 more

Computer Vision and Pattern Recognition · · cited 115× · DOI ↗ · more like this ·

With Internet delivery of video content surging to an unprecedented level, online video advertising is becoming increasingly pervasive. In this paper, we present a novel advertising system for online video service called VideoSense, which automatically associates the most relevant video ads with online videos and seamlessly inserts the ads at the most appropriate positions within each individual video. Unlike most current video-oriented sites that only display a video ad at the beginning or the end of a video, VideoSense aims to embed more contextually relevant ads at less intrusive positions within the video stream. Given an online video, VideoSense is able to detect a set of candidate ad insertion points based on content discontinuity and attractiveness, select a list of relevant candidate ads ranked according to global textual relevance, and compute local visual-aural relevance…

Clickage

Xian‐Sheng Hua, Linjun Yang, Jingdong Wang +5 more

Computer Vision and Pattern Recognition · · cited 101× · DOI ↗ · more like this ·

The semantic gap between low-level visual features and high-level semantics has been investigated for decades but still remains a big challenge in multimedia. When "search" became one of the most frequently used applications, "intent gap", the gap between query expressions and users' search intents, emerged. Researchers have been focusing on three approaches to bridge the semantic and intent gaps: 1) developing more representative features, 2) exploiting better learning approaches or statistical models to represent the semantics, and 3) collecting more training data with better quality. However, it remains a challenge to close the gaps. In this paper, we argue that the massive amount of click data from commercial search engines provides a data set that is unique in the bridging of the semantic and intent gap. Search engines generate millions of click data (a.k.a. image-query pairs)…

Supervised reranking for web image search

Linjun Yang, Alan Hanjalić

Computer Vision and Pattern Recognition · · cited 86× · DOI ↗ · more like this ·

Visual search reranking that aims to improve the text-based image search with the help from visual content analysis has rapidly grown into a hot research topic. The interestingness of the topic stems mainly from the fact that the search reranking is an unsupervised process and therefore has the potential to scale better than its main alternative, namely the search based on offline-learned semantic concepts. However, the unsupervised nature of the reranking paradigm also makes it suffer from problems, the main of which can be identified as the difficulty to optimally determine the role of visual modality over different application scenarios. Inspired by the success of the "learning-to-rank" idea proposed in the field of information retrieval, we propose in this paper the "learning-to-rerank" paradigm, which derives the reranking function in a supervised fashion from the human-labeled…

Inference with Reference: Lossless Acceleration of Large Language Models

Nan Yang, Tao Ge, Liang Wang +5 more

cs.CL · · cited 83× · arXiv ↗ · more like this ·

We propose LLMA, an LLM accelerator to losslessly speed up Large Language Model (LLM) inference with references. LLMA is motivated by the observation that there are abundant identical text spans between the decoding result by an LLM and the reference that is available in many real world scenarios (e.g., retrieved documents). LLMA first selects a text span from the reference and copies its tokens to the decoder and then efficiently checks the tokens' appropriateness as the decoding result in parallel within one decoding step. The improved computational parallelism allows LLMA to achieve over 2x speed-up for LLMs with identical generation results as greedy decoding in many practical generation scenarios where significant overlap between in-context reference and outputs exists (e.g., search engines and multi-turn conversations).

GPS Estimation for Places of Interest From Social Users' Uploaded Photos

Jing Li, Xueming Qian, Yuan Yan Tang +2 more

Computer Vision and Pattern Recognition · IEEE Transactions on Multimedia · · cited 73× · DOI ↗ · more like this ·

Social media has become a very popular way for people to share their photos with friends. Because most of the social images are attached with GPS (geo-tags), a photo's GPS information can be estimated with the help of the large geo-tagged image set while using a visual searching based approach. This paper proposes an unsupervised image GPS location estimation approach with hierarchical global feature clustering and local feature refinement. It consists of two parts: an offline system and an online system. In the offline system, a hierarchical structure is constructed for a large-scale offline social image set with GPS information. Representative images are selected for each GPS location refined cluster, and an inverted file structure is proposed. In the online system, when given an input image, its GPS information can be estimated by hierarchical global clusters selection and local…

Web-Scale Responsive Visual Search at Bing

Houdong Hu, Yan Wang, Linjun Yang +7 more

cs.CV · · cited 67× · arXiv ↗ · more like this ·

In this paper, we introduce a web-scale general visual search system deployed in Microsoft Bing. The system accommodates tens of billions of images in the index, with thousands of features for each image, and can respond in less than 200 ms. In order to overcome the challenges in relevance, latency, and scalability in such large scale of data, we employ a cascaded learning-to-rank framework based on various latest deep learning visual features, and deploy in a distributed heterogeneous computing platform. Quantitative and qualitative experiments show that our system is able to support various applications on Bing website and apps.

Multi-modality web video categorization

Linjun Yang, Jiemin Liu, Xiaokang Yang +1 more

Computer Vision and Pattern Recognition · · cited 67× · DOI ↗ · more like this ·

This paper reports a first comprehensive study and large-scale test on web video (so-called user generated video or micro video) categorization. Observing that web videos are characterized by a much higher diversity of quality, subject, style, and genres compared with traditional video programs, we focus on studying the effectiveness of different modalities in dealing with such high variation. Specifically, we propose two novel modalities including a semantic modality and a surrounding text modality, as effective complements to most commonly used low-level features. The semantic modality includes three feature representations, i.e., concept histogram, visual word vector model and visual word Latent Semantic Analysis (LSA), while text modality includes the titles, descriptions and tags of web videos. We conduct a set of comprehensive experiments for evaluating the effectiveness of the…

MSRA-MM: Bridging Research and Industrial Societies for Multimedia Information Retrieval

Meng Wang, Linjun Yang, Xian‐Sheng Hua

Computer Vision and Pattern Recognition · · cited 66× · OpenAlex ↗ · more like this ·

This report introduces a dataset named Microsoft Research Asia Multimedia (MSRA-MM)1 that aims to en-courage research in multimedia information retrieval and the related areas. The images and videos in the dataset are collected from Microsoft Live Search and the performance of state-of-the-art industrial techniques can be evaluated accordingly. Therefore, conducting research using the dataset is able to demonstrate the practical usefulness of the

The role of attractiveness in web image search

Bo Geng, Linjun Yang, Chao Xu +2 more

Computer Vision and Pattern Recognition · · cited 55× · DOI ↗ · more like this ·

Existing web image search engines are mainly designed to optimize topical relevance. However, according to our user study, attractiveness is becoming a more and more important factor for web image search engines to satisfy users' search intentions. Important as it can be, web image attractiveness from the search users' perspective has not been sufficiently recognized in both the industry and the academia. In this paper, we present a definition of web image attractiveness with three levels according to the end users' feedback, including perceptual quality, aesthetic sensitivity and affective tune. Corresponding to each level of the definition, various visual features are investigated on their applicability to attractiveness estimation of web images. To further deal with the unreliability of visual features induced by the large variations of web images, we propose a contextual approach to…

Million-scale near-duplicate video retrieval system

Yang Cai, Linjun Yang, Wei Ping +4 more

Computer Vision and Pattern Recognition · · cited 55× · DOI ↗ · more like this ·

In this paper, we present a novel near-duplicate video retrieval system serving one million web videos. To achieve both the effectiveness and efficiency, a visual word based approach is proposed, which quantizes each video frame into a word and represents the whole video as a bag of words. The system can respond to a query in 41ms with 78.4% MAP on average.

Ranking model adaptation for domain-specific search

Bo Geng, Linjun Yang, Chao Xu +1 more

Artificial Intelligence · · cited 49× · DOI ↗ · more like this ·

Recently, various domain-specific search engines emerge, which are restricted to specific topicalities or document formats, and vertical to the broad-based search. Simply applying the ranking model trained for the broad-based search to the verticals cannot achieve a sound performance due to the domain differences, while building different ranking models for each domain is both laborious for labeling sufficient training samples and time-consuming or the training process. In this paper, to address the above difficulties, we investigate two problems: (1) whether we can adapt the ranking model learned for existing Web page search or verticals, to the new domain, so that the amount of labeled data and the training cost is reduced, while the performance requirement is still satisfied; and (2) how to adapt the ranking model from auxiliary domains to a new target domain. We address the second…