arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Rui Men

46 papers · 2009–2026

2009: 12010: 02011: 02012: 02013: 02014: 02015: 22016: 02017: 02018: 02019: 22020: 22021: 52022: 32023: 22024: 72025: 112026: 11
2009peak 11 in 20252026

Frequent co-authors

Papers

showing 1–25 of 46 RSS

Qwen3 Technical Report

An Yang, Anfeng Li, Baosong Yang +57 more

cs.CL · · cited 8,634× · arXiv ↗ · more like this ·

In this work, we present Qwen3, the latest version of the Qwen model family. Qwen3 comprises a series of large language models (LLMs) designed to advance performance, efficiency, and multilingual capabilities. The Qwen3 series includes models of both dense and Mixture-of-Expert (MoE) architectures, with parameter scales ranging from 0.6 to 235 billion. A key innovation in Qwen3 is the integration of thinking mode (for complex, multi-step reasoning) and non-thinking mode (for rapid, context-driven responses) into a unified framework. This eliminates the need to switch between different models--such as chat-optimized models (e.g., GPT-4o) and dedicated reasoning models (e.g., QwQ-32B)--and enables dynamic mode switching based on user queries or chat templates. Meanwhile, Qwen3 introduces a thinking budget mechanism, allowing users to allocate computational resources adaptively during…

Qwen2.5 Technical Report

Qwen, :, An Yang +41 more

cs.CL · · cited 5,256× · arXiv ↗ · more like this ·

In this report, we introduce Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. Compared to previous iterations, Qwen 2.5 has been significantly improved during both the pre-training and post-training stages. In terms of pre-training, we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens. This provides a strong foundation for common sense, expert knowledge, and reasoning capabilities. In terms of post-training, we implement intricate supervised finetuning with over 1 million samples, as well as multistage reinforcement learning. Post-training techniques enhance human preference, and notably improve long text generation, structural data analysis, and instruction following. To handle diverse and varied use cases effectively, we present Qwen2.5 LLM series in rich sizes. Open-weight…

Qwen Technical Report

Jinze Bai, Shuai Bai, Yunfei Chu +45 more

cs.CL · · cited 4,377× · arXiv ↗ · more like this ·

Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans. In this work, we introduce Qwen, the first installment of our large language model series. Qwen is a comprehensive language model series that encompasses distinct models with varying parameter counts. It includes Qwen, the base pretrained language models, and Qwen-Chat, the chat models finetuned with human alignment techniques. The base language models consistently demonstrate superior performance across a multitude of downstream tasks, and the chat models, particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are highly competitive. The chat models possess advanced tool-use and planning capabilities for creating agent applications, showcasing impressive performance even…

Qwen2 Technical Report

An Yang, Baosong Yang, Binyuan Hui +59 more

cs.CL · · cited 2,519× · arXiv ↗ · more like this ·

This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models. We release a comprehensive suite of foundational and instruction-tuned language models, encompassing a parameter range from 0.5 to 72 billion, featuring dense models and a Mixture-of-Experts model. Qwen2 surpasses most prior open-weight models, including its predecessor Qwen1.5, and exhibits competitive performance relative to proprietary models across diverse benchmarks on language understanding, generation, multilingual proficiency, coding, mathematics, and reasoning. The flagship model, Qwen2-72B, showcases remarkable performance: 84.2 on MMLU, 37.9 on GPQA, 64.6 on HumanEval, 89.5 on GSM8K, and 82.4 on BBH as a base language model. The instruction-tuned variant, Qwen2-72B-Instruct, attains 9.1 on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench. Moreover…

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Peng Wang, An Yang, Rui Men +7 more

cs.CV · · cited 1,112× · arXiv ↗ · more like this ·

In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for downstream tasks. In comparison with the recent state-of-the-art vision & language models that rely on extremely large cross-modal datasets, OFA is pretrained on only 20M publicly available image-text pairs. Despite its simplicity and relatively small-scale training data, OFA achieves new SOTAs in a…

Natural Language Inference by Tree-Based Convolution and Heuristic Matching

Lili Mou, Rui Men, Ge Li +4 more

cs.CL · · cited 371× · arXiv ↗ · more like this ·

In this paper, we propose the TBCNN-pair model to recognize entailment and contradiction between two sentences. In our model, a tree-based convolutional neural network (TBCNN) captures sentence-level semantics; then heuristic matching layers like concatenation, element-wise product/difference combine the information in individual sentences. Experimental results show that our model outperforms existing sentence encoding-based approaches by a large margin.

Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

An Yang, Junshu Pan, Junyang Lin +4 more

cs.CV · · cited 224× · arXiv ↗ · more like this ·

The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive…

M6: A Chinese Multimodal Pretrainer

Junyang Lin, Rui Men, An Yang +22 more

cs.CL · · cited 151× · arXiv ↗ · more like this ·

In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 billion and 100 billion parameters, and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding performance in comparison with strong baselines. Furthermore, we specifically design a downstream task of text-guided image generation, and show that the finetuned M6 can create high-quality images with high resolution and abundant details.

On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Zihan Qiu, Zekun Wang, Xiao Li +33 more

cs.CL · · cited 56× · arXiv ↗ · more like this ·

We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and additional 51B parameters of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs. Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global attention, with one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call the Gated Residual…

M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers

Zhu Zhang, Jianxin Ma, Chang Zhou +6 more

cs.CV · · cited 49× · arXiv ↗ · more like this ·

Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, M6-UFC, to unify any number of multi-modal controls. In M6-UFC, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches such as DALL-E and VQGAN, M6-UFC adopts non-autoregressive generation (NAR) at the second stage to enhance the holistic consistency of the synthesized image, to support preserving specified image blocks, and to improve the synthesis speed. Further, we design a progressive algorithm that iteratively…

On End-to-End Program Generation from User Intention by Deep Neural Networks

Lili Mou, Rui Men, Ge Li +2 more

cs.SE · · cited 48× · arXiv ↗ · more like this ·

This paper envisions an end-to-end program generation scenario using recurrent neural networks (RNNs): Users can express their intention in natural language; an RNN then automatically generates corresponding code in a characterby-by-character fashion. We demonstrate its feasibility through a case study and empirical analysis. To fully make such technique useful in practice, we also point out several cross-disciplinary challenges, including modeling user intention, providing datasets, improving model architectures, etc. Although much long-term research shall be addressed in this new field, we believe end-to-end program generation would become a reality in future decades, and we are looking forward to its practice.

Learning Relation Alignment for Calibrated Cross-modal Retrieval

Shuhuai Ren, Junyang Lin, Guangxiang Zhao +5 more

cs.CL · · cited 39× · arXiv ↗ · more like this ·

Despite the achievements of large-scale multimodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the visual relation among the regions. The neglect of such relation consistency impairs the contextualized representation of image-text pairs and hinders the model performance and the interpretability. In this paper, we first propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. In response, we present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD…

M6: Multi-Modality-to-Multi-Modality Multitask Mega-transformer for Unified Pretraining

Junyang Lin, Rui Men, Yang An +6 more

Computer Vision and Pattern Recognition · · cited 30× · DOI ↗ · more like this ·

Multimodal pretraining has demonstrated success in the downstream tasks of cross-modal representation learning. However, it is limited to the English data, and there is still a lack of large-scale dataset for multimodal pretraining in Chinese. In this work, we propose the largest dataset for pretraining in Chinese, which consists of over 1.9TB images and 292GB texts. The dataset has large coverage over domains, including encyclopedia, question answering, forum discussion, etc. Besides, we propose a method called M6, referring to Multi-Modality-to-Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. The model is pretrained with our proposed tasks, including text-to-text transfer, image-to-text transfer, as well as multi-modality-to-text transfer. The tasks endow the model with strong capability of understanding and…

M6-T: Exploring Sparse Expert Models and Beyond

An Yang, Junyang Lin, Rui Men +12 more

cs.LG · · cited 24× · arXiv ↗ · more like this ·

Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost, and thus it has become a trend in model scaling. Still it is a mystery how MoE layers bring quality gains by leveraging the parameters with sparse activation. In this work, we investigate several key factors in sparse expert models. We observe that load imbalance may not be a significant problem affecting model quality, contrary to the perspectives of recent studies, while the number of sparsely activated experts $k$ and expert capacity $C$ in top-$k$ routing can significantly make a difference in this context. Furthermore, we take a step forward to propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational…

Hierarchical Aerial Computing for Task Offloading and Resource Allocation in 6G-Enabled Vehicular Networks

Rui Men, Xiumei Fan, Kok‐Lim Alvin Yau +2 more

Computer Networks and Communications · IEEE Transactions on Network Science and Engineering · · cited 12× · DOI ↗ · more like this ·

The upsurge of mobile service demands underscores the inevitability of the emergence of 6G technology. Amidst the challenges that accompany the 6G networks, the exponential growth of computation-intensive and delay-sensitive applications in vehicular networks poses a significant hurdle. Vehicular edge computing relying solely on terrestrial infrastructures cannot meet the performance requirements due to insufficient resources in edge servers. To address this challenge, we introduce a novel three-layer hierarchical architecture called the air-ground-end aerial computing (AGEAC) architecture, which consists of three types of resource providers, namely high-altitude platform stations, roadside units, and volunteer vehicles willing to share their underutilized resources. We select service vehicles based on the utilization rate of their underutilized resources to ensure fairness. Our…

Dynamic Selfish Node Detection With Link Quality Consideration in Vehicular Networks

Axida Shan, Xiumei Fan, Celimuge Wu +2 more

Computer Networks and Communications · IEEE Transactions on Vehicular Technology · · cited 12× · DOI ↗ · more like this ·

With the emergence of various applications in intelligent transportation systems (ITS), the cooperative communication among the nodes on roads is becoming more and more important in vehicular networks. However, selfish nodes, which are not willing to relay data packets for others, drastically decrease the communication performance of the network in various aspects including packet delivery ratio, end-to-end delay, communication bandwidth, etc. It is critical to detect the selfish nodes in the network for avoiding the decrement of network performance. Nevertheless, it is extremely challenging to correctly and fairly detect selfish nodes in vehicular networks where the network topology is highly dynamic and link quality between nodes is lossy. In order to address this issue, a novel detection scheme is proposed in this paper. The proposed scheme jointly considers the link quality…

Deep-AutoCoder: Learning to Complete Code Precisely with Induced Code Tokens

Xing Hu, Rui Men, Ge Li +1 more

Information Systems · · cited 9× · DOI ↗ · more like this ·

Code completion is an essential part of modern IDEs. It assists the developers to speed up the process of coding and reducing typos. In this paper, we exploit the deep learning technique called LSTM to learn language models over large code corpus and make predictions of code elements. Unlike natural language, the innumerable identifiers lead to the vocabulary explosion and more difficult to predict. Therefore, we propose a new approach, the Induced Token based LSTM, to deal with the massive identifiers, thus decrease the vocabulary size. In order to induce the code tokens, we present two approaches, one is a constraint character-level LSTM and the other one is encoding identifiers with various preceding context before feeding them into a token-level LSTM. Based on the two approaches, a tool named Deep-AutoCoder is developed and evaluated in two classic completion scenarios, that is…

Research on access control method of Digital Archives based on blockchain

Rui Men

Information Systems · Journal of Physics Conference Series · · cited 4× · DOI ↗ · more like this ·

Abstract This paper analyzes the security problems that need to be solved urgently in digital archives system: First, the centralized storage of digital archives makes it overwhelmingly dependent on the third-party storage organization. Second, every transaction has its own access proof and its own authorization proof, it’s not transparent to others. Then the blockchain technology is introduced to solve the existing problems. We store the digital file attachment and file attributes in the private IPFS cluster after encryption. The address of IPFS and digital fingerprint of the file are stored on the private blockchain. Additionally, this paper stores and checks the height and hash value of the latest irreversible block of the private blockchain, to guarantee the authenticity of data on private blockchain. To solve the second problem, we store the ABAC on public blockchain, add time and…

The research on intelligent RGV dynamic scheduling based on hybrid genetic algorithm

Likang Wang, Yaya Mu, Hongmei Gao +1 more

Artificial Intelligence · Journal of Physics Conference Series · · cited 4× · DOI ↗ · more like this ·

Abstract With the rapid development of automation, mechatronics, information technology and other fields, production and processing systems are constantly moving towards unmanned and intelligent development. The RGV rail-guided vehicle system integrates various high-new technologies and is widely used in modern processing systems. Compared with the circular RGV orbit system, the linear reciprocating RGV system adopts a linear orbit, which is not easy to be blocked during operation, but its scheduling response mechanism is not perfect enough, which makes the operation efficiency of the whole system low. Therefore, how to improve the response mechanism and the operational efficiency of the RGV intelligent processing system has become an important factor in prompting the rapid development of the modern intelligent processing industry. In this paper, aiming at the intelligent linear…

Enhancing Large-Scale AI Training Efficiency: The C4 Solution for Real-Time Anomaly Detection and Communication Optimization

Jianbo Dong, Bin Luo, Jun Zhang +22 more

cs.DC · · cited 3× · arXiv ↗ · more like this ·

The emergence of Large Language Models (LLMs) has necessitated the adoption of distributed training techniques, involving the deployment of thousands of GPUs to train a single model. Unfortunately, the efficiency of large-scale distributed training systems is often suboptimal due to the increased likelihood of hardware errors in high-end GPU products and the heightened risk of network traffic collisions. Moreover, any local hardware failure can disrupt training tasks, and the inability to swiftly identify faulty components leads to a significant waste of GPU resources. And, prolonged communication due to traffic collisions can substantially increase GPU waiting times. To address these challenges, we propose a communication-driven solution, namely the C4. The key insights of C4 are twofold. First, the load in distributed training exhibits homogeneous characteristics and is divided into…

OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models

Jinze Bai, Rui Men, Hao Yang +15 more

cs.CV · · cited 3× · arXiv ↗ · more like this ·

Generalist models, which are capable of performing diverse multi-modal tasks in a task-agnostic way within a single model, have been explored recently. Being, hopefully, an alternative to approaching general-purpose AI, existing generalist models are still at an early stage, where modality and task coverage is limited. To empower multi-modal task-scaling and speed up this line of research, we release a generalist model learning system, OFASys, built on top of a declarative task interface named multi-modal instruction. At the core of OFASys is the idea of decoupling multi-modal task representations from the underlying model implementations. In OFASys, a task involving multiple modalities can be defined declaratively even with just a single line of code. The system automatically generates task plans from such instructions for training and inference. It also facilitates multi-task training…

ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control

Peng Jin, Zihan Qiu, Zekun Wang +8 more

cs.LG · · cited 2× · arXiv ↗ · more like this ·

Scaling Large Language Models (LLMs) via Mixture-of-Experts (MoE) enables massive parameter growth with nearly constant per-token computation. However, further scaling the parameter count requires increasingly sparse routing, where expert load imbalance becomes more severe. This imbalance reduces parameter utilization and training efficiency, and can undermine training stability, becoming a bottleneck to reliable scaling. In this work, we unify two representative auxiliary-loss-free methods as incomplete Proportional-Integral-Derivative (PID) controllers: DeepSeek's loss-free method acts as a fixed-step integral controller, while Kimi K3's Quantile Balancing functions as a generalized proportional controller. Building on this control perspective, we propose ID Balancing, an Integral-Derivative controller. It scales its integral term with load error and activates its derivative term only…

Qwen3-Omni Technical Report

Jin Xu, Zhifang Guo, Hangrui Hu +35 more

cs.CL · · cited 1× · arXiv ↗ · more like this ·

We present Qwen3-Omni, a single multimodal model that, for the first time, maintains state-of-the-art performance across text, image, audio, and video without any degradation relative to single-modal counterparts. Qwen3-Omni matches the performance of same-sized single-modal models within the Qwen series and excels particularly on audio tasks. Across 36 audio and audio-visual benchmarks, Qwen3-Omni achieves open-source SOTA on 32 benchmarks and overall SOTA on 22, outperforming strong closed-source models such as Gemini-2.5-Pro, Seed-ASR, and GPT-4o-Transcribe. Qwen3-Omni adopts a Thinker-Talker MoE architecture that unifies perception and generation across text, images, audio, and video, yielding fluent text and natural real-time speech. It supports text interaction in 119 languages, speech understanding in 19 languages, and speech generation in 10 languages. To reduce first-packet…

HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning

Xiaoyuan Li, Moxin Li, Rui Men +6 more

cs.CL · · cited 1× · arXiv ↗ · more like this ·

Large language models (LLMs) have shown remarkable capabilities in commonsense reasoning; however, some variations in questions can trigger incorrect responses. Do these models truly understand commonsense knowledge, or just memorize expression patterns? To investigate this question, we present the first extensive robustness evaluation of LLMs in commonsense reasoning. We introduce HellaSwag-Pro, a large-scale bilingual benchmark consisting of 11,200 cases, by designing and compiling seven types of question variants. To construct this benchmark, we propose a two-stage method to develop Chinese HellaSwag, a finely annotated dataset comprising 12,000 instances across 56 categories. We conduct extensive experiments on 41 representative LLMs, revealing that these LLMs are far from robust in commonsense reasoning. Furthermore, this robustness varies depending on the language in which the LLM…

A communication link lifetime prediction-supported V2V partial computation offloading scheme for autonomous driving

Rui Men, Xiumei Fan, Jun Yan +2 more

Computer Networks and Communications · Journal of Intelligent & Fuzzy Systems · · cited 1× · DOI ↗ · more like this ·

Vehicle Edge Computing (VEC) is a promising technique to improve the quality of service (QoS) and quality of experience (QoE) in autonomous driving by exploiting the resources at the network edge. However, the high mobility of the vehicles leads to stochastic communication link duration, and the tasks generated by various applications in autonomous driving incur fierce competition for resources. These challenges cause excessive task completion delays. In this paper, we propose a vehicle-to-vehicle (V2V) partial computation offloading scheme that leverages the prediction results of the communication link lifetime between vehicles. A History track, Current interactions and Future planning trajectory-aware Gated Recurrent Units (HCF-GRU) network is built to capture the essential factors to improve the prediction accuracy. Then, we design a GRU-based Proximal Policy Optimization (GRU-PPO)…