arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Zeqi Lin

37 papers · 2014–2026

2014: 22015: 02016: 02017: 22018: 12019: 12020: 52021: 42022: 72023: 52024: 82025: 12026: 1
2014peak 8 in 20242026

Frequent co-authors

Papers

showing 1–25 of 37 RSS

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Marah Abdin, Jyoti Aneja, Hany Awadalla +126 more

cs.CL · · cited 2,528× · arXiv ↗ · more like this ·

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce…

CodeT: Code Generation with Generated Tests

Bei Chen, Fengji Zhang, Anh Nguyen +4 more

cs.CL · · cited 630× · arXiv ↗ · more like this ·

The task of generating code solutions for a given programming problem can benefit from the use of pre-trained language models such as Codex, which can produce multiple diverse samples. However, a major challenge for this task is to select the most appropriate solution from the multiple samples generated by the pre-trained language models. A natural way to evaluate the quality and correctness of a code solution is to run it against a set of test cases, but the manual creation of such test cases is often costly and time-consuming. In this paper, we propose a novel method, CodeT, that leverages the same pre-trained language models to automatically generate test cases for the code samples, thus reducing the human effort and increasing the coverage of the test scenarios. CodeT then executes the code samples using the generated test cases, and performs a dual execution agreement, which…

TAPEX: Table Pre-training via Learning a Neural SQL Executor

Qian Liu, Bei Chen, Jiaqi Guo +4 more

cs.CL · · cited 393× · arXiv ↗ · more like this ·

Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This…

Making Large Language Models Better Reasoners with Step-Aware Verifier

Yifei Li, Zeqi Lin, Shizhuo Zhang +4 more

cs.CL · · cited 349× · arXiv ↗ · more like this ·

Few-shot learning is a challenging task that requires language models to generalize from limited examples. Large language models like GPT-3 and PaLM have made impressive progress in this area, but they still face difficulties in reasoning tasks such as GSM8K, a benchmark for arithmetic problems. To improve their reasoning skills, previous work has proposed to guide the language model with prompts that elicit a series of reasoning steps before giving the final answer, achieving a significant improvement on GSM8K from 17.9% to 58.1% in problem-solving rate. In this paper, we present DIVERSE (Diverse Verifier on Reasoning Step), a novel approach that further enhances the reasoning capability of language models. DIVERSE has three main components: first, it generates diverse prompts to explore different reasoning paths for the same question; second, it uses a verifier to filter out incorrect…

Make Your LLM Fully Utilize the Context

Shengnan An, Zexiong Ma, Zeqi Lin +2 more

cs.CL · · cited 212× · arXiv ↗ · more like this ·

While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the lost-in-the-middle challenge. We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information. Based on this intuition, our study presents information-intensive (IN2) training, a purely data-driven solution to overcome lost-in-the-middle. Specifically, IN2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) fine-grained information awareness on a short segment (~128 tokens) within a synthesized long context (4K-32K tokens), and (2) the integration and reasoning of information from two or more short segments. Through applying this…

CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation

Daoguang Zan, Bei Chen, Dejian Yang +6 more

cs.SE · · cited 167× · arXiv ↗ · more like this ·

Code generation is a longstanding challenge, aiming to generate a code snippet based on a natural language description. Usually, expensive text-code paired data is essential for training a code generation model. Recently, thanks to the success of pre-training techniques, large language models are trained on large-scale unlabelled code corpora and perform well in code generation. In this paper, we investigate how to leverage an unlabelled code corpus to train a model for library-oriented code generation. Since it is a common practice for programmers to reuse third-party libraries, in which case the text-code paired data are harder to obtain due to the huge number of libraries. We observe that library-oriented code snippets are more likely to share similar code sketches. Hence, we present CERT with two steps: a sketcher generates the sketch, then a generator fills the details in the…

When Language Model Meets Private Library

Daoguang Zan, Bei Chen, Zeqi Lin +3 more

cs.PL · · cited 106× · arXiv ↗ · more like this ·

With the rapid development of pre-training techniques, a number of language models have been pre-trained on large-scale code corpora and perform well in code generation. In this paper, we investigate how to equip pre-trained language models with the ability of code generation for private libraries. In practice, it is common for programmers to write code using private libraries. However, this is a challenge for language models since they have never seen private APIs during training. Motivated by the fact that private libraries usually come with elaborate API documentation, we propose a novel framework with two modules: the APIRetriever finds useful APIs, and then the APICoder generates code using these APIs. For APIRetriever, we present a dense retrieval system and also design a friendly interaction to involve uses. For APICoder, we can directly use off-the-shelf language models, or…

How Do In-Context Examples Affect Compositional Generalization?

Shengnan An, Zeqi Lin, Qiang Fu +4 more

cs.CL · · cited 79× · arXiv ↗ · more like this ·

Compositional generalization--understanding unseen combinations of seen primitives--is an essential reasoning capability in human intelligence. The AI community mainly studies this capability by fine-tuning neural networks on lots of training samples, while it is still unclear whether and how in-context learning--the prevailing few-shot paradigm based on large language models--exhibits compositional generalization. In this paper, we present CoFe, a test suite to investigate in-context compositional generalization. We find that the compositional generalization performance can be easily affected by the selection of in-context examples, thus raising the research question what the key factors are to make good in-context examples for compositional generalization. We study three potential factors: similarity, diversity and complexity. Our systematic experiments indicate that in-context…

Compositional Generalization by Learning Analytical Expressions

Qian Liu, Shengnan An, Jian-Guang Lou +6 more

cs.AI · · cited 78× · arXiv ↗ · more like this ·

Compositional generalization is a basic and essential intellective capability of human beings, which allows us to recombine known parts readily. However, existing neural network based models have been proven to be extremely deficient in such a capability. Inspired by work in cognition which argues compositionality can be captured by variable slots with symbolic functions, we present a refreshing view that connects a memory-augmented neural model with analytical expressions, to achieve compositional generalization. Our model consists of two cooperative neural modules, Composer and Solver, fitting well with the cognitive argument while being able to be trained in an end-to-end manner via a hierarchical reinforcement learning algorithm. Experiments on the well-known benchmark SCAN demonstrate that our model seizes a great ability of compositional generalization, solving all challenges…

Skill-Based Few-Shot Selection for In-Context Learning

Shengnan An, Bo Zhou, Zeqi Lin +5 more

cs.CL · · cited 65× · arXiv ↗ · more like this ·

In-context learning is the paradigm that adapts large language models to downstream tasks by providing a few examples. Few-shot selection -- selecting appropriate examples for each test instance separately -- is important for in-context learning. In this paper, we propose Skill-KNN, a skill-based few-shot selection method for in-context learning. The key advantages of Skill-KNN include: (1) it addresses the problem that existing methods based on pre-trained embeddings can be easily biased by surface natural language features that are not important for the target task; (2) it does not require training or fine-tuning of any models, making it suitable for frequently expanding or changing example banks. The key insight is to optimize the inputs fed into the embedding model, rather than tuning the model itself. Technically, Skill-KNN generates the skill-based descriptions for each test case…

Reasoning Like Program Executors

Xinyu Pi, Qian Liu, Bei Chen +6 more

cs.CL · · cited 61× · arXiv ↗ · more like this ·

Reasoning over natural language is a long-standing goal for the research community. However, studies have shown that existing language models are inadequate in reasoning. To address the issue, we present POET, a novel reasoning pre-training paradigm. Through pre-training language models with programs and their execution results, POET empowers language models to harvest the reasoning knowledge possessed by program executors via a data-driven approach. POET is conceptually simple and can be instantiated by different kinds of program executors. In this paper, we showcase two simple instances POET-Math and POET-Logic, in addition to a complex instance, POET-SQL. Experimental results on six benchmarks demonstrate that POET can significantly boost model performance in natural language reasoning, such as numerical reasoning, logical reasoning, and multi-hop reasoning. POET opens a new gate on…

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

Shengnan An, Yifei Li, Zeqi Lin +6 more

cs.CL · · cited 47× · arXiv ↗ · more like this ·

Recently the prompt-tuning paradigm has attracted significant attention. By only tuning continuous prompts with a frozen pre-trained language model (PLM), prompt-tuning takes a step towards deploying a shared frozen PLM to serve numerous downstream tasks. Although prompt-tuning shows good performance on certain natural language understanding (NLU) tasks, its effectiveness on natural language generation (NLG) tasks is still under-explored. In this paper, we argue that one of the factors hindering the development of prompt-tuning on NLG tasks is the unfamiliar inputs (i.e., inputs are linguistically different from the pretraining corpus). For example, our preliminary exploration reveals a large performance gap between prompt-tuning and fine-tuning when unfamiliar inputs occur frequently in NLG tasks. This motivates us to propose input-tuning, which fine-tunes both the continuous prompts…

Adaptive Deep Code Search

Chunyang Ling, Zeqi Lin, Yanzhen Zou +1 more

Information Systems · · cited 42× · DOI ↗ · more like this ·

Searching code in a large-scale codebase using natural language queries is a common practice during software development. Deep learning-based code search methods demonstrate superior performance if models are trained with large amount of text-code pairs. However, few deep code search models can be easily transferred from one codebase to another. It can be very costly to prepare training data for a new codebase and re-train an appropriate deep learning model. In this paper, we propose AdaCS, an adaptive deep code search method that can be trained once and transferred to new codebases. AdaCS decomposes the learning process into embedding domain-specific words and matching general syntactic patterns. Firstly, an unsupervised word embedding technique is used to construct a matching matrix to represent the lexical similarities. Then, a recurrent neural network is used to capture latent…

Compositional API Recommendation for Library-Oriented Code Generation

Zexiong Ma, Shengnan An, Bing Xie +1 more

cs.SE · · cited 40× · arXiv ↗ · more like this ·

Large language models (LLMs) have achieved exceptional performance in code generation. However, the performance remains unsatisfactory in generating library-oriented code, especially for the libraries not present in the training data of LLMs. Previous work utilizes API recommendation technology to help LLMs use libraries: it retrieves APIs related to the user requirements, then leverages them as context to prompt LLMs. However, developmental requirements can be coarse-grained, requiring a combination of multiple fine-grained APIs. This granularity inconsistency makes API recommendation a challenging task. To address this, we propose CAPIR (Compositional API Recommendation), which adopts a "divide-and-conquer" strategy to recommend APIs for coarse-grained requirements. Specifically, CAPIR employs an LLM-based Decomposer to break down a coarse-grained task description into several…

Learning Algebraic Recombination for Compositional Generalization

Chenyao Liu, Shengnan An, Zeqi Lin +6 more

cs.CL · · cited 38× · arXiv ↗ · more like this ·

Neural sequence models exhibit limited compositional generalization ability in semantic parsing tasks. Compositional generalization requires algebraic recombination, i.e., dynamically recombining structured expressions in a recursive manner. However, most previous studies mainly concentrate on recombining lexical units, which is an important but not sufficient part of algebraic recombination. In this paper, we propose LeAR, an end-to-end neural model to learn algebraic recombination for compositional generalization. The key insight is to model the semantic parsing task as a homomorphism between a latent syntactic algebra and a semantic algebra, thus encouraging algebraic recombination. Specifically, we learn two modules jointly: a Composer for producing latent syntax, and an Interpreter for assigning semantic operations. Experiments on two realistic and comprehensive compositional…

Hierarchical Poset Decoding for Compositional Generalization in Language

Yinuo Guo, Zeqi Lin, Jian-Guang Lou +1 more

cs.CL · · cited 34× · arXiv ↗ · more like this ·

We formalize human language understanding as a structured prediction task where the output is a partially ordered set (poset). Current encoder-decoder architectures do not take the poset structure of semantics into account properly, thus suffering from poor compositional generalization ability. In this paper, we propose a novel hierarchical poset decoding paradigm for compositional generalization in language. Intuitively: (1) the proposed paradigm enforces partial permutation invariance in semantics, thus avoiding overfitting to bias ordering information; (2) the hierarchical mechanism allows to capture high-level structures of posets. We evaluate our proposed decoder on Compositional Freebase Questions (CFQ), a large and realistic natural language question answering dataset that is specifically designed to measure compositional generalization. Results show that it outperforms current…

Mining API Usage Examples from Test Code

Zixiao Zhu, Yanzhen Zou, Bing Xie +3 more

Information Systems · · cited 31× · DOI ↗ · more like this ·

Lack of effective usage examples in API documents has been proven to be a great obstacle to API learning. To deal with this issue, several approaches have been proposed to automatically extract usage examples from client code or related web pages, which are unfortunately not available for newly released API libraries. In this paper, we propose a novel approach to mining API usage examples from test code. Although test code can be a good source of usage examples, the issue of multiple test scenarios might lead to repetitive and interdependent API usages in a test method, which make it complicated and difficult to extract API usage examples. To address this issue, we study the JUnit test code and summarize a set of test code patterns. We employ a code pattern based heuristic slicing approach to separate test scenarios into code examples. Then we cluster the similar usage examples for…

Revisiting Iterative Back-Translation from the Perspective of Compositional Generalization

Yinuo Guo, Hualei Zhu, Zeqi Lin +3 more

cs.CL · · cited 29× · arXiv ↗ · more like this ·

Human intelligence exhibits compositional generalization (i.e., the capacity to understand and produce unseen combinations of seen components), but current neural seq2seq models lack such ability. In this paper, we revisit iterative back-translation, a simple yet effective semi-supervised method, to investigate whether and how it can improve compositional generalization. In this work: (1) We first empirically show that iterative back-translation substantially improves the performance on compositional generalization benchmarks (CFQ and SCAN). (2) To understand why iterative back-translation is useful, we carefully examine the performance gains and find that iterative back-translation can increasingly correct errors in pseudo-parallel data. (3) To further encourage this mechanism, we propose curriculum iterative back-translation, which better improves the quality of pseudo-parallel data…

CoRA: Decomposing and Describing Tangled Code Changes for Reviewer

Min Wang, Zeqi Lin, Yanzhen Zou +1 more

Information Systems · · cited 29× · DOI ↗ · more like this ·

Code review is an important mechanism for code quality assurance both in open source software and industrial software. Reviewers usually suffer from numerous, tangled and loosely related code changes that are bundled in a single commit, which makes code review very difficult. In this paper, we propose CoRA (Code Review Assistant), an automatic approach to decompose a commit into different parts and generate concise descriptions for reviewers. More specifically, CoRA can decompose a commit into independent parts (e.g., bug fixing, new feature adding, or refactoring) by code dependency analysis and tree-based similar-code detection, then identify the most important code changes in each part based on the PageRank algorithm and heuristic rules. As a result, CoRA can generate a concise description for each part of the commit. We evaluate our approach in seven open source software projects…

Improving software text retrieval using conceptual knowledge in source code

Zeqi Lin, Yanzhen Zou, Junfeng Zhao +1 more

Information Systems · · cited 22× · DOI ↗ · more like this ·

A large software project usually has lots of various textual learning resources about its API, such as tutorials, mailing lists, user forums, etc. Text retrieval technology allows developers to search these API learning resources for related documents using free-text queries, but it suffers from the lexical gap between search queries and documents. In this paper, we propose a novel approach for improving the retrieval of API learning resources through leveraging software-specific conceptual knowledge in software source code. The basic idea behind this approach is that the semantic relatedness between queries and documents could be measured according to software-specific concepts involved in them, and software source code contains a large amount of software-specific conceptual knowledge. In detail, firstly we extract an API graph from software source code and use it as software-specific…

Can Neural Clone Detection Generalize to Unseen Functionalitiesƒ

Chenyao Liu, Zeqi Lin, Jian–Guang Lou +2 more

Information Systems · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · · cited 13× · DOI ↗ · more like this ·

Many recently proposed code clone detectors exploit neural networks to capture latent semantics of source code, thus achieving impressive results for detecting semantic clones. These neural clone detectors rely on the availability of large amounts of labeled training data. We identify a key oversight in the current evaluation methodology for neural clone detection: cross-functionality generalization (i.e., detecting semantic clones of which the functionalities are unseen in training). Specifically, we focus on this question: do neural clone detectors truly learn the ability to detect semantic clones, or they just learn how to model specific functionalities in training data while cannot generalize to realistic unseen functionalitiesƒ This paper investigates how the generalizability can be evaluated and improved.Our contributions are 3-folds: (1) We propose an evaluation methodology that…

Nufix

Zhenming Li, Ying Wang, Zeqi Lin +2 more

Information Systems · Proceedings of the 44th International Conference on Software Engineering · · cited 7× · DOI ↗ · more like this ·

Developers usually suffer from <u>d</u>ependency <u>m</u>aze (DM) issues, i.e., package dependency constraints are violated when a project's platform or dependencies are changed. This problem is especially serious in .NET ecosystem due to its fragmented platforms (e.g., .NET Framework, .NET Core, and .NET Standard). Fixing DM issues is challenging due to the complexity of dependency constraints: multiple DM issues often occur in one project; solving one DM issue usually causes another DM issue cropping up; the exponential search space of possible dependency combinations is also a barrier.

Graph Embedding based Code Search in Software Project

Yanzhen Zou, Chunyang Ling, Zeqi Lin +1 more

Information Systems · · cited 6× · DOI ↗ · more like this ·

Source code search is one of the most important methods to study and reuse software project. Currently, natural language based code search mainly faces the following two challenges: 1) More accurate search results are required when software projects evolve to be more heterogeneous and complex. 2) The semantic relationships between code elements (classes, methods, etc.) need to be illustrated so that developers could better understand their usage scenarios. To deal with these issues, we propose a novel approach to searching a software project's source code based on graph embedding. First, we build a software project's code graph automatically from its source code and represent each code element in the code graph with graph embedding. Second, we search code graph with natural language questions, return corresponding subgraph that composed of relevant code elements and their associated…

Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

Microsoft, :, Abdelrahman Abouelenin +73 more

cs.CL · · cited 5× · arXiv ↗ · more like this ·

We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language model trained on high-quality web and synthetic data, significantly outperforming recent open-source models of similar size and matching the performance of models twice its size on math and coding tasks requiring complex reasoning. This achievement is driven by a carefully curated synthetic data recipe emphasizing high-quality math and coding datasets. Compared to its predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of 200K tokens to better support multilingual applications, as well as group query attention for more efficient long-sequence generation. Phi-4-Multimodal is a multimodal model that integrates text, vision, and speech/audio input modalities into a single model. Its novel modality extension approach…