Author

Fengdi Che

11 papers · 2019–2026

2019peak 4 in 20252026

Frequent co-authors

Papers

showing 1–11 of 11 RSS

Bayesian Q-learning With Imperfect Expert Demonstrations

Fengdi Che, Xiru Zhu, Doina Precup +2 more

cs.LG · October 2022 · cited 3× · arXiv ↗ · more like this ·

Guided exploration with expert demonstrations improves data efficiency for reinforcement learning, but current algorithms often overuse expert information. We propose a novel algorithm to speed up Q-learning with the help of a limited amount of imperfect expert demonstrations. The algorithm avoids excessive reliance on expert data by relaxing the optimal expert assumption and gradually reducing the usage of uninformative expert data. Experimentally, we evaluate our approach on a sparse-reward chain environment and six more complicated Atari games with delayed rewards. With the proposed methods, we can achieve better results than Deep Q-learning from Demonstrations (Hester et al., 2017) in most environments.

3SGAN: 3D Shape Embedded Generative Adversarial Networks

Fengdi Che, Xiru Zhu, Tianzi Yang +1 more

Computer Vision and Pattern Recognition · October 2019 · cited 3× · DOI ↗ · more like this ·

Despite recent advances in Generative Adversarial Models(GAN) for image generation, significant gaps remain concerning the generation of boundary and spatial structure. In this paper, we propose a new approach to generate edge and depth information combined with an RGB image to solve this problem. More specifically, we propose two new regularization models. Our first model enforces image-depth-edge alignments by controlling the second-order derivative of depth and the first-order derivative of RGB maps, enforcing smoothness and consistency. The second model leverages multiview synthesis to regularize RGB and depth by computing the difference between an expected rotated object compared to a conditionally generated view of the object; enforcing projection consistency enables the model to directly learn spatial structures and depths. To evaluate our approach, we generated an RGB-D dataset…

Detecting GAN generated errors

Xiru Zhu, Fengdi Che, Tianzi Yang +3 more

eess.IV · 2 Dec 2019 · cited 1× · arXiv ↗ · more like this ·

Despite an impressive performance from the latest GAN for generating hyper-realistic images, GAN discriminators have difficulty evaluating the quality of an individual generated sample. This is because the task of evaluating the quality of a generated image differs from deciding if an image is real or fake. A generated image could be perfect except in a single area but still be detected as fake. Instead, we propose a novel approach for detecting where errors occur within a generated image. By collaging real images with generated images, we compute for each pixel, whether it belongs to the real distribution or generated distribution. Furthermore, we leverage attention to model long-range dependency; this allows detection of errors which are reasonable locally but not holistically. For evaluation, we show that our error detection can act as a quality metric for an individual image, unlike…

VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Code

Lingfei Zeng, Fengdi Che, Xuhan Huang +4 more

cs.PL · October 2025 · cited 0× · arXiv ↗ · more like this ·

Formal verification is the next frontier for ensuring the correctness of code generated by Large Language Models (LLMs). While methods that co-generate code and formal specifications in formal languages, like Dafny, can, in principle, prove alignment with user intent, progress is bottlenecked by specification quality evaluation. Current benchmarks rely on matching against ground-truth specifications, a manual and expertise-intensive process that has limited existing datasets to a few hundred simple problems and also suffers from a reliability issue. To address this, we introduce VeriEquivBench, a new benchmark with $2,389$ complex algorithmic problems that probe the limitations of current models in both code generation and formal reasoning. Our evaluation framework replaces ground-truth matching with a formally grounded metric, the equivalence score, and rigorously verifies the quality…

A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory

Fengdi Che

cs.LG · 11 Aug 2025 · cited 0× · arXiv ↗ · more like this ·

Offline reinforcement learning (RL) aims to optimize the return given a fixed dataset of agent trajectories without additional interactions with the environment. While algorithm development has progressed rapidly, significant theoretical advances have also been made in understanding the fundamental challenges of offline RL. However, bridging these theoretical insights with practical algorithm design remains an ongoing challenge. In this survey, we explore key intuitions derived from theoretical work and their implications for offline RL algorithms. We begin by listing the conditions needed for the proofs, including function representation and data coverage assumptions. Function representation conditions tell us what to expect for generalization, and data coverage assumptions describe the quality requirement of the data. We then examine counterexamples, where offline RL is not solvable…

Re:Form -- Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny

Chuanhao Yan, Fengdi Che, Xuhan Huang +12 more

cs.CL · July 2025 · cited 0× · arXiv ↗ · more like this ·

Existing informal language-based (e.g., human language) Large Language Models (LLMs) trained with Reinforcement Learning (RL) face a significant challenge: their verification processes, which provide crucial training signals, are neither reliable nor scalable. In fact, the prevalent large proprietary models could hardly generate verifiable programs. A promising yet largely uncharted alternative is formal language-based reasoning. Grounding LLMs in rigorous formal systems where generative models operate in formal language spaces (e.g., Dafny) enables the automatic and mathematically provable verification of their reasoning processes and outcomes. This capability is pivotal for achieving large-scale, reliable formal software verification. It is a common practice to employ human-annotated chain-of-thought and answers to induce the reasoning and coding capabilities of LLMs. Unfortunately…

Average-DICE: Stationary Distribution Correction by Regression

Fengdi Che, Bryan Chan, Chen Ma +1 more

cs.LG · March 2025 · cited 0× · arXiv ↗ · more like this ·

Off-policy policy evaluation (OPE), an essential component of reinforcement learning, has long suffered from stationary state distribution mismatch, undermining both stability and accuracy of OPE estimates. While existing methods correct distribution shifts by estimating density ratios, they often rely on expensive optimization or backward Bellman-based updates and struggle to outperform simpler baselines. We introduce AVG-DICE, a computationally simple Monte Carlo estimator for the density ratio that averages discounted importance sampling ratios, providing an unbiased and consistent correction. AVG-DICE extends naturally to nonlinear function approximation using regression, which we roughly tune and test on OPE tasks based on Mujoco Gym environments and compare with state-of-the-art density-ratio estimators using their reported hyperparameters. In our experiments, AVG-DICE is at least…

Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation

Fengdi Che, Chenjun Xiao, Jincheng Mei +6 more

cs.LG · May 2024 · cited 0× · arXiv ↗ · more like this ·

We prove that the combination of a target network and over-parameterized linear function approximation establishes a weaker convergence condition for bootstrapped value estimation in certain cases, even with off-policy data. Our condition is naturally satisfied for expected updates over the entire state-action space or learning with a batch of complete trajectories from episodic Markov decision processes. Notably, using only a target network or an over-parameterized model does not provide such a convergence guarantee. Additionally, we extend our results to learning with truncated trajectories, showing that convergence is achievable for all tasks with minor modifications, akin to value truncation for the final states in trajectories. Our primary result focuses on temporal difference estimation for prediction, providing high-probability value estimation error bounds and empirical analysis…

Correcting discount-factor mismatch in on-policy policy gradient methods

Fengdi Che, Gautham Vasan, A. Rupam Mahmood

cs.LG · 23 Jun 2023 · cited 0× · arXiv ↗ · more like this ·

The policy gradient theorem gives a convenient form of the policy gradient in terms of three factors: an action value, a gradient of the action likelihood, and a state distribution involving discounting called the \emph{discounted stationary distribution}. But commonly used on-policy methods based on the policy gradient theorem ignores the discount factor in the state distribution, which is technically incorrect and may even cause degenerate learning behavior in some environments. An existing solution corrects this discrepancy by using $γ^t$ as a factor in the gradient estimate. However, this solution is not widely adopted and does not work well in tasks where the later states are similar to earlier states. We introduce a novel distribution correction to account for the discounted stationary distribution that can be plugged into many existing gradient estimators. Our correction…

Learning with a Single Rollout via Pass@k Policy Optimization

Fengdi Che, Yang Liu, Lei Yu +4 more

cs.LG · 24 Jun 2026 · arXiv ↗ · more like this ·

Group Relative Policy Optimization (GRPO) typically requires multiple rollouts per prompt to estimate relative credit, incurring substantial sampling cost and exacerbating training asynchrony. This limitation is particularly severe for scientific reasoning agents, where each trajectory may involve long-horizon reasoning, multiple turns, and parallel sub-runs. We study single-rollout reinforcement learning, where only one rollout is collected per prompt, and find that naive training can stagnate or become severely unstable. We introduce Single-Rollout Pass@k Policy Optimization (SR-PPO), which combines Monte Carlo token-level advantage estimation with Pass@k gradient reweighting. An online critic without large-scale pretraining is used both as a prefix-dependent baseline for credit assignment and to estimate the Pass@k gradient weight. Using only one rollout per prompt, SR-PPO achieves…

GenPage: Towards End-to-End Generative Homepage Construction at Netflix

Lequn Wang, Jiangwei Pan, Fengdi Che +1 more

cs.IR · June 2026 · arXiv ↗ · more like this ·

We present GenPage, an end-to-end generative approach to Netflix homepage construction that replaces the traditional multi-stage recommender stack with a single transformer. GenPage treats the user and request context as a prompt and autoregressively generates the entire structured, multi-row homepage as the response. We adapt the LLM training recipe: pretraining on production pages, followed by post-training via weighted binary classification (WBC) or reinforcement learning (RL). For industry-scale deployment, we introduce techniques addressing cold start, model freshness, business-rule enforcement, and serving efficiency. In online A/B tests against a mature, highly optimized production homepage recommender, GenPage delivered a substantial lift on the core user engagement metric we use for launch decisions, while reducing end-to-end serving latency by 20%. Offline, two findings stand…