arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Jacob Beck

35 papers · 1961–2026

1967: 01968: 01969: 01970: 01971: 01972: 01973: 01974: 01975: 01976: 01977: 01978: 01979: 01980: 01981: 01982: 01983: 01984: 01985: 01986: 01987: 01988: 01989: 01990: 01991: 01992: 01993: 01994: 01995: 01996: 11997: 01998: 01999: 02000: 02001: 02002: 02003: 02004: 02005: 02006: 02007: 02008: 02009: 02010: 12011: 02012: 12013: 02014: 12015: 02016: 02017: 02018: 12019: 22020: 22021: 32022: 32023: 52024: 62025: 42026: 3
1967peak 6 in 20242026

Frequent co-authors

Papers

showing 1–25 of 35 RSS

A Tutorial on Meta-Reinforcement Learning

Jacob Beck, Risto Vuorio, Evan Zheran Liu +4 more

cs.LG · · cited 178× · arXiv ↗ · more like this ·

While deep reinforcement learning (RL) has fueled multiple high-profile successes in machine learning, it is held back from more widespread adoption by its often poor data efficiency and the limited generality of the policies it produces. A promising approach for alleviating these limitations is to cast the development of better RL algorithms as a machine learning problem itself in a process called meta-RL. Meta-RL is most commonly studied in a problem setting where, given a distribution of tasks, the goal is to learn a policy that is capable of adapting to any new task from the task distribution with as little data as possible. In this survey, we describe the meta-RL problem setting in detail as well as its major variations. We discuss how, at a high level, meta-RL research can be clustered based on the presence of a task distribution and the learning budget available for each…

The Generality Constraint and the Structure of Thought

Jacob Beck

Statistics and Probability · Mind · · cited 127× · DOI ↗ · more like this ·

According to the Generality Constraint, mental states with conceptual content must be capable of recombining in certain systematic ways. Drawing on empirical evidence from cognitive science, I argue that so-called analogue magnitude states violate this recombinability condition and thus have nonconceptual content. I further argue that this result has two significant consequences: it demonstrates that nonconceptual content seeps beyond perception and infiltrates cognition; and it shows that whether mental states have nonconceptual content is largely an empirical matter determined by the structure of the neural representations underlying them.

Analogue Magnitude Representations: A Philosophical Introduction

Jacob Beck

Statistics and Probability · The British Journal for the Philosophy of Science · · cited 113× · DOI ↗ · more like this ·

Empirical discussions of mental representation appeal to a wide variety of representational kinds. Some of these kinds, such as the sentential representations underlying language use and the pictorial representations of visual imagery, are thoroughly familiar to philosophers. Others have received almost no philosophical attention at all. Included in this latter category are analogue magnitude representations, which enable a wide range of organisms to primitively represent spatial, temporal, numerical, and related magnitudes. This article aims to introduce analogue magnitude representations to a philosophical audience by rehearsing empirical evidence for their existence and analysing their format, their content, and the computations they support. 1 Background 1.1 Evidence of analogue magnitude representations 1.2 Weber’s law 1.3 Scepticism about analogue magnitude representations2 Format…

The number sense represents (rational) numbers

Sam Clarke, Jacob Beck

Statistics and Probability · Behavioral and Brain Sciences · · cited 69× · DOI ↗ · more like this ·

On a now orthodox view, humans and many other animals possess a "number sense," or approximate number system (ANS), that represents number. Recently, this orthodox view has been subject to numerous critiques that question whether the ANS genuinely represents number. We distinguish three lines of critique - the arguments from congruency, confounds, and imprecision - and show that none succeed. We then provide positive reasons to think that the ANS genuinely represents numbers, and not just non-numerical confounds or exotic substitutes for number, such as "numerosities" or "quanticals," as critics propose. In so doing, we raise a neglected question: numbers of what kind? Proponents of the orthodox view have been remarkably coy on this issue. But this is unsatisfactory since the predictions of the orthodox view, including the situations in which the ANS is expected to succeed or fail, turn…

The Scaling of Pitch by the Method of Magnitude-Estimation

Jacob Beck, William A. Shaw

Computer Vision and Pattern Recognition · The American Journal of Psychology · · cited 26× · DOI ↗ · more like this ·

An important problem is the degree to which ratio-scales represent stable relationships which are not affected by biases.' Stevens has argued that the direct estimations of sensory magnitude generally yield functions of apparent magnitude primarily determined by the functional input-output relations of the receptors,2 but there is evidence that under some conditions response-biases and context-effects may strongly influence ratio-judgments.3 The problem of the invariance of scale-values calls for further investigation. The purpose of the present study is to determine the effect upon the pitchfunction of the frequency of the standard.

Quality aspects of annotated data

Jacob Beck

Computer Science Applications · AStA Wirtschafts- und Sozialstatistisches Archiv · · cited 12× · DOI ↗ · more like this ·

Abstract The quality of Machine Learning (ML) applications is commonly assessed by quantifying how well an algorithm fits its respective training data. Yet, a perfect model that learns from and reproduces erroneous data will always be flawed in its real-world application. Hence, a comprehensive assessment of ML quality must include an additional data perspective, especially for models trained on human-annotated data. For the collection of human-annotated training data, best practices often do not exist and leave researchers to make arbitrary decisions when collecting annotations. Decisions about the selection of annotators or label options may affect training data quality and model performance. In this paper, I will outline and summarize previous research and approaches to the collection of annotated training data. I look at data annotation and its quality confounders from two…

Magnitude Estimations of Pitch

Jacob Beck, William A. Shaw

Signal Processing · The Journal of the Acoustical Society of America · · cited 12× · DOI ↗ · more like this ·

Using the method of magnitude estimation, the experiments reported investigated the effect of the frequency of the standard upon the pitch functions for piano tones and randomly chosen pure tones. The pitch functions obtained are found to agree in an important respect with each other and with the function previously reported for pure tones chosen to correspond to notes on the musical scale. When the frequency of the standard was the lowest tone presented, the magnitude estimations yielded a function similar to the revised mel scale, and when the standard was a frequency nearer the middle of the series, the magnitude estimations yielded a steeper function closer to the original mel scale. Data are also presented showing that the numerical value of the standard can affect the magnitude estimation function. These results indicate that response biases relating to the frequency and numerical…

AMRL: Aggregated Memory For Reinforcement Learning

Jacob Beck, Kamil Ciosek, Sam Devlin +3 more

Artificial Intelligence · International Conference on Learning Representations · · cited 6× · OpenAlex ↗ · more like this ·

In many partially observable scenarios, Reinforcement Learning (RL) agents must rely on long-term memory in order to learn an optimal policy. We demonstrate that using techniques from NLP and supervised learning fails at RL tasks due to stochasticity from the environment and from exploration. Utilizing our insights on the limitations of traditional memory methods in RL, we propose AMRL, a class of models that can learn better policies with greater sample efficiency and are resilient to noisy inputs. Specifically, our models use a standard memory module to summarize short-term context, and then aggregate all prior states from the standard model without respect to order. We show that this provides advantages both in terms of gradient decay and signal-to-noise ratio over time. Evaluating in Minecraft and maze environments that test long-term memory, we find that our model improves average…

Toward Integrating ChatGPT Into Satellite Image Annotation Workflows: A Comparison of Label Quality and Costs of Human and Automated Annotators

Jacob Beck, Lukas Malte Kemeter, Konrad Dürrbeck +2 more

Computer Vision and Pattern Recognition · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · · cited 5× · DOI ↗ · more like this ·

High-quality annotations are a critical success factor for machine learning (ML) applications. To achieve this, we have traditionally relied on human annotators, navigating the challenges of limited budgets and the varying task-specific expertise, costs, and availability. Since the emergence of large language models (LLMs), their popularity for generating automated annotations has grown, extending possibilities and complexity of designing an efficient annotation strategy. Increasingly, computer vision capabilities have been integrated into general-purpose LLMs like ChatGPT. This raises the question of how effectively LLMs can be used in satellite image annotation tasks and how they compare to traditional annotator types. This study presents a comprehensive investigation and comparison of various human and automated annotators for image classification. We evaluate the feasibility and…

Hypernetworks in Meta-Reinforcement Learning

Jacob Beck, Matthew Thomas Jackson, Risto Vuorio +1 more

cs.LG · · cited 5× · arXiv ↗ · more like this ·

Training a reinforcement learning (RL) agent on a real-world robotics task remains generally impractical due to sample inefficiency. Multi-task RL and meta-RL aim to improve sample efficiency by generalizing over a distribution of related tasks. However, doing so is difficult in practice: In multi-task RL, state of the art methods often fail to outperform a degenerate solution that simply learns each task separately. Hypernetworks are a promising path forward since they replicate the separate policies of the degenerate solution while also allowing for generalization across tasks, and are applicable to meta-RL. However, evidence from supervised learning suggests hypernetwork performance is highly sensitive to the initialization. In this paper, we 1) show that hypernetwork initialization is also a critical factor in meta-RL, and that naive initializations yield poor performance; 2)…

Annotation Sensitivity: Training Data Collection Methods Affect Model Performance

Christoph Kern, Stephanie Eckman, Jacob Beck +3 more

stat.ML · · cited 4× · arXiv ↗ · more like this ·

When training data are collected from human annotators, the design of the annotation instrument, the instructions given to annotators, the characteristics of the annotators, and their interactions can impact training data. This study demonstrates that design choices made when creating an annotation instrument also impact the models trained on the resulting annotations. We introduce the term annotation sensitivity to refer to the impact of annotation data collection methods on the annotations themselves and on downstream model performance and predictions. We collect annotations of hate speech and offensive language in five experimental conditions of an annotation instrument, randomly assigning annotators to conditions. We then fine-tune BERT models on each of the five resulting datasets and evaluate model performance on a holdout portion of each condition. We find considerable…

Numbers, numerosities, and new directions

Sam Clarke, Jacob Beck

Statistics and Probability · Behavioral and Brain Sciences · · cited 4× · DOI ↗ · more like this ·

In our target article, we argued that the number sense represents natural and rational numbers. Here, we respond to the 26 commentaries we received, highlighting new directions for empirical and theoretical research. We discuss two background assumptions, arguments against the number sense, whether the approximate number system (ANS) represents numbers or numerosities, and why the ANS represents rational (but not irrational) numbers.

Metalic: Meta-Learning In-Context with Protein Language Models

Jacob Beck, Shikha Surana, Manus McAuliffe +4 more

cs.LG · · cited 3× · arXiv ↗ · more like this ·

Predicting the biophysical and functional properties of proteins is essential for in silico protein design. Machine learning has emerged as a promising technique for such prediction tasks. However, the relative scarcity of in vitro annotations means that these models often have little, or no, specific data on the desired fitness prediction task. As a result of limited data, protein language models (PLMs) are typically trained on general protein sequence modeling tasks, and then fine-tuned, or applied zero-shot, to protein fitness prediction. When no task data is available, the models make strong assumptions about the correlation between the protein sequence likelihood and fitness scores. In contrast, we propose meta-learning over a distribution of standard fitness prediction tasks, and demonstrate positive transfer to unseen fitness prediction tasks. Our method, called Metalic…

Bias in the Loop: How Humans Evaluate AI-Generated Suggestions

Jacob Beck, Stephanie Eckman, Christoph Kern +1 more

cs.HC · · cited 2× · arXiv ↗ · more like this ·

Human-AI collaboration increasingly drives decision-making across industries, from medical diagnosis to content moderation. While AI systems promise efficiency gains by providing automated suggestions for human review, these workflows can trigger cognitive biases that degrade performance. We know little about the psychological factors that determine when these collaborations succeed or fail. We conducted a randomized experiment with 2,784 participants to examine how task design and individual characteristics shape human responses to AI-generated suggestions. Using a controlled annotation task, we manipulated three factors: AI suggestion quality in the first three instances, task burden through required corrections, and performance-based financial incentives. We collected demographics, attitudes toward AI, and behavioral data to assess four performance metrics: accuracy, correction…

Between perception and thought

Jacob Beck

Statistics and Probability · Philosophy and Phenomenological Research · · cited 2× · DOI ↗ · more like this ·

There is a difference between seeing and thinking—for example, between seeing the red lines on your COVID test and judging that you have COVID. Block (2023) is interested in the nature of this border—or more broadly, in the nature of the border between perception, which includes not just seeing but also hearing and touching and perhaps smelling and tasting, and cognition, which involves “capacities for propositional thought, reasoning, evaluating, and decision making” (p. 13). There have been a number of proposals lately about how to understand this border (Clarke & Beck 2023). Block pursues an approach grounded in representational format: whereas perception is constitutively nonpropositional, nonconceptual, and iconic (NNI), cognition is constitutively none of those things. Paridigmatically, it is the opposite: propositional, conceptual, and discursive (PCD). Block's format approach…

Universal Morphology Control via Contextual Modulation

Zheng Xiong, Jacob Beck, Shimon Whiteson

cs.AI · · cited 2× · arXiv ↗ · more like this ·

Learning a universal policy across different robot morphologies can significantly improve learning efficiency and generalization in continuous control. However, it poses a challenging multi-task reinforcement learning problem, as the optimal policy may be quite different across robots and critically depend on the morphology. Existing methods utilize graph neural networks or transformers to handle heterogeneous state and action spaces across different morphologies, but pay little attention to the dependency of a robot's control policy on its morphology context. In this paper, we propose a hierarchical architecture to better model this dependency via contextual modulation, which includes two key submodules: (1) Instead of enforcing hard parameter sharing across robots, we use hypernetworks to generate morphology-dependent control parameters; (2) We propose a fixed attention mechanism that…

Hill Sampling for Test-Time Scaling: A Simple and Better Alternative to Repeated Sampling, Evolution, and Training

Jacob Beck, Philip V. Ogren, Ari Kobren

cs.LG · · cited 1× · arXiv ↗ · more like this ·

Large language models (LLMs) can improve solutions to verifiable scientific and algorithmic problems by spending additional computation at test time. Recent systems achieve strong results with increasingly elaborate evolutionary search harnesses or by updating model parameters during test-time training. We ask how much of this machinery is necessary. We introduce Hill Sampling, a simple procedure that repeatedly samples candidate program edits from a frozen LLM, retains the best program found so far, and conditions all subsequent samples on that program. We evaluate the method on circle packing, sums/differences of sets, and Erdos' minimum-overlap problem using three open-weight models. Hill Sampling sets a new state of the art on circle packing among published methods, improves over the AlphaEvolve reference on Erdos' minimum-overlap problem, and achieves strong results on sums and…

Cracking the Code: Evaluating Zero-Shot Prompting Methods for Providing Programming Feedback

Niklas Ippisch, Anna-Carolina Haensch, Jan Simson +3 more

cs.SE · · cited 1× · arXiv ↗ · more like this ·

Despite the growing use of large language models (LLMs) for providing feedback, limited research has explored how to achieve high-quality feedback. This case study introduces an evaluation framework to assess different zero-shot prompt engineering methods. We varied the prompts systematically and analyzed the provided feedback on programming errors in R. The results suggest that prompts suggesting a stepwise procedure increase the precision, while omitting explicit specifications about which provided data to analyze improves error identification.

SplAgger: Split Aggregation for Meta-Reinforcement Learning

Jacob Beck, Matthew Jackson, Risto Vuorio +2 more

cs.LG · · cited 1× · arXiv ↗ · more like this ·

A core ambition of reinforcement learning (RL) is the creation of agents capable of rapid learning in novel tasks. Meta-RL aims to achieve this by directly learning such agents. Black box methods do so by training off-the-shelf sequence models end-to-end. By contrast, task inference methods explicitly infer a posterior distribution over the unknown task, typically using distinct objectives and sequence models designed to enable task inference. Recent work has shown that task inference methods are not necessary for strong performance. However, it remains unclear whether task inference sequence models are beneficial even when task inference objectives are not. In this paper, we present evidence that task inference sequence models are indeed still beneficial. In particular, we investigate sequence models with permutation invariant aggregation, which exploit the fact that, due to the Markov…

Trust Region Bounds for Decentralized PPO Under Non-stationarity

Mingfei Sun, Sam Devlin, Jacob Beck +2 more

cs.LG · · cited 1× · arXiv ↗ · more like this ·

We present trust region bounds for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary. This new analysis provides a theoretical understanding of the strong performance of two recent actor-critic methods for MARL, which both rely on independent ratios, i.e., computing probability ratios separately for each agent's policy. We show that, despite the non-stationarity that independent ratios cause, a monotonic improvement guarantee still arises as a result of enforcing the trust region constraint over all decentralized policies. We also show this trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training, providing a theoretical foundation for proximal ratio clipping. Finally, our empirical results…

Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputs

Pranjal Garg, Jacob Beck

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Language models sometimes produce correct outputs even when their inputs are corrupted by deletion, replacement, or misspelling. We study the internal processes accompanying this behavior, which we call context restoration, in controlled attention-only transformers and five pretrained LLMs (1B-32B parameters) across arithmetic, reading comprehension, and multiple-choice reasoning tasks. In the attention-only transformers, restoration emerges spontaneously despite training exclusively on clean sequences, without corruption training or an explicit denoising objective. We find that context restoration follows a two-phase process: early layers localize effects associated with repair at corrupted positions, while later layers accumulate these effects at uncorrupted positions through the residual stream and ultimately concentrate them at the output position. Repair outcome is predictable from…

Softening the Border: A Capacities Approach to the Perception–Cognition Distinction

Jacob Beck, Casey O’Callaghan

Statistics and Probability · Philosophy and Phenomenological Research · · cited 0× · DOI ↗ · more like this ·

ABSTRACT Approaches to the perception–cognition distinction tend toward two extremes. Many embrace a hard border, treating perception and cognition as mutually exclusive, non‐overlapping categories. By contrast, eliminativism denies that any principled, theoretically useful distinction exists between perception and cognition. This article offers a third way, describing a principled but soft border between perception and cognition. This non‐exclusive approach differentiates perception from cognition while allowing that they overlap. Because it rests on the notion of a psychological capacity, we call it the capacities approach . Perception and cognition are distinct psychological faculties, each of which comprises a collection of psychological capacities. What unifies and differentiates each such collection is the distinctive way in which its capacities are exercised. Perceptual…

Recurrent Hypernetworks are Surprisingly Strong in Meta-RL

Jacob Beck, Risto Vuorio, Zheng Xiong +1 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Deep reinforcement learning (RL) is notoriously impractical to deploy due to sample inefficiency. Meta-RL directly addresses this sample inefficiency by learning to perform few-shot learning when a distribution of related tasks is available for meta-training. While many specialized meta-RL methods have been proposed, recent work suggests that end-to-end learning in conjunction with an off-the-shelf sequential model, such as a recurrent network, is a surprisingly strong baseline. However, such claims have been controversial due to limited supporting evidence, particularly in the face of prior work establishing precisely the opposite. In this paper, we conduct an empirical investigation. While we likewise find that a recurrent network can achieve strong performance, we demonstrate that the use of hypernetworks is crucial to maximizing their potential. Surprisingly, when combined with…

An Investigation of the Bias-Variance Tradeoff in Meta-Gradients

Risto Vuorio, Jacob Beck, Shimon Whiteson +2 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Meta-gradients provide a general approach for optimizing the meta-parameters of reinforcement learning (RL) algorithms. Estimation of meta-gradients is central to the performance of these meta-algorithms, and has been studied in the setting of MAML-style short-horizon meta-RL problems. In this context, prior work has investigated the estimation of the Hessian of the RL objective, as well as tackling the problem of credit assignment to pre-adaptation behavior by making a sampling correction. However, we show that Hessian estimation, implemented for example by DiCE and its variants, always adds bias and can also add variance to meta-gradient estimation. Meanwhile, meta-gradient estimation has been studied less in the important long-horizon setting, where backpropagation through the full inner optimization trajectories is not feasible. We study the bias and variance tradeoff arising from…