arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Ankush

1,196 papers · 1753–2026

1967: 01968: 01969: 01970: 01971: 01972: 01973: 01974: 01975: 01976: 01977: 01978: 01979: 01980: 01981: 01982: 01983: 01984: 01985: 01986: 01987: 01988: 01989: 01990: 01991: 01992: 01993: 01994: 01995: 01996: 01997: 01998: 11999: 02000: 22001: 22002: 02003: 12004: 92005: 82006: 92007: 62008: 132009: 102010: 122011: 212012: 192013: 332014: 342015: 392016: 452017: 732018: 592019: 622020: 792021: 862022: 792023: 1042024: 1312025: 1502026: 105
1967peak 150 in 20252026

Frequent co-authors

Papers

showing 1–25 of 1,196 RSS

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Gemini Team, Petko Georgiev, Ving Ian Lei +1134 more

cs.CL · · cited 4,033× · arXiv ↗ · more like this ·

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's…

Synthetic Data for Text Localisation in Natural Images

Ankush Gupta, Andrea Vedaldi, Andrew Zisserman

cs.CV · · cited 1,582× · arXiv ↗ · more like this ·

In this paper we introduce a new method for text detection in natural images. The method comprises two contributions: First, a fast and scalable engine to generate synthetic images of text in clutter. This engine overlays synthetic text to existing background images in a natural way, accounting for the local 3D scene geometry. Second, we use the synthetic images to train a Fully-Convolutional Regression Network (FCRN) which efficiently performs text detection and bounding-box regression at all locations and multiple scales in an image. We discuss the relation of FCRN to the recently-introduced YOLO detector, as well as other end-to-end object detection systems based on deep learning. The resulting detection network significantly out performs current methods for text detection in natural images, achieving an F-measure of 84.2% on the standard ICDAR 2013 benchmark. Furthermore, it can…

Gemini: A Family of Highly Capable Multimodal Models

Gemini Team, Rohan Anil, Sebastian Borgeaud +1348 more

cs.CL · · cited 816× · arXiv ↗ · more like this ·

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models…

CrossTransformers: spatially-aware few-shot transfer

Carl Doersch, Ankush Gupta, Andrew Zisserman

cs.CV · · cited 420× · arXiv ↗ · more like this ·

Given new tasks with very little data$-$such as new classes in a classification problem or a domain shift in the input$-$performance of modern vision systems degrades remarkably quickly. In this work, we illustrate how the neural network representations which underpin modern vision systems are subject to supervision collapse, whereby they lose any information that is not necessary for performing the training task, including information that may be necessary for transfer to new tasks or domains. We then propose two methods to mitigate this problem. First, we employ self-supervised learning to encourage general-purpose features that transfer better. Second, we propose a novel Transformer based neural network architecture called CrossTransformers, which can take a small number of labeled images and an unlabeled query, find coarse spatial correspondence between the query and the labeled…

Perception Test: A Diagnostic Benchmark for Multimodal Video Models

Viorica Pătrăucean, Lucas Smaira, Ankush Gupta +21 more

cs.CV · · cited 389× · arXiv ↗ · more like this ·

We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GPT-4). Compared to existing benchmarks that focus on computational tasks (e.g. classification, detection or tracking), the Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities, to provide a comprehensive and efficient evaluation tool. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime. For these purposes, the Perception Test introduces 11.6k real-world videos, 23s average length, designed to show perceptually interesting situations, filmed by around 100 participants worldwide. The videos are…

A multimodal generative AI copilot for human pathology

Ming Y. Lu, Bowen Chen, Drew F. K. Williamson +9 more

Artificial Intelligence · Nature · · cited 366× · DOI ↗ · more like this ·

Abstract Computational pathology 1,2 has witnessed considerable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders 3,4 . However, despite the explosive growth of generative artificial intelligence (AI), there have been few studies on building general-purpose multimodal AI assistants and copilots 5 tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology. We built PathChat by adapting a foundational vision encoder for pathology, combining it with a pretrained large language model and fine-tuning the whole system on over 456,000 diverse visual-language instructions consisting of 999,202 question and answer turns. We compare PathChat with several multimodal vision-language AI assistants and GPT-4V, which powers the commercially available multimodal general-purpose AI…

TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement

Carl Doersch, Yi Yang, Mel Vecerik +5 more

cs.CV · · cited 360× · arXiv ↗ · more like this ·

We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages: (1) a matching stage, which independently locates a suitable candidate point match for the query point on every other frame, and (2) a refinement stage, which updates both the trajectory and query features based on local correlations. The resulting model surpasses all baseline methods by a significant margin on the TAP-Vid benchmark, as demonstrated by an approximate 20% absolute average Jaccard (AJ) improvement on DAVIS. Our model facilitates fast inference on long and high-resolution video sequences. On a modern GPU, our implementation has the capacity to track points faster than real-time, and can be flexibly extended to higher-resolution videos. Given the high-quality trajectories extracted from a large…

TAP-Vid: A Benchmark for Tracking Any Point in a Video

Carl Doersch, Ankush Gupta, Larisa Markeeva +6 more

cs.CV · · cited 336× · arXiv ↗ · more like this ·

Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences about 3D shape, physical properties and object interactions. While the problem of tracking arbitrary physical points on surfaces over longer video clips has received some attention, no dataset or benchmark for evaluation existed, until now. In this paper, we first formalize the problem, naming it tracking any point (TAP). We introduce a companion benchmark, TAP-Vid, which is composed of both real-world videos with accurate human annotations of point tracks, and synthetic videos with perfect ground-truth point tracks. Central to the construction of our benchmark is a novel semi-automatic crowdsourced pipeline which uses optical flow estimates to compensate for easier, short-term motion like camera shake, allowing…

Incidence of Infertility and Pregnancy Complications in US Female Surgeons

Erika L. Rangel, Manuel Castillo‐Angeles, Sarah Rae Easter +6 more

Gender Studies · JAMA Surgery · · cited 300× · DOI ↗ · more like this ·

Importance: While surgeons often delay pregnancy and childbearing because of training and establishing early careers, little is known about risks of infertility and pregnancy complications among female surgeons. Objective: To describe the incidence of infertility and pregnancy complications among female surgeons in the US and to identify workplace factors associated with increased risk compared with a sociodemographically similar nonsurgeon population. Design, Setting, and Participants: This self-administered survey questionnaire was electronically distributed and collected from November 2020 to January 2021 through multiple surgical societies in the US and social media among male and female attending and resident surgeons with children. Nonchildbearing surgeons were asked to answer questions regarding the pregnancies of their nonsurgeon partners as applicable. Exposures: Surgical…

SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text

Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi +1 more

Artificial Intelligence · · cited 289× · DOI ↗ · more like this ·

In this paper, we present the SemEval-2019 Task 3 -EmoContext: Contextual Emotion Detection in Text. Lack of facial expressions and voice modulations make detecting emotions in text a challenging problem. For instance, as humans, on reading "Why don't you ever text me!" we can either interpret it as a sad or angry emotion and the same ambiguity exists for machines. However, the context of dialogue can prove helpful in detection of the emotion. In this task, given a textual dialogue i.e. an utterance along with two previous turns of context, the goal was to infer the underlying emotion of the utterance by choosing from four emotion classes -Happy, Sad, Angry and Others. To facilitate the participation in this task, textual dialogues from user interaction with a conversational agent were taken and annotated for emotion classes after several data processing steps. A training data set of…

Unsupervised Learning of Object Landmarks through Conditional Image Generation

Tomas Jakab, Ankush Gupta, Hakan Bilen +1 more

cs.CV · · cited 275× · arXiv ↗ · more like this ·

We propose a method for learning landmark detectors for visual objects (such as the eyes and the nose in a face) without any manual supervision. We cast this as the problem of generating images that combine the appearance of the object as seen in a first example image with the geometry of the object as seen in a second example image, where the two examples differ by a viewpoint change and/or an object deformation. In order to factorize appearance and geometry, we introduce a tight bottleneck in the geometry-extraction process that selects and distils geometry-related features. Compared to standard image generation problems, which often use generative adversarial networks, our generation task is conditioned on both appearance and geometry and thus is significantly less ambiguous, to the point that adopting a simple perceptual loss formulation is sufficient. We demonstrate that our…

A Deep Generative Framework for Paraphrase Generation

Ankush Gupta, Arvind Agarwal, Prawaan Singh +1 more

cs.CL · · cited 268× · arXiv ↗ · more like this ·

Paraphrase generation is an important problem in NLP, especially in question answering, information retrieval, information extraction, conversation systems, to name a few. In this paper, we address the problem of generating paraphrases automatically. Our proposed method is based on a combination of deep generative models (VAE) with sequence-to-sequence models (LSTM) to generate paraphrases, given an input sentence. Traditional VAEs when combined with recurrent neural networks can generate free text but they are not suitable for paraphrase generation for a given sentence. We address this problem by conditioning the both, encoder and decoder sides of VAE, on the original sentence, so that it can generate the given sentence's paraphrases. Unlike most existing models, our model is simple, modular and can generate multiple paraphrases, for a given sentence. Quantitative evaluation of the…

Unsupervised Learning of Object Keypoints for Perception and Control

Tejas Kulkarni, Ankush Gupta, Catalin Ionescu +4 more

cs.CV · · cited 222× · arXiv ↗ · more like this ·

The study of object representations in computer vision has primarily focused on developing representations that are useful for image classification, object detection, or semantic segmentation as downstream tasks. In this work we aim to learn object representations that are useful for control and reinforcement learning (RL). To this end, we introduce Transporter, a neural network architecture for discovering concise geometric object representations in terms of keypoints or image-space coordinates. Our method learns from raw video frames in a fully unsupervised manner, by transporting learnt image features between video frames using a keypoint bottleneck. The discovered keypoints track objects and object parts across long time-horizons more accurately than recent similar methods. Furthermore, consistent long-term tracking enables two notable results in control domains -- (1) using the…

SuS-X: Training-Free Name-Only Transfer of Vision-Language Models

Vishaal Udandarao, Ankush Gupta, Samuel Albanie

cs.CV · · cited 174× · arXiv ↗ · more like this ·

Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models. CLIP demonstrates impressive zero-shot classification and retrieval on diverse downstream tasks. However, to leverage its full potential, fine-tuning still appears to be necessary. Fine-tuning the entire CLIP model can be resource-intensive and unstable. Moreover, recent methods that aim to circumvent this need for fine-tuning still require access to images from the target distribution. In this paper, we pursue a different approach and explore the regime of training-free "name-only transfer" in which the only knowledge we possess about the downstream task comprises the names of downstream target categories. We propose a novel method, SuS-X, consisting of two key building blocks -- SuS and TIP-X, that requires neither intensive fine-tuning nor costly…

Towards Automatic Resource Bound Analysis for OCaml

Jan Hoffmann, Ankush Das, Shu-Chun Weng

cs.PL · · cited 142× · arXiv ↗ · more like this ·

This article presents a resource analysis system for OCaml programs. This system automatically derives worst-case resource bounds for higher-order polymorphic programs with user-defined inductive types. The technique is parametric in the resource and can derive bounds for time, memory allocations and energy usage. The derived bounds are multivariate resource polynomials which are functions of different size parameters that depend on the standard OCaml types. Bound inference is fully automatic and reduced to a linear optimization problem that is passed to an off-the-shelf LP solver. Technically, the analysis system is based on a novel multivariate automatic amortized resource analysis (AARA). It builds on existing work on linear AARA for higher-order programs with user-defined inductive types and on multivariate AARA for first-order programs with built-in lists and binary trees. For the…

Choosing Linguistics over Vision to Describe Images

Ankush Gupta, Yashaswi Verma, C. V. Jawahar

Computer Vision and Pattern Recognition · Proceedings of the AAAI Conference on Artificial Intelligence · · cited 126× · DOI ↗ · more like this ·

In this paper, we address the problem of automatically generating human-like descriptions for unseen images, given a collection of images and their corresponding human-generated descriptions. Previous attempts for this task mostly rely on visual clues and corpus statistics, but do not take much advantage of the semantic information inherent in the available image descriptions. Here, we present a generic method which benefits from all these three sources (i.e. visual clues, corpus statistics and available descriptions) simultaneously, and is capable of constructing novel descriptions. Our approach works on syntactically and linguistically motivated phrases extracted from the human descriptions. Experimental evaluations demonstrate that our formulation mostly generates lucid and semantically correct descriptions, and significantly outperforms the previous methods on automatic evaluation…

The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein +7 more

cs.CL · · cited 123× · arXiv ↗ · more like this ·

Automatic evaluation of machine translation (MT) is a critical tool driving the rapid iterative development of MT systems. While considerable progress has been made on estimating a single scalar quality score, current metrics lack the informativeness of more detailed schemes that annotate individual errors, such as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap by proposing AutoMQM, a prompting technique which leverages the reasoning and in-context learning capabilities of large language models (LLMs) and asks them to identify and categorize errors in translations. We start by evaluating recent LLMs, such as PaLM and PaLM-2, through simple score prediction prompting, and we study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just…

P

Ankush Desai, Vivek Gupta, Ethan K. Jackson +3 more

Software · · cited 123× · DOI ↗ · more like this ·

We describe the design and implementation of P, a domain-specific language to write asynchronous event driven code. P allows the programmer to specify the system as a collection of interacting state machines, which communicate with each other using events. P unifies modeling and programming into one activity for the programmer. Not only can a P program be compiled into executable code, but it can also be tested using model checking techniques. P allows the programmer to specify the environment, used to "close" the system during testing, as nondeterministic ghost machines. Ghost machines are erased during compilation to executable code; a type system ensures that the erasure is semantics preserving.

An Adaptive Differential Evolution Algorithm for Global Optimization in Dynamic Environments

Swagatam Das, Ankush Mandal, Rohan Mukherjee

Artificial Intelligence · IEEE Transactions on Cybernetics · · cited 114× · DOI ↗ · more like this ·

This article proposes a multipopulation-based adaptive differential evolution (DE) algorithm to solve dynamic optimization problems (DOPs) in an efficient way. The algorithm uses Brownian and adaptive quantum individuals in conjunction with the DE individuals to maintain the diversity and exploration ability of the population. This algorithm, denoted as dynamic DE with Brownian and quantum individuals (DDEBQ), uses a neighborhood-driven double mutation strategy to control the perturbation and thereby prevents the algorithm from converging too quickly. In addition, an exclusion rule is used to spread the subpopulations over a larger portion of the search space as this enhances the optima tracking ability of the algorithm. Furthermore, an aging mechanism is incorporated to prevent the algorithm from stagnating at any local optimum. The performance of DDEBQ is compared with several…

DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documents

Kunal Dahiya, Deepak Saini, Anshul Mittal +6 more

cs.LG · · cited 111× · arXiv ↗ · more like this ·

Scalability and accuracy are well recognized challenges in deep extreme multi-label learning where the objective is to train architectures for automatically annotating a data point with the most relevant subset of labels from an extremely large label set. This paper develops the DeepXML framework that addresses these challenges by decomposing the deep extreme multi-label task into four simpler sub-tasks each of which can be trained accurately and efficiently. Choosing different components for the four sub-tasks allows DeepXML to generate a family of algorithms with varying trade-offs between accuracy and scalability. In particular, DeepXML yields the Astec algorithm that could be 2-12% more accurate and 5-30x faster to train than leading deep extreme classifiers on publically available short text datasets. Astec could also efficiently train on Bing short text datasets containing up to…

Temporal Query Networks for Fine-grained Video Understanding

Chuhan Zhang, Ankush Gupta, Andrew Zisserman

cs.CV · · cited 111× · arXiv ↗ · more like this ·

Our objective in this work is fine-grained classification of actions in untrimmed videos, where the actions may be temporally extended or may span only a few frames of the video. We cast this into a query-response mechanism, where each query addresses a particular question, and has its own response label set. We make the following four contributions: (I) We propose a new model - a Temporal Query Network - which enables the query-response functionality, and a structural understanding of fine-grained actions. It attends to relevant segments for each query with a temporal attention mechanism, and can be trained using only the labels for each query. (ii) We propose a new way - stochastic feature bank update - to train a network on videos of various lengths with the dense sampling required to respond to fine-grained queries. (iii) We compare the TQN to other architectures and text…

Blockchain-Enabled Authenticated Key Agreement Scheme for Mobile Vehicles-Assisted Precision Agricultural IoT Networks

Anusha Vangala, Ashok Kumar Das, Ankush Mitra +2 more

Computer Networks and Communications · IEEE Transactions on Information Forensics and Security · · cited 109× · DOI ↗ · more like this ·

Precision farming has a positive potential in the agricultural industry regarding water conservation, increased productivity, better development of rural areas, and increased income. Blockchain technology is a better alternative for storing and sharing farm data as it is reliable, transparent, immutable, and decentralized. Remote monitoring of an agricultural field requires security systems to ensure that any sensitive information is exchanged only among authenticated entities in the network. To this end, we design an efficient blockchain-enabled authenticated key agreement scheme for mobile vehicles-assisted precision agricultural Internet of Things (IoT) networks called <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$AgroMobiBlock$ </tex-math></inline-formula> . The limited existing work on…

Combining Neural, Statistical and External Features for Fake News Stance Identification

Gaurav Bhatt, Aman Sharma, Shivam Sharma +3 more

Sociology and Political Science · · cited 104× · DOI ↗ · more like this ·

Identifying the veracity of a news article is an interesting problem while automating this process can be a challenging task. Detection of a news article as fake is still an open question as it is contingent on many factors which the current state-of-the-art models fail to incorporate. In this paper, we explore a subtask to fake news identification, and that is stance detection. Given a news article, the task is to determine the relevance of the body and its claim. We present a novel idea that combines the neural, statistical and external features to provide an efficient solution to this problem. We compute the neural embedding from the deep recurrent model, statistical features from the weighted n-gram bag-of-words model and handcrafted external features with the help of feature engineering heuristics. Finally, using deep neural layer all the features are combined, thereby classifying…

Do Current Multi-Task Optimization Methods in Deep Learning Even Help?

Derrick Xin, Behrooz Ghorbani, Ankush Garg +2 more

cs.LG · · cited 91× · arXiv ↗ · more like this ·

Recent research has proposed a series of specialized optimization algorithms for deep multi-task models. It is often claimed that these multi-task optimization (MTO) methods yield solutions that are superior to the ones found by simply optimizing a weighted average of the task losses. In this paper, we perform large-scale experiments on a variety of language and vision tasks to examine the empirical validity of these claims. We show that, despite the added design and computational complexity of these algorithms, MTO methods do not yield any performance improvements beyond what is achievable via traditional optimization approaches. We highlight alternative strategies that consistently yield improvements to the performance profile and point out common training pitfalls that might cause suboptimal results. Finally, we outline challenges in reliably evaluating the performance of MTO…

A control-theoretic approach to dynamic voltage scheduling

Ankush Varma, Brinda Ganesh, Mainak Sen +3 more

Hardware and Architecture · · cited 90× · DOI ↗ · more like this ·

The development of energy-conscious embedded and/or mobile systems exposes a trade-off between energy consumption and system performance. Recent microprocessors have incorporated dynamic voltage scaling as a tool that system software can use to explore this trade-off. Developing appropriate heuristics to control this feature is a non-trivial venture; as has been shown in the past, voltage-scaling heuristics that closely track perceived performance requirements do not save much energy, while those that save the most energy tend to do so at the expense of performance resulting in poor response time, for example. We note that the task of dynamically scaling processor speed and voltage to meet changing performance requirements resembles a classical control-systems problem, and so we apply a bit of control theory to the task in order to define a new voltage-scaling algorithm. We find that…