arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Sonal Kumar

55 papers · 2009–2026

2009: 12010: 12011: 32012: 12013: 12014: 12015: 12016: 02017: 02018: 12019: 02020: 12021: 22022: 22023: 102024: 172025: 82026: 5
2009peak 17 in 20242026

Frequent co-authors

Papers

showing 1–25 of 55 RSS

GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities

Sreyan Ghosh, Sonal Kumar, Ashish Seth +6 more

cs.SD · · cited 234× · arXiv ↗ · more like this ·

Perceiving and understanding non-speech sounds and non-verbal speech is essential to making decisions that help us interact with our surroundings. In this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model (LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We build GAMA by integrating an LLM with multiple types of audio representations, including features from a custom Audio Q-Former, a multi-layer aggregator that aggregates features from multiple layers of an audio encoder. We fine-tune GAMA on a large-scale audio-language dataset, which augments it with audio understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for Complex Audio Reasoning), a synthetically generated instruction-tuning (IT) dataset with instructions that require the model to perform complex reasoning on the input audio. We instruction-tune GAMA with…

AV-RIR: Audio-Visual Room Impulse Response Estimation

Anton Ratnarajah, Sreyan Ghosh, Sonal Kumar +2 more

cs.SD · · cited 41× · arXiv ↗ · more like this ·

Accurate estimation of Room Impulse Response (RIR), which captures an environment's acoustic properties, is important for speech processing and AR/VR applications. We propose AV-RIR, a novel multi-modal multi-task learning approach to accurately estimate the RIR from a given reverberant speech signal and the visual cues of its corresponding environment. AV-RIR builds on a novel neural codec-based architecture that effectively captures environment geometry and materials properties and solves speech dereverberation as an auxiliary task by using multi-task learning. We also propose Geo-Mat features that augment material information into visual cues and CRIP that improves late reverberation components in the estimated RIR via image-to-RIR retrieval by 86%. Empirical results show that AV-RIR quantitatively outperforms previous audio-only and visual-only approaches by achieving 36% - 63%…

RECAP: Retrieval-Augmented Audio Captioning

Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru +2 more

eess.AS · · cited 41× · arXiv ↗ · more like this ·

We present RECAP (REtrieval-Augmented Audio CAPtioning), a novel and effective audio captioning system that generates captions conditioned on an input audio and other captions similar to the audio retrieved from a datastore. Additionally, our proposed method can transfer to any domain without the need for any additional fine-tuning. To generate a caption for an audio sample, we leverage an audio-text model CLAP to retrieve captions similar to it from a replaceable datastore, which are then used to construct a prompt. Next, we feed this prompt to a GPT-2 decoder and introduce cross-attention layers between the CLAP encoder and GPT-2 to condition the audio for caption generation. Experiments on two benchmark datasets, Clotho and AudioCaps, show that RECAP achieves competitive performance in in-domain settings and significant improvements in out-of-domain settings. Additionally, due to its…

DALE: Generative Data Augmentation for Low-Resource Legal NLP

Sreyan Ghosh, Chandra Kiran Evuru, Sonal Kumar +4 more

cs.CL · · cited 29× · arXiv ↗ · more like this ·

We present DALE, a novel and effective generative Data Augmentation framework for low-resource LEgal NLP. DALE addresses the challenges existing frameworks pose in generating effective data augmentations of legal documents - legal language, with its specialized vocabulary and complex semantics, morphology, and syntax, does not benefit from data augmentations that merely rephrase the source sentence. To address this, DALE, built on an Encoder-Decoder Language Model, is pre-trained on a novel unsupervised text denoising objective based on selective masking - our masking strategy exploits the domain-specific language characteristics of templatized legal documents to mask collocated spans of text. Denoising these spans helps DALE acquire knowledge about legal concepts, principles, and language usage. Consequently, it develops the ability to generate coherent and diverse augmentations with…

CoSyn: Detecting Implicit Hate Speech in Online Conversations Using a Context Synergized Hyperbolic Network

Sreyan Ghosh, Manan Suri, Purva Chiniya +3 more

cs.LG · · cited 26× · arXiv ↗ · more like this ·

The tremendous growth of social media users interacting in online conversations has led to significant growth in hate speech, affecting people from various demographics. Most of the prior works focus on detecting explicit hate speech, which is overt and leverages hateful phrases, with very little work focusing on detecting hate speech that is implicit or denotes hatred through indirect or coded language. In this paper, we present CoSyn, a context-synergized neural network that explicitly incorporates user- and conversational context for detecting implicit hate speech in online conversations. CoSyn introduces novel ways to encode these external contexts and employs a novel context interaction mechanism that clearly captures the interplay between them, making independent assessments of the amounts of information to be retrieved from these noisy contexts. Additionally, it carries out all…

BioAug: Conditional Generation based Data Augmentation for Low-Resource Biomedical NER

Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar +1 more

cs.CL · · cited 22× · arXiv ↗ · more like this ·

Biomedical Named Entity Recognition (BioNER) is the fundamental task of identifying named entities from biomedical text. However, BioNER suffers from severe data scarcity and lacks high-quality labeled data due to the highly specialized and expert knowledge required for annotation. Though data augmentation has shown to be highly effective for low-resource NER in general, existing data augmentation techniques fail to produce factual and diverse augmentations for BioNER. In this paper, we present BioAug, a novel data augmentation framework for low-resource BioNER. BioAug, built on BART, is trained to solve a novel text reconstruction task based on selective masking and knowledge augmentation. Post training, we perform conditional generation and generate diverse augmentations conditioning BioAug on selectively corrupted text similar to the training stage. We demonstrate the effectiveness…

ACLM: A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER

Sreyan Ghosh, Utkarsh Tyagi, Manan Suri +3 more

cs.CL · · cited 9× · arXiv ↗ · more like this ·

Complex Named Entity Recognition (NER) is the task of detecting linguistically complex named entities in low-context text. In this paper, we present ACLM Attention-map aware keyword selection for Conditional Language Model fine-tuning), a novel data augmentation approach based on conditional generation to address the data scarcity problem in low-resource complex NER. ACLM alleviates the context-entity mismatch issue, a problem existing NER data augmentation techniques suffer from and often generates incoherent augmentations by placing complex named entities in the wrong context. ACLM builds on BART and is optimized on a novel text reconstruction or denoising task - we use selective masking (aided by attention maps) to retain the named entities and certain keywords in the input sentence that provide contextually relevant additional knowledge or hints about the named entities. Compared…

DatUS: Data-Driven Unsupervised Semantic Segmentation With Pretrained Self-Supervised Vision Transformer

Sonal Kumar, Arijit Sur, Rashmi Dutta Baruah

Computer Vision and Pattern Recognition · IEEE Transactions on Cognitive and Developmental Systems · · cited 4× · DOI ↗ · more like this ·

Successive proposals of several self-supervised training schemes continue to emerge, taking one step closer to developing a universal foundation model. In this process, unsupervised downstream tasks are recognized as one of the evaluation methods to validate the quality of visual features learned with self-supervised training. However, unsupervised dense semantic segmentation has yet to be explored as a downstream task, which can utilize and evaluate the quality of semantic information introduced in patch-level feature representations during self-supervised training of vision transformers. Therefore, we propose a novel data-driven framework, DatUS, to perform unsupervised dense semantic segmentation as a downstream task. DatUS generates semantically consistent pseudo-segmentation masks for an unlabeled image dataset without using visual-prior or synchronized data. The experiment shows…

“The Reynolds–McElhenney effect?”: effect of celebrity endorsement on colonoscopy-related videos on TikTok

Ali Lahooti, Brian Critelli, Amier Hassan +8 more

Health · iGIE · · cited 4× · DOI ↗ · more like this ·

Background and Aims: Cases of young-onset colorectal cancer (CRC) have risen substantially over the last 20 years, and physicians are increasingly harnessing the influence of celebrities to raise awareness for CRC screening. Celebrity actors Ryan Reynolds and Rob McElhenney filmed their experiences undergoing colonoscopy as part of an initiative to raise awareness for CRC screening. Herein, we assess the quality of information on TikTok regarding colonoscopies and evaluate the effect of this celebrity endorsement. Methods: Hashtags encompassing colonoscopy were searched using TikTok's algorithm, and 75 videos were considered. The DISCERN questionnaire and analytics were used to quantify the quality and popularity of each video. Results: < .05) after the celebrity Reynolds-McElhenney endorsement compared with those released before. After the endorsement, the difference between the…

Enabling web services for Classification of Satellite Image.

Sonal Kumar, Daya Gupta, V. K. Panchal +1 more

Artificial Intelligence · SWWS · · cited 4× · DOI ↗ · more like this ·

The basic characteristics of the electroencephalograms (EEGs) of the entorhinal cortex (EC) were compared to those of the olfactory bulb (OB) and prepiriform cortex (PPC) in the awake cat, using methods that take into account linear (coherence functions) and non-linear relationships (mutual information). The frequency spectra of EC signals differ from those of EEG signals recorded from OB or PPC, but in the beta frequency band (35-40 Hz) coherent activity between these latter two areas and the EC could be put in evidence. Taking the values of coherence as a measure of linear coupling between populations of neurons, this coupling is strong between PPC and the rostral part of the EC, and becomes weaker towards more lateral, caudal and medial areas. This decrease in coupling is confirmed by non-linear analysis methods. On the basis of intracortical depth profiles it was shown that the beta…

SteerDuplex: Steerable Duplex Speech Dialogue Models

Utkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai +13 more

cs.AI · · cited 3× · arXiv ↗ · more like this ·

Full-duplex spoken dialogue models support low-latency turn taking, interruption handling, and backchanneling, yet a key capability remains underexplored: steerability, the ability to reliably shift conversational behavior along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. We introduce a taxonomy of text- and audio-based steerability that identifies substantial gaps in current full-duplex models. To address this gap, we introduce SteerDuplex, a Moshi-based full-duplex speech model fine-tuned on natural conversations and synthetic dialogues targeting instruction following, vocal delivery, reasoning, and duplex interaction. We further apply two-stage reinforcement learning (RL) with hybrid rewards, combining verifiable interaction checks and judge-based semantic feedback to improve timing and response continuity. To evaluate…

ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds

Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru +3 more

eess.AS · · cited 3× · arXiv ↗ · more like this ·

Open-vocabulary audio-language models, like CLAP, offer a promising approach for zero-shot audio classification (ZSAC) by enabling classification with any arbitrary set of categories specified with natural language prompts. In this paper, we propose a simple but effective method to improve ZSAC with CLAP. Specifically, we shift from the conventional method of using prompts with abstract category labels (e.g., Sound of an organ) to prompts that describe sounds using their inherent descriptive features in a diverse context (e.g.,The organ's deep and resonant tones filled the cathedral.). To achieve this, we first propose ReCLAP, a CLAP model trained with rewritten audio captions for improved understanding of sounds in the wild. These rewritten captions describe each sound event in the original caption using their unique discriminative characteristics. ReCLAP outperforms all baselines on…

ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions

Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar +4 more

cs.CL · · cited 3× · arXiv ↗ · more like this ·

We present ABEX, a novel and effective generative data augmentation methodology for low-resource Natural Language Understanding (NLU) tasks. ABEX is based on ABstract-and-EXpand, a novel paradigm for generating diverse forms of an input document -- we first convert a document into its concise, abstract description and then generate new documents based on expanding the resultant abstraction. To learn the task of expanding abstract descriptions, we first train BART on a large-scale synthetic dataset with abstract-document pairs. Next, to generate abstract descriptions for a document, we propose a simple, controllable, and training-free method based on editing AMR graphs. ABEX brings the best of both worlds: by expanding from abstract representations, it preserves the original semantic properties of the documents, like style and meaning, thereby maintaining alignment with the original…

A Closer Look at the Limitations of Instruction Tuning

Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar +5 more

cs.CL · · cited 3× · arXiv ↗ · more like this ·

Instruction Tuning (IT), the process of training large language models (LLMs) using instruction-response pairs, has emerged as the predominant method for transforming base pre-trained LLMs into open-domain conversational agents. While IT has achieved notable success and widespread adoption, its limitations and shortcomings remain underexplored. In this paper, through rigorous experiments and an in-depth analysis of the changes LLMs undergo through IT, we reveal various limitations of IT. In particular, we show that (1) IT fails to enhance knowledge or skills in LLMs. LoRA fine-tuning is limited to learning response initiation and style tokens, and full-parameter fine-tuning leads to knowledge degradation. (2) Copying response patterns from IT datasets derived from knowledgeable sources leads to a decline in response quality. (3) Full-parameter fine-tuning increases hallucination by…

Span Classification with Structured Information for Disfluency Detection in Spoken Utterances

Sreyan Ghosh, Sonal Kumar, Yaman Kumar Singla +2 more

cs.CL · · cited 3× · arXiv ↗ · more like this ·

Existing approaches in disfluency detection focus on solving a token-level classification task for identifying and removing disfluencies in text. Moreover, most works focus on leveraging only contextual information captured by the linear sequences in text, thus ignoring the structured information in text which is efficiently captured by dependency trees. In this paper, building on the span classification paradigm of entity recognition, we propose a novel architecture for detecting disfluencies in transcripts from spoken utterances, incorporating both contextual information through transformers and long-distance structured information captured by dependency trees, through graph convolutional networks (GCNs). Experimental results show that our proposed model achieves state-of-the-art results on the widely used English Switchboard for disfluency detection and outperforms prior-art by a…

Learning a Variety of Return Periods in Digital Altitude based Flood Risk and Liability

S. Dhamodaran, Kumar Shivam, Sonal Kumar Pandit

Information Systems · Indian Journal of Science and Technology · · cited 3× · DOI ↗ · more like this ·

Background/Objectives: As the world is advancing to its peaks the issue of flood in various regions, which leads to a very dangerous problem in the field of crop cultivation. Farmers cannot compare the characteristics of crop and the property the soil due to rain for the cultivation. The issue of cultivation of crop arises according to the rainfall in the area. Methods/ Statistical Analysis: In that case there is a need for effective analysis and suggestion is needed. Thereby it provides the spirit of encouragement and more profit in cultivation. This journal portrays the flood hazard mapping, crops and settlement vulnerability assessment in a low laying reverie flood prone area for different flood magnitudes. This system is used for analyzing the best crop that can be used by farmers, which need to be cultivated easily in given prone areas. The analysis of frequency has been carried…

EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning

Ashish Seth, Ramaneswaran Selvakumar, S Sakshi +3 more

cs.SD · · cited 1× · arXiv ↗ · more like this ·

In this paper, we present EH-MAM (Easy-to-Hard adaptive Masked Acoustic Modeling), a novel self-supervised learning approach for speech representation learning. In contrast to the prior methods that use random masking schemes for Masked Acoustic Modeling (MAM), we introduce a novel selective and adaptive masking strategy. Specifically, during SSL training, we progressively introduce harder regions to the model for reconstruction. Our approach automatically selects hard regions and is built on the observation that the reconstruction loss of individual frames in MAM can provide natural signals to judge the difficulty of solving the MAM pre-text task for that frame. To identify these hard regions, we employ a teacher model that first predicts the frame-wise losses and then decides which frames to mask. By learning to create challenging problems, such as identifying harder frames and…

LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition

Sreyan Ghosh, Sonal Kumar, Ashish Seth +4 more

eess.AS · · cited 1× · arXiv ↗ · more like this ·

Visual cues, like lip motion, have been shown to improve the performance of Automatic Speech Recognition (ASR) systems in noisy environments. We propose LipGER (Lip Motion aided Generative Error Correction), a novel framework for leveraging visual cues for noise-robust ASR. Instead of learning the cross-modal correlation between the audio and visual modalities, we make an LLM learn the task of visually-conditioned (generative) ASR error correction. Specifically, we instruct an LLM to predict the transcription from the N-best hypotheses generated using ASR beam-search. This is further conditioned on lip motions. This approach addresses key challenges in traditional AVSR learning, such as the lack of large-scale paired datasets and difficulties in adapting to new domains. We experiment on 4 datasets in various settings and show that LipGER improves the Word Error Rate in the range of…

CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP

Chandra Kiran Reddy Evuru, Sreyan Ghosh, Sonal Kumar +3 more

cs.CL · · cited 1× · arXiv ↗ · more like this ·

We present CoDa (Constrained Generation based Data Augmentation), a controllable, effective, and training-free data augmentation technique for low-resource (data-scarce) NLP. Our approach is based on prompting off-the-shelf instruction-following Large Language Models (LLMs) for generating text that satisfies a set of constraints. Precisely, we extract a set of simple constraints from every instance in the low-resource dataset and verbalize them to prompt an LLM to generate novel and diverse training instances. Our findings reveal that synthetic data that follows simple constraints in the downstream dataset act as highly effective augmentations, and CoDa can achieve this without intricate decoding-time constrained generation techniques or fine-tuning with complex algorithms that eventually make the model biased toward the small number of training instances. Additionally, CoDa is the…

Do Vision-Language Models Understand Compound Nouns?

Sonal Kumar, Sreyan Ghosh, S Sakshi +2 more

cs.CV · · cited 1× · arXiv ↗ · more like this ·

Open-vocabulary vision-language models (VLMs) like CLIP, trained using contrastive loss, have emerged as a promising new paradigm for text-to-image retrieval. However, do VLMs understand compound nouns (CNs) (e.g., lab coat) as well as they understand nouns (e.g., lab)? We curate Compun, a novel benchmark with 400 unique and commonly used CNs, to evaluate the effectiveness of VLMs in interpreting CNs. The Compun benchmark challenges a VLM for text-to-image retrieval where, given a text prompt with a CN, the task is to select the correct image that shows the CN among a pair of distractor images that show the constituent nouns that make up the CN. Next, we perform an in-depth analysis to highlight CLIPs' limited understanding of certain types of CNs. Finally, we present an alternative framework that moves beyond hand-written templates for text prompts widely used by CLIP-like models. We…

IPCL: Iterative Pseudo-Supervised Contrastive Learning to Improve Self-Supervised Feature Representation

Sonal Kumar, Anirudh Phukan, Arijit Sur

Artificial Intelligence · · cited 1× · DOI ↗ · more like this ·

Self-supervised learning with a contrastive batch approach has become a powerful tool for representation learning in computer vision. The performance of downstream tasks is proportional to the quality of visual features learned while self-supervised pre-training. The existing contrastive batch approaches heavily depend on data augmentation to learn latent information from unlabelled datasets. We argue that introducing the dataset’s intra-class variation in a contrastive batch approach improves visual representation quality further. In this paper, we propose a novel self-supervised learning approach named Iterative Pseudo-supervised Contrastive Learning (IPCL), which utilizes a balanced combination of image augmentations and pseudo-class information to improve the visual representation iteratively. Experimental results illustrate that our proposed method surpasses the baseline…

CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models

Sreyan Ghosh, Ashish Seth, Sonal Kumar +7 more

cs.SD · · cited 1× · arXiv ↗ · more like this ·

A fundamental characteristic of audio is its compositional nature. Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP) that learns a shared representation between audio and language modalities have improved performance in many downstream applications, including zero-shot audio classification, audio retrieval, etc. However, the ability of these models to effectively perform compositional reasoning remains largely unexplored and necessitates additional research. In this paper, we propose CompA, a collection of two expert-annotated benchmarks with a majority of real-world audio samples, to evaluate compositional reasoning in ALMs. Our proposed CompA-order evaluates how well an ALM understands the order or occurrence of acoustic events in audio, and CompA-attribute evaluates attribute-binding of acoustic events. An instance from either benchmark consists of two…

Problem Based Learning: An Experience of B.P. Koirala Institute of Health Sciences, Nepal

Prakash Baral, S Koirala, Suraj Shrestha +3 more

Education · Journal of Gandaki Medical College-Nepal · · cited 1× · DOI ↗ · more like this ·

Introduction: Problem-based learning (PBL) is a kind of teaching- learning method in which students’ role is prime and they learn about a subject by active participation in problem solving.Methods: A total of 96 students studying in second year MBBS program were enrolled into present study. A questionnaire was given to students and they were asked to choose the option they felt about PBL methods. The questionnaire comprised of ten likert items out of which three were negative and seven were positive items.Results: Mean score for each statement in the questionnaire and the overall mean score were calculated. Overall mean score was 3.47.Conclusions: It was concluded that students of second year MBBS at BPKIHS, had a positive attitude towards PBL method of teaching-learning activity. J-GMC-N | Volume 11 | Issue 01 | January-June 2018, Page: 42-45

SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays

Sonal Kumar, Sinan Hersek, Artem Dementyev +6 more

cs.SD · · cited 0× · arXiv ↗ · more like this ·

Embodied, ego-centric intelligence fundamentally requires the ability to comprehend spatial audio within complex environments. While large audio-language models excel at mono-channel reasoning, they lack spatial awareness, discarding critical spatial cues that enable sound localization and that can improve the disentanglement of overlapping sound sources. To address this, we present SEA-LM, a Spatial Audio Understanding model. First, we introduce FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. We train a Multimodal Large Language Model (MLLM) to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially…

Unsupervised Contrastive Refinement with Graph-Aware Multimodal Interaction for Radiology Report Generation

Akshay Daydar, Mr.G. Chenna Keshava Reddy, Sonal Kumar +3 more

Artificial Intelligence · IEEE Journal of Biomedical and Health Informatics · · cited 0× · DOI ↗ · more like this ·

Automatic Radiology Report Generation (RRG) reduces medical workload by automatically generating diagnostic reports from X-ray images. In recent studies, an improved performance has been observed by first classifying the image and then incorporating clinical outcomes into the RRG pipeline, with cross-modal interactions between image and text. However, these studies have limitations, including : (1) inconsistent cross-modal learning due to under-utilization of textual information, and (2) reliance on noisy classification labels, resulting in inaccurate attention and invariant report generation. To address these issues, a Medical Context-aware RRG Framework (MedCARF) is proposed, which introduces a Disease-Aware Visual Textual Alignment (D-ViTAlign) module to capture organ-abnormality relationships from scene graphs and visual features. In addition, label refinement is achieved using an…