arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Mishra

7,984 papers · 1960–2024

1965: 01966: 01967: 11968: 11969: 11970: 41971: 01972: 01973: 11974: 01975: 11976: 21977: 81978: 31979: 21980: 41981: 111982: 41983: 51984: 61985: 91986: 81987: 91988: 31989: 51990: 51991: 111992: 161993: 161994: 181995: 151996: 241997: 271998: 271999: 262000: 272001: 422002: 992003: 762004: 762005: 912006: 1182007: 1152008: 1832009: 2152010: 3282011: 4212012: 4942013: 5772014: 6642015: 6402016: 5902017: 2502018: 2582019: 3072020: 3952021: 4292022: 4752023: 4752024: 364
1965peak 664 in 20142024

Frequent co-authors

Papers

showing 1–25 of 7,984 RSS

Llama 2: Open Foundation and Fine-Tuned Chat Models

Hugo Touvron, Louis Martin, Kevin Stone +65 more

cs.CL · · cited 18,418× · arXiv ↗ · more like this ·

In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.

Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge

Punya Mishra, Matthew J. Koehler

Education · Teachers College Record The Voice of Scholarship in Education · · cited 9,666× · DOI ↗ · more like this ·

Research in the area of educational technology has often been critiqued for a lack of theoretical grounding. In this article we propose a conceptual framework for educational technology by building on Shulrnan's formulation of “pedagogical content knowledge” and extend it to the phenomenon of teachers integrating technology into their pedagogy. This framework is the result of 5 years of work on a program of research focused on teacher professional development and faculty development in higher education. It attempts to capture some of the essential qualities of teacher knowledge required for technology integration in teaching, while addressing the complex, multifaceted, and situated nature of this knowledge. We argue, briefly, that thoughtful pedagogical uses of technology require the development of a complex, situated form of knowledge that we call Technological Pedagogical Content…

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin +64 more

cs.CL · · cited 8,437× · arXiv ↗ · more like this ·

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite…

Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, Shayne Longpre +32 more

cs.LG · · cited 4,465× · arXiv ↗ · more like this ·

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve…

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra +4 more

cs.CL · · cited 3,696× · arXiv ↗ · more like this ·

Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with…

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

BigScience Workshop, :, Teven Le Scao +391 more

cs.CL · · cited 3,043× · arXiv ↗ · more like this ·

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we…

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Pan Lu, Swaroop Mishra, Tony Xia +6 more

cs.CL · · cited 2,714× · arXiv ↗ · more like this ·

When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (ScienceQA), a new benchmark that consists of ~21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We further design language models to learn to generate lectures and…

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448 more

cs.CL · · cited 2,670× · arXiv ↗ · more like this ·

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of…

Implementation issues in spectrum sensing for cognitive radios

Danijela Čabrić, Shridhar Mubaraq Mishra, R.W. Brodersen

Computer Networks and Communications · · cited 2,597× · DOI ↗ · more like this ·

There are new system implementation challenges involved in the design of cognitive radios, which have both the ability to sense the spectral environment and the flexibility to adapt transmission parameters to maximize system capacity while coexisting with legacy wireless networks. The critical design problem is the need to process multigigahertz wide bandwidth and reliably detect presence of primary users. This places severe requirements on sensitivity, linearity and dynamic range of the circuitry in the RF front-end. To improve radio sensitivity of the sensing function through processing gain we investigated three digital signal processing techniques: matched filtering, energy detection and cyclostationary feature detection. Our analysis shows that cyclostationary feature detection has advantages due to its ability to differentiate modulated signals, interference and noise in low…

Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

Kai Greshake, Sahar Abdelnabi, Shailesh Mishra +3 more

cs.CR · · cited 2,116× · arXiv ↗ · more like this ·

Large Language Models (LLMs) are increasingly being integrated into various applications. The functionalities of recent LLMs can be flexibly modulated via natural language prompts. This renders them susceptible to targeted adversarial prompting, e.g., Prompt Injection (PI) attacks enable attackers to override original instructions and employed controls. So far, it was assumed that the user is directly prompting the LLM. But, what if it is not the user prompting? We argue that LLM-Integrated Applications blur the line between data and instructions. We reveal new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts…

Technological Pedagogical Content Knowledge (TPACK)

Denise Schmidt, Evrim Baran, Ann D. Thompson +3 more

Education · Journal of Research on Technology in Education · · cited 1,785× · DOI ↗ · more like this ·

Based in Shulman’s idea of Pedagogical Content Knowledge, Technological Pedagogical Content Knowledge (TPACK) has emerged as a useful frame for describing and understanding the goals for technology use in preservice teacher education. This paper addresses the need for a survey instrument designed to assess TPACK for preservice teachers. The paper describes survey development process and results from a pilot study on 124 preservice teachers. Data analysis procedures included Cronbach’s alpha statistics on the TPACK knowledge domains and factor analysis for each domain. Results suggest that, with the modification and/or deletion of 18 of the survey items, the survey is a reliable and valid instrument that will help educators design longitudinal studies to assess preservice teachers’ development of TPACK.

What Is Technological Pedagogical Content Knowledge

Matthew J. Koehler, Punya Mishra

Education · Contemporary Issues in Technology and Teacher Education Journal · · cited 1,705× · OpenAlex ↗ · more like this ·

This paper describes a framework for teacher knowledge for technology integration called technological pedagogical content knowledge (originally TPCK, now known as TPACK, or technology, pedagogy, and content knowledge). This framework builds on Lee Shulman’s construct of pedagogical content knowledge (PCK) to include technology knowledge. The development of TPACK by teachers is critical to effective teaching with technology. The paper begins with a brief introduction to the complex, illstructured nature of teaching. The nature of technologies (both analog and digital) is considered, as well as how the inclusion of technology in pedagogy further complicates teaching. The TPACK framework for teacher knowledge is described in detail, as a complex interaction among three bodies of knowledge: Content, pedagogy, and technology. The interaction of these bodies of knowledge, both theoretically…

PaLM 2 Technical Report

Rohan Anil, Andrew M. Dai, Orhan Firat +125 more

cs.CL · · cited 1,563× · arXiv ↗ · more like this ·

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables…

Cooperative Sensing among Cognitive Radios

Shridhar Mubaraq Mishra, Anant Sahai, R.W. Brodersen

Computer Networks and Communications · 2006 IEEE International Conference on Communications · · cited 1,459× · DOI ↗ · more like this ·

Cognitive Radios have been advanced as a technology for the opportunistic use of under-utilized spectrum since they are able to sense the spectrum and use frequency bands if no Primary user is detected. However, the required sensitivity is very demanding since any individual radio might face a deep fade. We propose light-weight cooperation in sensing based on hard decisions to mitigate the sensitivity requirements on individual radios. We show that the "link budget" that system designers have to reserve for fading is a significant function of the required probability of detection. Even a few cooperating users (~10-20) facing independent fades are enough to achieve practical threshold levels by drastically reducing individual detection requirements. Hard decisions perform almost as well as soft decisions in achieving these gains. Cooperative gains in a environment where shadowing is…

StarCoder: may the source be with you!

Raymond Li, Loubna Ben Allal, Yangtian Zi +64 more

cs.CL · · cited 1,336× · arXiv ↗ · more like this ·

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python…

Large Language Models Cannot Self-Correct Reasoning Yet

Jie Huang, Xinyun Chen, Swaroop Mishra +4 more

cs.CL · · cited 1,259× · arXiv ↗ · more like this ·

Large Language Models (LLMs) have emerged as a groundbreaking technology with their unparalleled text generation capabilities across various applications. Nevertheless, concerns persist regarding the accuracy and appropriateness of their generated content. A contemporary methodology, self-correction, has been proposed as a remedy to these issues. Building upon this premise, this paper critically examines the role and efficacy of self-correction within LLMs, shedding light on its true potential and limitations. Central to our investigation is the notion of intrinsic self-correction, whereby an LLM attempts to correct its initial responses based solely on its inherent capabilities, without the crutch of external feedback. In the context of reasoning, our research indicates that LLMs struggle to self-correct their responses without external feedback, and at times, their performance even…

Manopt, a Matlab toolbox for optimization on manifolds

Nicolas Boumal, Bamdev Mishra, P. -A. Absil +1 more

cs.MS · · cited 1,213× · arXiv ↗ · more like this ·

Optimization on manifolds is a rapidly developing branch of nonlinear optimization. Its focus is on problems where the smooth geometry of the search space can be leveraged to design efficient numerical algorithms. In particular, optimization on manifolds is well-suited to deal with rank and orthogonality constraints. Such structured constraints appear pervasively in machine learning applications, including low-rank matrix completion, sensor network localization, camera network registration, independent component analysis, metric learning, dimensionality reduction and so on. The Manopt toolbox, available at www.manopt.org, is a user-friendly, documented piece of software dedicated to simplify experimenting with state of the art Riemannian optimization algorithms. We aim particularly at reaching practitioners outside our field.

Instruction-Following Evaluation for Large Language Models

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra +5 more

cs.CL · · cited 1,186× · arXiv ↗ · more like this ·

One core capability of Large Language Models (LLMs) is to follow natural language instructions. However, the evaluation of such abilities is not standardized: Human evaluations are expensive, slow, and not objectively reproducible, while LLM-based auto-evaluation is potentially biased or limited by the ability of the evaluator LLM. To overcome these issues, we introduce Instruction-Following Eval (IFEval) for large language models. IFEval is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times". We identified 25 types of those verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions. We show evaluation results of two widely available LLMs on the market. Our code and data can be…

Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi +37 more

cs.CL · · cited 1,144× · arXiv ↗ · more like this ·

How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones. Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing…

Teachers’ Technological Pedagogical Content Knowledge and Learning Activity Types

Judi Harris, Punya Mishra, Matthew J. Koehler

Information Systems · Journal of Research on Technology in Education · · cited 1,066× · DOI ↗ · more like this ·

In this paper we critically analyze extant approaches to technology integration in teaching, arguing that many current methods are technocentric, often omitting sufficient consideration of the dynamic and complex relationships among content, technology, pedagogy, and context. We recommend using the technology, pedagogy, and content knowledge (TPACK) framework as a way to think about effective technology integration, recognizing technology, pedagogy, content and context as interdependent aspects of teachers’ knowledge necessary to teach content-based curricula effectively with educational technologies. We offer TPACK-based “activity types,” rooted in previous research about content-specific activity structures, as an alternative to existing professional development approaches and explain how this new way of thinking may authentically and successfully assist teachers’ and teacher…

PaLI: A Jointly-Scaled Multilingual Language-Image Model

Xi Chen, Xiao Wang, Soravit Changpinyo +26 more

cs.CV · · cited 1,044× · arXiv ↗ · more like this ·

Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI (Pathways Language and Image model), a model that extends this approach to the joint modeling of language and vision. PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train PaLI, we make use of large pre-trained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify the benefits from even larger-capacity vision models. To…

Cross-Task Generalization via Natural Language Crowdsourcing Instructions

Swaroop Mishra, Daniel Khashabi, Chitta Baral +1 more

cs.CL · · cited 925× · arXiv ↗ · more like this ·

Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models…

Clustering data streams: theory and practice

Suvajyoti Guha, Adam Meyerson, Nita Mishra +2 more

Signal Processing · IEEE Transactions on Knowledge and Data Engineering · · cited 900× · DOI ↗ · more like this ·

The data stream model has recently attracted attention for its applicability to numerous types of data, including telephone records, Web documents, and clickstreams. For analysis of such data, the ability to process the data in a single pass, or a small number of passes, while using little memory, is crucial. We describe such a streaming algorithm that effectively clusters large data streams. We also provide empirical evidence of the algorithm's performance on synthetic and real data streams.

An empirical analysis of the IEEE 802.11 MAC layer handoff process

Arunesh Mishra, Minho Shin, William A. Arbaugh

Computer Networks and Communications · ACM SIGCOMM Computer Communication Review · · cited 854× · DOI ↗ · more like this ·

IEEE 802.11 based wireless networks have seen rapid growth and deployment in the recent years. Critical to the 802.11 MAC operation, is the handoff function which occurs when a mobile node moves its association from one access point to another. In this paper, we present an empirical study of this handoff process at the link layer, with a detailed breakup of the latency into various components. In particular, we show that a MAC layer function - probe is the primary contributor to the overall handoff latency. In our study, we observe that the latency is significant enough to affect the quality of service for many applications (or network connections). Further we find variations in the latency from one hand-off to another as well as with APs and STAs used from different vendors. Finally, we discuss optimizations on the probe phase which can potentially reduce the probe latency by as much…

The Effect of Internet Security Breach Announcements on Market Value: Capital Market Reactions for Breached Firms and Internet Security Developers

Huseyin Cavusoglu, Birendra K. Mishra, Srinivasan Raghunathan

Information Systems · International Journal of Electronic Commerce · · cited 767× · DOI ↗ · more like this ·

Assessing the value of information technology (IT) security is challenging because of the difficulty of measuring the cost of security breaches. An event-study analysis, using market valuations, was used to assess the impact of security breaches on the market value of breached firms. The information-transfer effect of security breaches (i.e., their effect on the market value of firms that develop security technology) was also studied. The results show that announcing an Internet security breach is negatively associated with the market value of the announcing firm. The breached firms in the sample lost, on average, 2.1 percent of their market value within two days of the announcement--an average loss in market capitalization of $1.65 billion per breach. Firm type, firm size, and the year the breach occurred help explain the cross-sectional variations in abnormal returns produced by…