arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Raefer Gabriel

12 papers · 2000–2023

2000: 12001: 02002: 02003: 02004: 02005: 02006: 02007: 02008: 02009: 02010: 02011: 02012: 02013: 02014: 02015: 02016: 02017: 02018: 62019: 22020: 02021: 12022: 12023: 1
2000peak 6 in 20182023

Frequent co-authors

Papers

showing 1–12 of 12 RSS

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448 more

cs.CL · · cited 2,670× · arXiv ↗ · more like this ·

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of…

Conversational AI: The Science Behind the Alexa Prize

Ashwin Ram, Rohit Prasad, Chandra Khatri +15 more

cs.AI · · cited 303× · arXiv ↗ · more like this ·

Conversational agents are exploding in popularity. However, much work remains in the area of social conversation as well as free-form conversation over a broad range of domains and topics. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million-dollar university competition where sixteen selected university teams were challenged to build conversational agents, known as socialbots, to converse coherently and engagingly with humans on popular topics such as Sports, Politics, Entertainment, Fashion and Technology for 20 minutes. The Alexa Prize offers the academic community a unique opportunity to perform research with a live system used by millions of users. The competition provided university teams with real user conversational data at scale, along with the user-provided ratings and feedback augmented with annotations by the Alexa team. This…

Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize

Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh +19 more

cs.CL · · cited 87× · arXiv ↗ · more like this ·

Building open domain conversational systems that allow users to have engaging conversations on topics of their choice is a challenging task. Alexa Prize was launched in 2016 to tackle the problem of achieving natural, sustained, coherent and engaging open-domain dialogs. In the second iteration of the competition in 2018, university teams advanced the state of the art by using context in dialog models, leveraging knowledge graphs for language understanding, handling complex utterances, building statistical and hierarchical dialog managers, and leveraging model-driven signals from user responses. The 2018 competition also included the provision of a suite of tools and models to the competitors including the CoBot (conversational bot) toolkit, topic and dialog act detection models, conversation evaluators, and a sensitive content detection model so that the competing teams could focus on…

On Evaluating and Comparing Open Domain Dialog Systems

Anu Venkatesh, Chandra Khatri, Ashwin Ram +10 more

cs.CL · · cited 86× · arXiv ↗ · more like this ·

Conversational agents are exploding in popularity. However, much work remains in the area of non goal-oriented conversations, despite significant growth in research interest over recent years. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million dollar university competition where sixteen selected university teams built conversational agents to deliver the best social conversational experience. Alexa Prize provided the academic community with the unique opportunity to perform research with a live system used by millions of users. The subjectivity associated with evaluating conversations is key element underlying the challenge of building non-goal oriented dialogue systems. In this paper, we propose a comprehensive evaluation strategy with multiple metrics designed to reduce subjectivity by selecting metrics which correlate well with human…

Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators

Sanghyun Yi, Rahul Goel, Chandra Khatri +6 more

cs.CL · · cited 67× · arXiv ↗ · more like this ·

Encoder-decoder based neural architectures serve as the basis of state-of-the-art approaches in end-to-end open domain dialog systems. Since most of such systems are trained with a maximum likelihood~(MLE) objective they suffer from issues such as lack of generalizability and the generic response problem, i.e., a system response that can be an answer to a large number of user utterances, e.g., "Maybe, I don't know." Having explicit feedback on the relevance and interestingness of a system response at each turn can be a useful signal for mitigating such issues and improving system quality by selecting responses from different approaches. Towards this goal, we present a system that evaluates chatbot responses at each dialog turn for coherence and engagement. Our system provides explicit turn-level dialog quality feedback, which we show to be highly correlated with human evaluation. To…

Alexa Prize — State of the Art in Conversational AI

Chandra Khatri, Anu Venkatesh, Behnam Hedayatnia +3 more

Artificial Intelligence · AI Magazine · · cited 52× · DOI ↗ · more like this ·

To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a $2.5 million competition that challenges university teams to build conversational agents, or “socialbots,” that can converse coherently and engagingly with humans on popular topics for 20 minutes. The Alexa Prize offers the academic community a unique opportunity to perform research at scale with real conversational data obtained by interacting with millions of Alexa users, along with user‐provided ratings and feedback, over several months. This opportunity enables teams to effectively iterate, improve, and evaluate their socialbots throughout the competition. Eighteen teams were selected for the inaugural competition last year. To build their socialbots, the students combined state‐of‐the‐art techniques with their own novel strategies in the areas of natural language understanding and…

Detecting Offensive Content in Open-domain Conversations using Two Stage Semi-supervision

Chandra Khatri, Behnam Hedayatnia, Rahul Goel +3 more

cs.CL · · cited 33× · arXiv ↗ · more like this ·

As open-ended human-chatbot interaction becomes commonplace, sensitive content detection gains importance. In this work, we propose a two stage semi-supervised approach to bootstrap large-scale data for automatic sensitive language detection from publicly available web resources. We explore various data selection methods including 1) using a blacklist to rank online discussion forums by the level of their sensitiveness followed by randomly sampling utterances and 2) training a weakly supervised model in conjunction with the blacklist for scoring sentences from online discussion forums to curate a dataset. Our data collection strategy is flexible and allows the models to detect implicit sensitive content for which manual annotations may be difficult. We train models using publicly available annotated datasets as well as using the proposed large-scale semi-supervised datasets. We evaluate…

Alexa Conversations: An Extensible Data-driven Approach for Building Task-oriented Dialogue Systems

Anish Acharya, Suranjit Adhikari, Sanchit Agarwal +28 more

cs.CL · · cited 27× · arXiv ↗ · more like this ·

Traditional goal-oriented dialogue systems rely on various components such as natural language understanding, dialogue state tracking, policy learning and response generation. Training each component requires annotations which are hard to obtain for every new domain, limiting scalability of such systems. Similarly, rule-based dialogue systems require extensive writing and maintenance of rules and do not scale either. End-to-End dialogue systems, on the other hand, do not require module-specific annotations but need a large amount of data for training. To overcome these problems, in this demo, we present Alexa Conversations, a new approach for building goal-oriented dialogue systems that is scalable, extensible as well as data efficient. The components of this system are trained in a data-driven manner, but instead of collecting annotated conversations for training, we generate them…

Contextual Topic Modeling For Dialog Systems

Chandra Khatri, Rahul Goel, Behnam Hedayatnia +4 more

cs.CL · · cited 26× · arXiv ↗ · more like this ·

Accurate prediction of conversation topics can be a valuable signal for creating coherent and engaging dialog systems. In this work, we focus on context-aware topic classification methods for identifying topics in free-form human-chatbot dialogs. We extend previous work on neural topic classification and unsupervised topic keyword detection by incorporating conversational context and dialog act features. On annotated data, we show that incorporating context and dialog acts leads to relative gains in topic classification accuracy by 35% and on unsupervised keyword detection recall by 11% for conversational interactions where topics frequently span multiple utterances. We show that topical metrics such as topical depth is highly correlated with dialog evaluation metrics such as coherence and engagement implying that conversational topic models can predict user satisfaction. Our work for…

Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations

Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen +5 more

cs.CL · · cited 5× · arXiv ↗ · more like this ·

Building socialbots that can have deep, engaging open-domain conversations with humans is one of the grand challenges of artificial intelligence (AI). To this end, bots need to be able to leverage world knowledge spanning several domains effectively when conversing with humans who have their own world knowledge. Existing knowledge-grounded conversation datasets are primarily stylized with explicit roles for conversation partners. These datasets also do not explore depth or breadth of topical coverage with transitions in conversations. We introduce Topical-Chat, a knowledge-grounded human-human conversation dataset where the underlying knowledge spans 8 broad topics and conversation partners don't have explicitly defined roles, to help further research in open-domain conversational AI. We also train several state-of-the-art encoder-decoder conversational models on Topical-Chat and…

Natural Language Generation at Scale: A Case Study for Open Domain Question Answering

Alessandra Cervone, Chandra Khatri, Rahul Goel +4 more

cs.CL · · cited 3× · arXiv ↗ · more like this ·

Current approaches to Natural Language Generation (NLG) for dialog mainly focus on domain-specific, task-oriented applications (e.g. restaurant booking) using limited ontologies (up to 20 slot types), usually without considering the previous conversation context. Furthermore, these approaches require large amounts of data for each domain, and do not benefit from examples that may be available for other domains. This work explores the feasibility of applying statistical NLG to scenarios requiring larger ontologies, such as multi-domain dialog applications or open-domain question answering (QA) based on knowledge graphs. We model NLG through an Encoder-Decoder framework using a large dataset of interactions between real-world users and a conversational agent for open-domain QA. First, we investigate the impact of increasing the number of slot types on the generation quality and experiment…

HealthDialogue: An XML-based Document System for Facilitating Patient-physician Communication

Raefer Gabriel, Drew Volpe, Yana Ageeva +1 more

Artificial Intelligence · PubMed Central · · cited 0× · OpenAlex ↗ · more like this ·

The current state of web-based information repositories in the medical field poses certain unanswered problems and challenges. There are often many different audiences for the same material, each desiring information appropriate in scope and detail. The needs of a layperson are often different from the medical student, or practicing clinician. The development of content for different audiences results in reduplication of efforts. Furthermore, the consumption of information by different audiences is insular, and does not encourage interaction among different audiences with similar interests. In this poster synopsis, we will present the work we have done in developing an XML-based system that addresses these concerns. The system, which is fully implemented and functional, offers the user the ability to see multiple views of a given document; it also allows the user to make and send…