arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Adam Fourney

68 papers · 2007–2026

2007: 12008: 02009: 22010: 22011: 32012: 42013: 32014: 22015: 32016: 12017: 32018: 82019: 62020: 62021: 52022: 32023: 32024: 52025: 62026: 2
2007peak 8 in 20182026

Frequent co-authors

Papers

showing 1–25 of 68 RSS

Guidelines for Human-AI Interaction

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu +9 more

Human-Computer Interaction · · cited 1,900× · DOI ↗ · more like this ·

Advances in artificial intelligence (AI) frame opportunities and challenges for user interface design. Principles for human-AI interaction have been discussed in the human-computer interaction community for over two decades, but more study and innovation are needed in light of advances in AI and the growing uses of AI technologies in human-facing applications. We propose 18 generally applicable design guidelines for human-AI interaction. These guidelines are validated through multiple rounds of evaluation including a user study with 49 design practitioners who tested the guidelines against 20 popular AI-infused products. The results verify the relevance of the guidelines over a spectrum of interaction scenarios and reveal gaps in our knowledge, highlighting opportunities for further research. Based on the evaluations, we believe the set of design guidelines can serve as a resource to…

Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming

Hussein Mozannar, Gagan Bansal, Adam Fourney +1 more

cs.SE · · cited 238× · arXiv ↗ · more like this ·

Code-recommendation systems, such as Copilot and CodeWhisperer, have the potential to improve programmer productivity by suggesting and auto-completing code. However, to fully realize their potential, we must understand how programmers interact with these systems and identify ways to improve that interaction. To seek insights about human-AI collaboration with code recommendations systems, we studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our study of 21 programmers, who completed coding tasks and retrospectively labeled their sessions with CUPS, showed that CUPS can help us understand how programmers interact with code-recommendation systems, revealing inefficiencies and time costs. Our insights reveal how programmers interact with Copilot and…

Interactive Debugging and Steering of Multi-Agent AI Systems

Will Epperson, Gagan Bansal, Victor Dibia +4 more

cs.MA · · cited 106× · arXiv ↗ · more like this ·

Fully autonomous teams of LLM-powered AI agents are emerging that collaborate to perform complex tasks for users. What challenges do developers face when trying to build and debug these AI agent teams? In formative interviews with five AI agent developers, we identify core challenges: difficulty reviewing long agent conversations to localize errors, lack of support in current tools for interactive debugging, and the need for tool support to iterate on agent configuration. Based on these needs, we developed an interactive multi-agent debugging tool, AGDebugger, with a UI for browsing and sending messages, the ability to edit and reset prior agent messages, and an overview visualization for navigating complex message histories. In a two-part user study with 14 participants, we identify common user strategies for steering agents and highlight the importance of interactive message resets…

Exploring the Role of Conversational Cues in Guided Task Support with Virtual Assistants

Alexandra Vtyurina, Adam Fourney

Artificial Intelligence · · cited 85× · DOI ↗ · more like this ·

Voice-based conversational assistants are growing in popularity on ubiquitous mobile and stationary devices. Cortana, as well as Google Home, Amazon Echo, and others, can provide support for various tasks from managing reminders to booking a hotel. However, with few exceptions, user input is limited to explicit queries or commands. In this work, we explore the role of implicit conversational cues in guided task completion scenarios. In a Wizard of Oz study, we found that, for the task of cooking a recipe, nearly one-quarter of all user-assistant exchanges were initiated from implicit conversational cues rather than from plain questions. Given that these implicit cues occur in such high frequency, we conclude by presenting a set of design implications for the design of guided task experiences in contemporary conversational assistants.

Geographic and Temporal Trends in Fake News Consumption During the 2016 US Presidential Election

Adam Fourney, Miklós Z. Rácz, Gireeja Ranade +2 more

Sociology and Political Science · · cited 67× · DOI ↗ · more like this ·

We present an analysis of traffic to websites known for publishing fake news in the months preceding the 2016 US presidential election. The study is based on the combined instrumentation data from two popular desktop web browsers: Internet Explorer 11 and Edge. We find that social media was the primary outlet for the circulation of fake news stories and that aggregate voting patterns were strongly correlated with the average daily fraction of users visiting websites serving fake news. This correlation was observed both at the state level and at the county level, and remained stable throughout the main election season. We propose a simple model based on homophily in social networks to explain the linear association. Finally, we highlight examples of different types of fake news stories: while certain stories continue to circulate in the population, others are short-lived and die out in a…

Planning for Natural Language Failures with the AI Playbook

Matthew Hong, Adam Fourney, Derek DeBellis +1 more

Safety Research · · cited 55× · DOI ↗ · more like this ·

Prototyping AI user experiences is challenging due in part to probabilistic AI models making it difficult to anticipate, test, and mitigate AI failures before deployment. In this work, we set out to support practitioners with early AI prototyping, with a focus on natural language (NL)-based technologies. Our interviews with 12 NL practitioners from a large technology company revealed that, in addition to challenges prototyping AI, prototyping was often not happening at all or focused only on idealized scenarios due to a lack of tools and tight timelines. These findings informed our design of the AI Playbook, an interactive and low-cost tool we developed to encourage proactive and systematic consideration of AI errors before deployment. Our evaluation of the AI Playbook demonstrates its potential to 1) encourage product teams to prioritize both ideal and failure scenarios, 2) standardize…

Characterizing the Internet Research Agency’s Social Media Operations During the 2016 U.S. Presidential Election using Linguistic Analyses

Ryan L. Boyd, Alexander Spangher, Adam Fourney +4 more

Artificial Intelligence · PsyArXiv (OSF Preprints) · · cited 54× · DOI ↗ · more like this ·

Converging investigations on the part of multiple agencies/agents have provided overwhelming evidence for Russian interference in the 2016 U.S. presidential election. As a part (and consequence) of recent reports, multiple datasets that capture actions taken by actors of the Internet Research Agency (IRA), have been released to the public. In the cur-rent paper, we present and abridged report of several preliminary forensic analyses of Facebook ad data and Twitter troll accounts that were run by the IRA during the election cycle. Through the use of language analysis, we characterize the evolution of IRA content over the course of the election cycle, providing a basis for understanding how left- and right-leaning ideologies were differentially targeted to spread enmity among the American electorate. Additionally, through an analysis of syntactic constructions, we find that the content…

When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming

Hussein Mozannar, Gagan Bansal, Adam Fourney +1 more

cs.HC · · cited 49× · arXiv ↗ · more like this ·

AI powered code-recommendation systems, such as Copilot and CodeWhisperer, provide code suggestions inside a programmer's environment (e.g., an IDE) with the aim of improving productivity. We pursue mechanisms for leveraging signals about programmers' acceptance and rejection of code suggestions to guide recommendations. We harness data drawn from interactions with GitHub Copilot, a system used by millions of programmers, to develop interventions that can save time for programmers. We introduce a utility-theoretic framework to drive decisions about suggestions to display versus withhold. The approach, conditional suggestion display from human feedback (CDHF), relies on a cascade of models that provide the likelihood that recommended code will be accepted. These likelihoods are used to selectively hide suggestions, reducing both latency and programmer verification time. Using data from…

Understanding the Needs of Searchers with Dyslexia

Meredith Ringel Morris, Adam Fourney, Abdullah Ali +1 more

Artificial Intelligence · · cited 46× · DOI ↗ · more like this ·

As many as 20% of English speakers have dyslexia, a language disability that impacts reading and spelling. Web search is an important modern literacy skill, yet the accessibility of this language-centric endeavor to people with dyslexia is largely unexplored. We interviewed ten adults with dyslexia and conducted an online survey with 81 dyslexic and 80 non-dyslexic adults, in which participants described challenges they face in various stages of web search (query formulation, search result triage, and information extraction). We also report the findings of an online study in which 174 adults with dyslexia and 172 without dyslexia rated the readability and relevance of sets of search query results. Our findings demonstrate differences in behaviors and preferences between dyslexic and non-dyslexic searchers, and indicate that factoring readability into search engine rankings and/or…

Conversations with Documents. An Exploration of Document-Centered Assistance

Maartje ter Hoeve, Robert Sim, Elnaz Nouri +3 more

cs.CL · · cited 45× · arXiv ↗ · more like this ·

The role of conversational assistants has become more prevalent in helping people increase their productivity. Document-centered assistance, for example to help an individual quickly review a document, has seen less significant progress, even though it has the potential to tremendously increase a user's productivity. This type of document-centered assistance is the focus of this paper. Our contributions are three-fold: (1) We first present a survey to understand the space of document-centered assistance and the capabilities people expect in this scenario. (2) We investigate the types of queries that users will pose while seeking assistance with documents, and show that document-centered questions form the majority of these queries. (3) We present a set of initial machine learned models that show that (a) we can accurately detect document-centered questions, and (b) we can build…

Exploring Time-Dependent Concerns about Pregnancy and Childbirth from Search Logs

Adam Fourney, Ryen W. White, Eric Horvitz

Sociology and Political Science · · cited 45× · DOI ↗ · more like this ·

We study time-dependent patterns of information seeking about pregnancy, birth, and the first several weeks of caring for newborns via analyses of queries drawn from anonymized search engine logs. We show how we can detect and align web search behavior for a population of searchers with the natural clock of gestational physiology via proxies for ground truth based on searchers' self-report queries (e.g., [I am 30 weeks pregnant and my baby is moving a lot]). Then, we present a methodology for performing additional alignments, that are valuable for learning about the concerns, curiosities, and needs that arise over time with pregnancy and early parenting. Our findings have implications for learning about the temporal dynamics of pregnancy-related interests and concerns, and also for the design of systems that tailor their responses to point estimates of each searcher's current stage in…

AutoGen Studio: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems

Victor Dibia, Jingya Chen, Gagan Bansal +5 more

cs.SE · · cited 43× · arXiv ↗ · more like this ·

Multi-agent systems, where multiple agents (generative AI models + tools) collaborate, are emerging as an effective pattern for solving long-running, complex tasks in numerous domains. However, specifying their parameters (such as models, tools, and orchestration mechanisms etc,.) and debugging them remains challenging for most developers. To address this challenge, we present AUTOGEN STUDIO, a no-code developer tool for rapidly prototyping, debugging, and evaluating multi-agent workflows built upon the AUTOGEN framework. AUTOGEN STUDIO offers a web interface and a Python API for representing LLM-enabled agents using a declarative (JSON-based) specification. It provides an intuitive drag-and-drop UI for agent workflow specification, interactive evaluation and debugging of workflows, and a gallery of reusable agent components. We highlight four design principles for no-code multi-agent…

VERSE

Alexandra Vtyurina, Adam Fourney, Meredith Ringel Morris +2 more

Human Factors and Ergonomics · · cited 42× · DOI ↗ · more like this ·

People with visual impairments often rely on screen readers when interacting with computer systems. Increasingly, these individuals also make extensive use of voice-based virtual assistants (VAs). We conducted a survey of 53 people who are legally blind to identify the strengths and weaknesses of both technologies, and the unmet opportunities at their intersection. We learned that virtual assistants are convenient and accessible, but lack the ability to deeply engage with content (e.g., read beyond the first few sentences of an article), and the ability to get a quick overview of the landscape (e.g., list alternative search results and suggestions). In contrast, screen readers allow for deep engagement with content (when content is accessible), and provide fine-grained navigation and control, but at the cost of reduced walk-up-and-use convenience. Based on these findings, we implemented…

Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions

Helena Vasconcelos, Gagan Bansal, Adam Fourney +2 more

cs.HC · · cited 40× · arXiv ↗ · more like this ·

Large-scale generative models enabled the development of AI-powered code completion tools to assist programmers in writing code. However, much like other AI-powered tools, AI-powered code completions are not always accurate, potentially introducing bugs or even security vulnerabilities into code if not properly detected and corrected by a human programmer. One technique that has been proposed and implemented to help programmers identify potential errors is to highlight uncertain tokens. However, there have been no empirical studies exploring the effectiveness of this technique -- nor investigating the different and not-yet-agreed-upon notions of uncertainty in the context of generative models. We explore the question of whether conveying information about uncertainty enables programmers to more quickly and accurately produce code when collaborating with an AI-powered code completion…

Social Media through Voice

Lotus Zhang, Lucy Jiang, Nicole Washington +5 more

Artificial Intelligence · Proceedings of the ACM on Human-Computer Interaction · · cited 39× · DOI ↗ · more like this ·

With advances in expressive speech synthesis and conversational understanding, an ever-increasing amount of digital content---including social and personal content---can be consumed through voice. Voice has long been known to convey personal characteristics and emotional states, both of which are prominent aspects of social media. Yet, no study has investigated voice design requirements for social media platforms. We interviewed 15 active social media users about their preferences on using synthesized voices to represent their profiles. Our findings show that participants want to have control over how a voice delivers their content, such as the personality and emotion with which the voice speaks, because these prosodic variations can impact users' online personas and interfere with impression management. We report motivations behind customizing or not customizing voice characteristics…

Bridging Screen Readers and Voice Assistants for Enhanced Eyes-Free Web Search

Alexandra Vtyurina, Adam Fourney, Meredith Ringel Morris +2 more

Artificial Intelligence · · cited 39× · DOI ↗ · more like this ·

People with visual impairments often rely on screen readers when interacting with computer systems. Increasingly, these individuals also make extensive use of voice-based virtual assistants (VAs). We conducted a survey of 53 people who are legally blind to identify the strengths and weaknesses of both technologies, as well as the unmet opportunities at their intersection. We learned that virtual assistants are convenient and accessible, but lack the ability to deeply engage with content (e.g., read beyond the first few sentences of Wikipedia), and the ability to get a quick overview of the landscape (list alternative search results & suggestions). In contrast, screen readers allow for deep engagement with content (when content is accessible), and provide fine-grained navigation & control, but at the cost of increased complexity, and reduced walk-up-and-use convenience. In this…

Query-feature graphs

Adam Fourney, Richard Mann, Michael Terry

Information Systems · · cited 37× · DOI ↗ · more like this ·

This paper introduces query-feature graphs, or QF-graphs. QF-graphs encode associations between high-level descriptions of user goals (articulated as natural language search queries) and the specific features of an interactive system relevant to achieving those goals. For example, a QF-graph for the GIMP graphics manipulation software links the query "GIMP black and white" to the commands "desaturate" and "grayscale." We demonstrate how QF-graphs can be constructed using search query logs, search engine results, web page content, and localization data from interactive systems. An analysis of QF-graphs shows that the associations produced by our approach exhibit levels of accuracy that make them eminently usable in a range of real-world applications. Finally, we present three hypothetical user interface mechanisms that illustrate the potential of QF-graphs: search-driven interaction…

DIY: Assessing the Correctness of Natural Language to SQL Systems

Arpit Narechania, Adam Fourney, Bongshin Lee +1 more

Information Systems · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · · cited 34× · DOI ↗ · more like this ·

Designing natural language interfaces for querying databases remains an important goal pursued by researchers in natural language processing, databases, and HCI. These systems receive natural language as input, translate it into a formal database query, and execute the query to compute a result. Because the responses from these systems are not always correct, it is important to provide people with mechanisms to assess the correctness of the generated query and computed result. However, this assessment can be challenging for people who lack expertise in query languages. We present Debug-It-Yourself (DIY), an interactive technique that enables users to assess the responses from a state-of-the-art natural language to SQL (NL2SQL) system for correctness and, if possible, fix errors. DIY provides users with a sandbox where they can interact with (1) the mappings between the question and the…

Characterizing the usability of interactive applications through query log analysis

Adam Fourney, Richard Mann, Michael Terry

Information Systems · · cited 33× · DOI ↗ · more like this ·

People routinely rely on Internet search engines to support their use of interactive systems: they issue queries to learn how to accomplish tasks, troubleshoot problems, and otherwise educate themselves on products. Given this common behavior, we argue that search query logs can usefully augment traditional usability methods by revealing the primary tasks and needs of a product's user population. We term this use of search query logs CUTS - characterizing usability through search. In this paper, we introduce CUTS and describe an automated process for harvesting, ordering, labeling, filtering, and grouping search queries related to a given product. Importantly, this data set can be assembled in minutes, is timely, has a high degree of ecological validity, and is arguably less prone to self-selection bias than data gathered via traditional usability methods. We demonstrate the utility of…

Constructing Face Image Logs that are Both Complete and Concise

Adam Fourney, Robert Laganière

Computer Vision and Pattern Recognition · · cited 31× · DOI ↗ · more like this ·

This paper describes a construct that we call a face image log. Face image logs are collections of time stamped images representing faces detected in surveillance videos. The techniques demonstrated in this paper strive to construct face image logs that are complete and concise in the sense that the logs contain only the best images available for each individual observed. We begin by describing how to assess and compare the quality of face images. We then illustrate a robust method for selecting high quality images. This selection process takes into consideration the limitations inherent in existing face detection and person tracking techniques. Experimental results demonstrate that face logs constructed in this manner generally contain fewer than 5% of all detected faces, yet these faces are of high quality, and they represent all individuals detected in the video sequence.

Managing Uncertainty in Time Expressions for Virtual Assistants

Xin Rong, Adam Fourney, Robin Brewer +2 more

Artificial Intelligence · · cited 29× · DOI ↗ · more like this ·

"Remind me to get milk later this afternoon." In communications and planning, people often express uncertainty about time using imprecise temporal expressions (ITEs). Unfortunately, modern virtual assistants often lack system support to capture the intents behind these expressions. This can result in unnatural interactions and undesirable interruptions (e.g., having a work reminder delivered at 12pm when out at lunch, because the user said "this afternoon"). In this paper we explore existing practices, expectations, and preferences surrounding the use of ITEs. Our mixed methods approach employs surveys, interviews, and an analysis of a large corpus of written communications. We find that people frequently use a diverse set of ITEs in both communication and planning. These uses reflect a variety of motivations, such as conveying uncertainty or task priority. In addition, we find that…

The Impact of Web Browser Reader Views on Reading Speed and User Experience

Qisheng Li, Meredith Ringel Morris, Adam Fourney +2 more

Education · · cited 28× · DOI ↗ · more like this ·

As reading increasingly shifts from paper to online media, many web browsers now provide a "Reader View,'' which modifies web page layout and design for better readability. However, research has yet to establish whether Reader Views are effective in improving readability and how they might change the user experience. We characterize how Mozilla Firefox's Reader View significantly reduces the visual complexity of websites by excluding menus, images, and content. We then conducted an online study with 391 participants (including 42 who self-reported having been diagnosed with dyslexia), showing that compared to standard websites the Reader View increased reading speed by 5% for readers on average, and significantly improved perceived readability and visual appeal. We suggest guidelines for the design of websites and browsers that better support people with varying reading skills.

Analysis of Strategy and Spread of Russia-sponsored Content in the US in 2017

Alexander Spangher, Gireeja Ranade, Besmira Nushi +2 more

cs.SI · · cited 24× · arXiv ↗ · more like this ·

The Russia-based Internet Research Agency (IRA) carried out a broad information campaign in the U.S. before and after the 2016 presidential election. The organization created an expansive set of internet properties: web domains, Facebook pages, and Twitter bots, which received traffic via purchased Facebook ads, tweets, and search engines indexing their domains. We investigate the scope of IRA activities in 2017, joining data from Facebook and Twitter with logs from the Internet Explorer 11 and Edge browsers and the Bing.com search engine. The studies demonstrate both the ease with which malicious actors can harness social media and search engines for propaganda campaigns, and the ability to track and understand such activities by fusing content and activity resources from multiple internet services. We show how cross-platform analyses can provide an unprecedented lens on attempts to…

<i>These Aren't the Commands You're Looking For</i>

Benjamin Lafreniere, Parmit K. Chilana, Adam Fourney +1 more

Information Systems · · cited 23× · DOI ↗ · more like this ·

The names, icons, and tooltips of commands in feature-rich software are an important source of guidance when locating and selecting amongst commands. Unfortunately, these cues can mislead users into believing that a command is appropriate for a given task, when another command would be more appropriate, resulting in wasted time and frustration. In this paper, we present command disambiguation techniques that inform the user of alternative commands before, during, and after an incorrect command has been executed. To inform the design of these techniques, we define categories of false-feedforward errors caused by misleading interface cues, and identify causes for each. Our techniques are the first designed explicitly to solve this problem in feature-rich software. A user study showed enthusiasm for the techniques, and revealed their potential to play a key role in learning of feature-rich…

Assessing the Readability of Web Search Results for Searchers with Dyslexia

Adam Fourney, Meredith Ringel Morris, Abdullah Ali +1 more

Artificial Intelligence · · cited 22× · DOI ↗ · more like this ·

Standards organizations, (e.g., the World Wide Web Consortium), are placing increased importance on the cognitive accessibility of online systems, including web search. Previous work has shown an association between query-document relevance judgments, and query-independent assessments of document readability. In this paper we study the lexical and aesthetic features of web documents that may underlie this relationship. Leveraging a data set consisting of relevance and readability judgments for 200 web pages as assessed by 174 adults with dyslexia and 172 adults without dyslexia, we answer the following research questions: (1) Which web page features are most associated with readability? (2) To what extent are these features also associated with relevance? And, (3) are any features associated with the differences in readability/relevance judgments observed between dyslexic and…