arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Tim Draws

22 papers · 2018–2023

2018: 12019: 02020: 52021: 82022: 22023: 6
2018peak 8 in 20212023

Frequent co-authors

Papers

showing 1–22 of 22 RSS

The JASP guidelines for conducting and reporting a Bayesian analysis

Johnny van Doorn, Don van den Bergh, Udo Böhm +9 more

Statistics and Probability · Psychonomic Bulletin & Review · · cited 1,080× · DOI ↗ · more like this ·

Despite the increasing popularity of Bayesian inference in empirical research, few practical guidelines provide detailed recommendations for how to apply Bayesian procedures and interpret the results. Here we offer specific guidelines for four different stages of Bayesian statistical reasoning in a research setting: planning the analysis, executing the analysis, interpreting the results, and reporting the results. The guidelines for each stage are illustrated with a running example. Although the guidelines are geared towards analyses performed with the open-source statistical software JASP, most guidelines extend to Bayesian inference in general.

A Checklist to Combat Cognitive Biases in Crowdsourcing

Tim Draws, Alisa Rieger, Oana Inel +2 more

Computer Science Applications · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · · cited 83× · DOI ↗ · more like this ·

Recent research has demonstrated that cognitive biases such as the confirmation bias or the anchoring effect can negatively affect the quality of crowdsourced data. In practice, however, such biases go unnoticed unless specifically assessed or controlled for. Task requesters need to ensure that task workflow and design choices do not trigger workers’ cognitive biases. Moreover, to facilitate the reuse of crowdsourced data collections, practitioners can benefit from understanding whether and which cognitive biases may be associated with the data. To this end, we propose a 12-item checklist adapted from business psychology to combat cognitive biases in crowdsourcing. We demonstrate the practical application of this checklist in a case study on viewpoint annotations for search results. Through a retrospective analysis of relevant crowdsourcing research that has been published at HCOMP in…

Disentangling Fairness Perceptions in Algorithmic Decision-Making: the Effects of Explanations, Human Oversight, and Contestability

Mireia Yurrita, Tim Draws, Agathe Balayn +3 more

Safety Research · Research Publications (Maastricht University) · · cited 67× · DOI ↗ · more like this ·

Recent research claims that information cues and system attributes of algorithmic decision-making processes affect decision subjects’ fairness perceptions. However, little is still known about how these factors interact. This paper presents a user study (N = 267) investigating the individual and combined effects of explanations, human oversight, and contestability on informational and procedural fairness perceptions for high- and low-stakes decisions in a loan approval scenario. We find that explanations and contestability contribute to informational and procedural fairness perceptions, respectively, but we find no evidence for an effect of human oversight. Our results further show that both informational and procedural fairness perceptions contribute positively to overall fairness perceptions but we do not find an interaction effect between them. A qualitative analysis exposes tensions…

This Item Might Reinforce Your Opinion: Obfuscation and Labeling of Search Results to Mitigate Confirmation Bias

Alisa Rieger, Tim Draws, Mariët Theune +1 more

Sociology and Political Science · Research Publications (Maastricht University) · · cited 46× · DOI ↗ · more like this ·

During online information search, users tend to select search results that confirm previous beliefs and ignore competing possibilities. This systematic pattern in human behavior is known as confirmation bias. In this paper, we study the effect of obfuscation (i.e., hiding the result unless the user clicks on it) with warning labels and the effect of task on interaction with attitude-confirming search results. We conducted a preregistered, between-subjects crowdsourced user study (N=328) comparing six groups: three levels of obfuscation (targeted, random, none) and two levels of task (joint, two separate) for four debated topics. We found that both types of obfuscation influence user interactions, and in particular that targeted obfuscation helps decrease interaction with attitude-confirming search results. Future work is needed to understand how much of the observed effect is due to the…

Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics

Tim Draws, Nava Tintarev, Ujwal Gadiraju +2 more

cs.IR · · cited 36× · arXiv ↗ · more like this ·

The way pages are ranked in search results influences whether the users of search engines are exposed to more homogeneous, or rather to more diverse viewpoints. However, this viewpoint diversity is not trivial to assess. In this paper we use existing and novel ranking fairness metrics to evaluate viewpoint diversity in search result rankings. We conduct a controlled simulation study that shows how ranking fairness metrics can be used for viewpoint diversity, how their outcome should be interpreted, and which metric is most suitable depending on the situation. This paper lays out important ground work for future research to measure and assess viewpoint diversity in real search result rankings.

This Is Not What We Ordered: Exploring Why Biased Search Result Rankings Affect User Attitudes on Debated Topics

Tim Draws, Nava Tintarev, Ujwal Gadiraju +2 more

Sociology and Political Science · Research Publications (Maastricht University) · · cited 35× · DOI ↗ · more like this ·

In web search on debated topics, algorithmic and cognitive biases strongly influence how users consume and process information. Recent research has shown that this can lead to a search engine manipulation effect (SEME): when search result rankings are biased towards a particular viewpoint, users tend to adopt this favored viewpoint. To better understand the mechanisms underlying SEME, we present a pre-registered, 5 x 3 factorial user study investigating whether order effects (i.e., users adopting the viewpoint pertaining to higher-ranked documents) can cause SEME. For five different debated topics, we evaluated attitude change after exposing participants with mild pre-existing attitudes to search results that were overall viewpoint-balanced but reflected one of three levels of algorithmic ranking bias. We found that attitude change did not differ across levels of ranking bias and did…

Comprehensive Viewpoint Representations for a Deeper Understanding of User Interactions With Debated Topics

Tim Draws, Oana Inel, Nava Tintarev +2 more

Computer Vision and Pattern Recognition · Research Publications (Maastricht University) · · cited 29× · DOI ↗ · more like this ·

Research in the area of human information interaction (HII) typically represents viewpoints on debated topics in a binary fashion, as either against or in favor of a given topic (e.g., the feminist movement). This simple taxonomy, however, greatly reduces the latent richness of viewpoints and thereby limits the potential of research and practical applications in this field. Work in the communication sciences has already demonstrated that viewpoints can be represented in much more comprehensive ways, which could enable a deeper understanding of users’ interactions with debated topics online. For instance, a viewpoint’s stance usually has a degree of strength (e.g., mild or strong), and, even if two viewpoints support or oppose something to the same degree, they may use different logics of evaluation (i.e., underlying reasons). In this paper, we draw from communication science practice to…

The Effects of Crowd Worker Biases in Fact-Checking Tasks

Tim Draws, David La Barbera, Michael Soprano +4 more

Sociology and Political Science · 2022 ACM Conference on Fairness, Accountability, and Transparency · · cited 21× · DOI ↗ · more like this ·

Due to the increasing amount of information shared online every day, the need for sound and reliable ways of distinguishing between trustworthy and non-trustworthy information is as present as ever. One technique for performing fact-checking at scale is to employ human intelligence in the form of crowd workers. Although earlier work has suggested that crowd workers can reliably identify misinformation, cognitive biases of crowd workers may reduce the quality of truthfulness judgments in this context. We performed a systematic exploratory analysis of publicly available crowdsourced data to identify a set of potential systematic biases that may occur when crowd workers perform fact-checking tasks. Following this exploratory study, we collected a novel data set of crowdsourced truthfulness judgments to validate our hypotheses. Our findings suggest that workers generally overestimate the…

Evaluating explainable social choice-based aggregation strategies for group recommendation

Francesco Barile, Tim Draws, Oana Inel +5 more

Sociology and Political Science · User Modeling and User-Adapted Interaction · · cited 19× · DOI ↗ · more like this ·

Abstract Social choice aggregation strategies have been proposed as an explainable way to generate recommendations to groups of users. However, it is not trivial to determine the best strategy to apply for a specific group. Previous work highlighted that the performance of a group recommender system is affected by the internal diversity of the group members’ preferences. However, few of them have empirically evaluated how the specific distribution of preferences in a group determines which strategy is the most effective. Furthermore, only a few studies evaluated the impact of providing explanations for the recommendations generated with social choice aggregation strategies, by evaluating explanations and aggregation strategies in a coupled way. To fill these gaps, we present two user studies ( N =399 and N =288) examining the effectiveness of social choice aggregation strategies in…

Nudges to Mitigate Confirmation Bias during Web Search on Debated Topics: Support vs. Manipulation

Alisa Rieger, Tim Draws, Mariët Theune +1 more

Sociology and Political Science · ACM Transactions on the Web · · cited 16× · DOI ↗ · more like this ·

When people use web search engines to find information on debated topics, the search results they encounter can influence opinion formation and practical decision-making with potentially far-reaching consequences for the individual and society. However, current web search engines lack support for information-seeking strategies that enable responsible opinion formation, e.g., by mitigating confirmation bias and motivating engagement with diverse viewpoints. We conducted two preregistered user studies to test the benefits and risks of an intervention aimed at confirmation bias mitigation. In the first study, we tested the effect of warning labels, warning of the risk of confirmation bias, combined with obfuscations, hiding selected search results per default. We observed that obfuscations with warning labels effectively reduce engagement with search results. These initial findings did not…

Investigating the Influence of Featured Snippets on User Attitudes

Markus Bink, Sebastian Schwarz, Tim Draws +1 more

Human-Computer Interaction · Research Repository (Delft University of Technology) · · cited 12× · DOI ↗ · more like this ·

Featured snippets that attempt to satisfy users’ information needs directly on top of the first search engine results page (SERP) have been shown to strongly impact users’ post-search attitudes and beliefs. In the context of debated but scientifically answerable topics, recent research has demonstrated that users tend to trust featured snippets to such an extent that they may reverse their original beliefs based on what such a snippet suggests; even when erroneous information is featured. This paper examines the effect of featured snippets in more nuanced and complicated search scenarios concerning debated topics that have no ground truth and where diverse arguments in favor and against can legitimately be made. We report on a preregistered, online user study (N = 182) investigating how the stances and logics of evaluation (i.e., underlying reasons behind stances) expressed in featured…

Explainable Cross-Topic Stance Detection for Search Results

Tim Draws, Karthikeyan Natesan Ramamurthy, Ioana Baldini +4 more

Artificial Intelligence · Research Publications (Maastricht University) · · cited 12× · DOI ↗ · more like this ·

One way to help users navigate debated topics online is to apply stance detection in web search. Automatically identifying whether search results are against, neutral, or in favor could facilitate diversification efforts and support interventions that aim to mitigate cognitive biases. To be truly useful in this context, however, stance detection models not only need to make accurate (cross-topic) predictions but also be sufficiently explainable to users when applied to search results – an issue that is currently unclear. This paper presents a study into the feasibility of using current stance detection approaches to assist users in their web search on debated topics. We train and evaluate 10 stance detection models using a stance-annotated data set of 1204 search results. In a preregistered user study (N = 291), we then investigate the quality of stance detection explanations created…

Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection

Oana Inel, Tim Draws, Lora Aroyo

cs.LG · · cited 9× · arXiv ↗ · more like this ·

The rapid entry of machine learning approaches in our daily activities and high-stakes domains demands transparency and scrutiny of their fairness and reliability. To help gauge machine learning models' robustness, research typically focuses on the massive datasets used for their deployment, e.g., creating and maintaining documentation for understanding their origin, process of development, and ethical considerations. However, data collection for AI is still typically a one-off practice, and oftentimes datasets collected for a certain purpose or application are reused for a different problem. Additionally, dataset annotations may not be representative over time, contain ambiguous or erroneous annotations, or be unable to generalize across issues or domains. Recent research has shown these practices might lead to unfair, biased, or inaccurate outcomes. We argue that data collection for…

Exploring User Concerns about Disclosing Location and Emotion Information in Group Recommendations

Shabnam Najafian, Tim Draws, Francesco Barile +3 more

Sociology and Political Science · Research Publications (Maastricht University) · · cited 9× · DOI ↗ · more like this ·

Recent research has shown that explanations serve as an important means to increase transparency in group recommendations while also increasing users' privacy concerns. However, it is currently unclear what personal and contextual factors affect users' privacy concerns about various types of personal information. This paper studies the effect of users' personality traits and preference scenarios ---having a majority or minority preference--- on their privacy concerns regarding location and emotion information. To create natural scenarios of group decision-making where users can control the amount of information disclosed, we develop TouryBot, a chat-bot agent that generates natural language explanations to help group members explain their arguments for suggestions to the group in the tourism domain. We conducted a user study in which we instructed 541 participants to convince the group…

Helping users discover perspectives: Enhancing opinion mining with joint topic models

Tim Draws, Jody Liu, Nava Tintarev

cs.CL · · cited 9× · arXiv ↗ · more like this ·

Support or opposition concerning a debated claim such as abortion should be legal can have different underlying reasons, which we call perspectives. This paper explores how opinion mining can be enhanced with joint topic modeling, to identify distinct perspectives within the topic, providing an informative overview from unstructured text. We evaluate four joint topic models (TAM, JST, VODUM, and LAM) in a user study assessing human understandability of the extracted perspectives. Based on the results, we conclude that joint topic models such as TAM can discover perspectives that align with human judgments. Moreover, our results suggest that users are not influenced by their pre-existing stance on the topic of abortion when interpreting the output of topic models.

Transparency Paths - Documenting the Diversity of User Perceptions

Fausto Giunchiglia, Styliani Kleanthous, Jahna Otterbacher +1 more

Safety Research · Research Repository (Delft University of Technology) · · cited 5× · DOI ↗ · more like this ·

<p>We are living in an era of global digital platforms, eco-systems of algorithmic processes that serve users worldwide. However, the increasing exposure to diversity online - of information and users - has led to important considerations of bias. A given platform, such as the Google search engine, may demonstrate behaviors that deviate from what users expect, or what they consider fair, relative to their own context and experiences. In this exploratory work, we put forward the notion of transparency paths, a process by which we document our position, choices, and perceptions when developing and/or using algorithmic platforms. We conducted a self-reflection exercise with seven researchers, who collected and analyzed two sets of images; one depicting an everyday activity, "washing hands,"and a second depicting the concept of "home."Participants had to document their process and…

Toward Benchmarking Group Explanations: Evaluating the Effect of Aggregation Strategies versus Explanation

Francesco Barile, Shabnam Najafian, Tim Draws +4 more

Information Systems · Zurich Open Repository and Archive (University of Zurich) · · cited 3× · DOI ↗ · more like this ·

In the context of group recommendations, explanations have been claimed to be useful for finding a satisfactory choice for all the group members and helping them agree on a common decision, improving perceived fairness, perceived consensus, and satisfaction. In this work, we present a preregistered evaluation of the impact of using social choice-based explanations for group recommendations (i.e., explanations that intuitively describe the strategy used to generate the recommendation). Our objective is to conceptually replicate a previous study and investigate whether a) the used aggregation strategy or b) the explanation affected the most users’ fairness perception, consensus perception, and satisfaction. Our results show that the participants are able to discriminate between the different strategies, assigning worse evaluations to the Most Pleasure strategy (which chooses the item with…

Understanding How Algorithmic and Cognitive Biases in Web Search Affect User Attitudes on Debated Topics

Tim Draws

Sociology and Political Science · Research Repository (Delft University of Technology) · · cited 1× · DOI ↗ · more like this ·

Web search increasingly provides a platform for users to seek advice on important personal decisions but may be biased in several different ways. One result of such biases is the search engine manipulation effect (SEME): when a list of search results relates to a debated topic (e.g., veganism) and promotes documents pertaining to a particular viewpoint (e.g., by ranking them higher), users tend to adopt this advantaged viewpoint. However, the detection and mitigation of SEME are complicated by the current lack of empirical understanding of its underlying mechanisms. This dissertation aims to investigate which (and to what degree) algorithmic and cognitive biases play a role in SEME concerning debated topics. RQ1. What set of labels can accurately represent viewpoints of textual documents on debated topics? Studying algorithmic and cognitive biases in the context of web search on debated…

Combine Statistical Thinking With Open Scientific Practice: A Protocol of a Bayesian Research Project

Alexandra Sarafoglou, Anna van der Heijden, Tim Draws +3 more

stat.AP · · cited 1× · arXiv ↗ · more like this ·

Current developments in the statistics community suggest that modern statistics education should be structured holistically, that is, by allowing students to work with real data and to answer concrete statistical questions, but also by educating them about alternative frameworks, such as Bayesian inference. In this article, we describe how we incorporated such a holistic structure in a Bayesian research project on ordered binomial probabilities. The project was conducted with a group of three undergraduate psychology students who had basic knowledge of Bayesian statistics and programming, but lacked formal mathematical training. The research project aimed to (1) convey the basic mathematical concepts of Bayesian inference; (2) have students experience the entire empirical cycle including collection, analysis, and interpretation of data and (3) teach students open science practices.

Disparate Impact Diminishes Consumer Trust Even for Advantaged Users

Tim Draws, Zoltán Szlávik, Benjamin Timmermans +3 more

cs.HC · · cited 0× · arXiv ↗ · more like this ·

Systems aiming to aid consumers in their decision-making (e.g., by implementing persuasive techniques) are more likely to be effective when consumers trust them. However, recent research has demonstrated that the machine learning algorithms that often underlie such technology can act unfairly towards specific groups (e.g., by making more favorable predictions for men than for women). An undesired disparate impact resulting from this kind of algorithmic unfairness could diminish consumer trust and thereby undermine the purpose of the system. We studied this effect by conducting a between-subjects user study investigating how (gender-related) disparate impact affected consumer trust in an app designed to improve consumers' financial decision-making. Our results show that disparate impact decreased consumers' trust in the system and made them less likely to use it. Moreover, we find that…

Perspective Discovery in Controversial Debates

Jody Liu, Tim Draws

Artificial Intelligence · OSF Preprints (OSF Preprints) · · cited 0× · OpenAlex ↗ · more like this ·

This repository contains all the material used for the research on Perspective Discovery in Controversial Debates. We provide the code, data and metadata in order to reproduce the research done. We researched the potential of topic models to automatically discover perspectives in text. For this we evaluated five topic models (LDA, TAM, VODUM, JST and LAM) in terms of their clustering ability of perspective labels and human understandability of the model's output. The results show that some of these models may discover perspectives from text.

Investigating the influence of search result rankings on user attitudes

Tim Draws

Information Systems · OSF Preprints (OSF Preprints) · · cited 0× · DOI ↗ · more like this ·

In this research, we aim to investigate the effect of search results rankings on user attitudes towards debated topics. For example, it is commonly debated whether zoos should exist. If a user without a strong pre-existing opinion on zoos conducts a web search in which documents opposing zoos are ranked higher than documents supporting zoos, that user might tend towards adopting an opposing viewpoint as well – and vice versa. Previous work has demonstrated that users’ decision-making can be affected by search result rankings. We will conduct a between-subjects user study to study this effect in more detail and with respect to debated topics. We expose study participants to different search result rankings (i.e., containing varying degrees of viewpoint diversity) related to different debated topics and measure their attitudes towards these topics. In line with related research on this…