arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Kevin Roth

16 papers · 2010–2021

2010: 12011: 02012: 22013: 02014: 12015: 02016: 12017: 22018: 12019: 22020: 32021: 3
2010peak 3 in 20202021

Frequent co-authors

Papers

showing 1–16 of 16 RSS

Stabilizing Training of Generative Adversarial Networks through Regularization

Kevin Roth, Aurelien Lucchi, Sebastian Nowozin +1 more

cs.LG · · cited 486× · arXiv ↗ · more like this ·

Deep generative models based on Generative Adversarial Networks (GANs) have demonstrated impressive sample quality but in order to work they require a careful choice of architecture, parameter initialization, and selection of hyper-parameters. This fragility is in part due to a dimensional mismatch or non-overlapping support between the model distribution and the data distribution, causing their density ratio and the associated f-divergence to be undefined. We overcome this fundamental limitation and propose a new regularization approach with low computational cost that yields a stable GAN training procedure. We demonstrate the effectiveness of this regularizer across several architectures trained on common benchmark image generation tasks. Our regularization turns GAN models into reliable building blocks for deep learning.

The Odds are Odd: A Statistical Test for Detecting Adversarial Examples

Kevin Roth, Yannic Kilcher, Thomas Hofmann

cs.LG · · cited 191× · arXiv ↗ · more like this ·

We investigate conditions under which test statistics exist that can reliably detect examples, which have been adversarially manipulated in a white-box attack. These statistics can be easily computed and calibrated by randomly corrupting inputs. They exploit certain anomalies that adversarial attacks introduce, in particular if they follow the paradigm of choosing perturbations optimally under p-norm constraints. Access to the log-odds is the only requirement to defend models. We justify our approach empirically, but also provide conditions under which detectability via the suggested test statistics is guaranteed to be effective. In our experiments, we show that it is even possible to correct test time predictions for adversarial attacks with high accuracy.

Hydra: Preserving Ensemble Diversity for Model Distillation

Linh Tran, Bastiaan S. Veeling, Kevin Roth +7 more

cs.LG · · cited 66× · arXiv ↗ · more like this ·

Ensembles of models have been empirically shown to improve predictive performance and to yield robust measures of uncertainty. However, they are expensive in computation and memory. Therefore, recent research has focused on distilling ensembles into a single compact model, reducing the computational and memory burden of the ensemble while trying to preserve its predictive behavior. Most existing distillation formulations summarize the ensemble by capturing its average predictions. As a result, the diversity of the ensemble predictions, stemming from each member, is lost. Thus, the distilled model cannot provide a measure of uncertainty comparable to that of the original ensemble. To retain more faithfully the diversity of the ensemble, we propose a distillation method based on a single multi-headed neural network, which we refer to as Hydra. The shared body network learns a joint…

How Good is the Bayes Posterior in Deep Neural Networks Really?

Florian Wenzel, Kevin Roth, Bastiaan S. Veeling +7 more

stat.ML · · cited 33× · arXiv ↗ · more like this ·

During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks. However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample efficiency there are---as of early 2020---no publicized deployments of Bayesian neural networks in industrial practice. In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD. Furthermore, we demonstrate that predictive performance is improved significantly through the use of a "cold posterior" that overcounts…

Adversarially Robust Training through Structured Gradient Regularization

Kevin Roth, Aurelien Lucchi, Sebastian Nowozin +1 more

stat.ML · · cited 24× · arXiv ↗ · more like this ·

We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations. Our regularizer can be derived as a controlled approximation from first principles, leveraging the fundamental link between training with noise and regularization. It adds very little computational overhead during learning and is simple to implement generically in standard deep learning frameworks. Our experiments provide strong evidence that structured gradient regularization can act as an effective first line of defense against attacks based on low-level signal corruption.

The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks

Jakub Swiatkowski, Kevin Roth, Bastiaan S. Veeling +7 more

cs.LG · · cited 22× · arXiv ↗ · more like this ·

Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance. In contrast, here we share a curious experimental finding that suggests instead restricting the variational distribution to a more compact parameterization. For a variety of deep Bayesian neural networks trained using Gaussian mean-field variational inference, we find that the posterior standard deviations consistently exhibit strong low-rank structure after convergence. This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance. Furthermore, we find that such factorized…

Adversarial Training is a Form of Data-dependent Operator Norm Regularization

Kevin Roth, Yannic Kilcher, Thomas Hofmann

cs.LG · · cited 13× · arXiv ↗ · more like this ·

We establish a theoretical link between adversarial training and operator norm regularization for deep neural networks. Specifically, we prove that $\ell_p$-norm constrained projected gradient ascent based adversarial training with an $\ell_q$-norm loss on the logits of clean and perturbed inputs is equivalent to data-dependent (p, q) operator norm regularization. This fundamental connection confirms the long-standing argument that a network's sensitivity to adversarial examples is tied to its spectral properties and hints at novel ways to robustify and defend against adversarial attacks. We provide extensive empirical evidence on state-of-the-art network architectures to support our theoretical results.

Emergency Medicine Gender-specific Education

John Ashurst, Alyson J. McGregor, Basmah Safdar +6 more

Gender Studies · Academic Emergency Medicine · · cited 10× · DOI ↗ · more like this ·

The 2014 Academic Emergency Medicine consensus conference has taken the first step in identifying gender-specific care as an area of importance to both emergency medicine (EM) and research. To improve patient care, we need to address educational gaps in this area concurrent with research gaps. In this article, the authors highlight the need for sex- and gender-specific education in EM and propose guidelines for medical student, resident, and faculty education. Specific examples of incorporating this content into grand rounds, simulation, bedside teaching, and journal club sessions are reviewed. Future challenges and strategies to fill the gaps in the current education model are also described. La conferencia consenso de Academic Emergency Medicine de 2014 ha dado el primer paso en identificar la atención específica de género como un área importante tanto en Medicina de Urgencias y…

Precise characterization of the prior predictive distribution of deep ReLU networks

Lorenzo Noci, Gregor Bachmann, Kevin Roth +2 more

cs.LG · · cited 5× · arXiv ↗ · more like this ·

Recent works on Bayesian neural networks (BNNs) have highlighted the need to better understand the implications of using Gaussian priors in combination with the compositional structure of the network architecture. Similar in spirit to the kind of analysis that has been developed to devise better initialization schemes for neural networks (cf. He- or Xavier initialization), we derive a precise characterization of the prior predictive distribution of finite-width ReLU networks with Gaussian weights. While theoretical results have been obtained for their heavy-tailedness, the full characterization of the prior predictive distribution (i.e. its density, CDF and moments), remained unknown prior to this work. Our analysis, based on the Meijer-G function, allows us to quantify the influence of architectural choices such as the width or depth of the network on the resulting shape of the prior…

Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect

Lorenzo Noci, Kevin Roth, Gregor Bachmann +2 more

cs.LG · · cited 2× · arXiv ↗ · more like this ·

The "cold posterior effect" (CPE) in Bayesian deep learning describes the uncomforting observation that the predictive performance of Bayesian neural networks can be significantly improved if the Bayes posterior is artificially sharpened using a temperature parameter T<1. The CPE is problematic in theory and practice and since the effect was identified many researchers have proposed hypotheses to explain the phenomenon. However, despite this intensive research effort the effect remains poorly understood. In this work we provide novel and nuanced evidence relevant to existing explanations for the cold posterior effect, disentangling three hypotheses: 1. The dataset curation hypothesis of Aitchison (2020): we show empirically that the CPE does not arise in a real curated data set but can be produced in a controlled experiment with varying curation strength. 2. The data augmentation…

Statuae Deorum Hominumque: The Distinction in Epigraphic Statuary Terminology between Divine and Human Representation in Africa Proconsularis and Beyond

Kevin Roth

Sociology and Political Science · DOAJ (DOAJ: Directory of Open Access Journals) · · cited 1× · OpenAlex ↗ · more like this ·

The sheer number of Latin words for ‘statue’ suggests that there might be some semantic difference among them. Some scholars have claimed that statua and imago refer only to statues of persons, while signum and simulacrum are reserved for statues of gods. Analysis of epigraphic evidence from Africa Proconsularis reveals that this assessment is only partially valid: statua is used indiscriminately for human and divine statues. Evidence from the rest of the Roman Empire confirms the flexibility of the term statua.

A Collaborative Recruitment Model between Honors and Athletic Programs for Student Engagement and Retention

Rich Eckert, Ashley Grimm, Kevin J. Roth +1 more

Education · Lincoln (University of Nebraska) · · cited 1× · OpenAlex ↗ · more like this ·

Acommon need in honors education is to recruit a student cohort that actively engages in educational experiences, demonstrates a motivation for academic challenge, and is likely to complete the honors program. Honors programs use varied quantitative (Green & Kimbrough) and qualitative admissions criteria to yield this desired student cohort. However, research is limited on the value of quantitative measures, i.e., SAT scores, grade point average, and/or class rank, in predicting qualities such as student engagement or outcomes such as program completion. Attempting to recruit a more diversified student cohort and to increase student engagement, the Clarion University Honors Program initiated a collaborative recruitment model with the athletic program. In addition to the goal of student engagement, this model was designed to be mutually beneficial through coordination of recruitment…

A Primer on Multi-Neuron Relaxation-based Adversarial Robustness Certification

Kevin Roth

cs.LG · · cited 0× · arXiv ↗ · more like this ·

The existence of adversarial examples poses a real danger when deep neural networks are deployed in the real world. The go-to strategy to quantify this vulnerability is to evaluate the model against specific attack algorithms. This approach is however inherently limited, as it says little about the robustness of the model against more powerful attacks not included in the evaluation. We develop a unified mathematical framework to describe relaxation-based robustness certification methods, which go beyond adversary-specific robustness evaluation and instead provide provable robustness guarantees against attacks by any adversary. We discuss the fundamental limitations posed by single-neuron relaxations and show how the recent ``k-ReLU'' multi-neuron relaxation framework of Singh et al. (2019) obtains tighter correlation-aware activation bounds by leveraging additional relational…

potentially co-operates with FOXO3a in neuronal cells to regulate PUMA and BIM expression in response to ER stress, PLoS One 7 (6

Arindam P. Ghosh, Barbara J. Klocke, Mary E. Ballestas +1 more

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

Endoplasmic reticulum (ER) stress-induced apoptosis has been implicated in various neurodegenerative diseases including Parkinson Disease, Alzheimer Disease and Huntington Disease. PUMA (p53 upregulated modulator of apoptosis) and BIM (BCL2 interacting mediator of cell death), pro-apoptotic BH3 domain-only, BCL2 family members, have previously been shown to regulate ER stress-induced cell death, but the upstream signaling pathways that regulate this response in neuronal cells are incompletely defined. Consistent with previous studies, we show that both PUMA and BIM are induced in response to ER stress in neuronal cells and that transcriptional induction of PUMA regulates ER stress-induced cell death, independent of p53. CHOP (C/EBP homologous protein also known as GADD153; gene name Ddit3), a critical initiator of ER stress-induced apoptosis, was found to regulate both PUMA and BIM…

Model of Brain Activation Predicts the Neural Collective Influence Map of the Brain

Flaviano Morone, Kevin Roth, Byungjoon Min +2 more

q-bio.NC · · arXiv ↗ · more like this ·

Efficient complex systems have a modular structure, but modularity does not guarantee robustness, because efficiency also requires an ingenious interplay of the interacting modular components. The human brain is the elemental paradigm of an efficient robust modular system interconnected as a network of networks (NoN). Understanding the emergence of robustness in such modular architectures from the interconnections of its parts is a long-standing challenge that has concerned many scientists. Current models of dependencies in NoN inspired by the power grid express interactions among modules with fragile couplings that amplify even small shocks, thus preventing functionality. Therefore, we introduce a model of NoN to shape the pattern of brain activations to form a modular environment that is robust. The model predicts the map of neural collective influencers (NCIs) in the brain, through…

Emergence of Robustness in Network of Networks

Kevin Roth, Flaviano Morone, Byungjoon Min +1 more

q-bio.NC · · arXiv ↗ · more like this ·

A model of interdependent networks of networks (NoN) has been introduced recently in the context of brain activation to identify the neural collective influencers in the brain NoN. Here we develop a new approach to derive an exact expression for the random percolation transition in Erdös-Rényi NoN. Analytical calculations are in excellent agreement with numerical simulations and highlight the robustness of the NoN against random node failures. Interestingly, the phase diagram of the model unveils particular patterns of interconnectivity for which the NoN is most vulnerable. Our results help to understand the emergence of robustness in such interdependent architectures.