Author

Mark Neumann

16 papers · 1997–2025

1997peak 5 in 20182025

Frequent co-authors

Papers

showing 1–16 of 16 RSS

Deep contextualized word representations

Matthew E. Peters, Mark Neumann, Mohit Iyyer +4 more

cs.CL · February 2018 · cited 12,289× · arXiv ↗ · more like this ·

We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

AllenNLP: A Deep Semantic Natural Language Processing Platform

Matt Gardner, Joel Grus, Mark Neumann +6 more

cs.CL · March 2018 · cited 1,338× · arXiv ↗ · more like this ·

This paper describes AllenNLP, a platform for research on deep learning methods in natural language understanding. AllenNLP is designed to support researchers who want to build novel language understanding models quickly and easily. It is built on top of PyTorch, allowing for dynamic computation graphs, and provides (1) a flexible data API that handles intelligent batching and padding, (2) high-level abstractions for common operations in working with text, and (3) a modular and extensible experiment framework that makes doing good science easy. It also includes reference implementations of high quality approaches for both core semantic problems (e.g. semantic role labeling (Palmer et al., 2005)) and language understanding applications (e.g. machine comprehension (Rajpurkar et al., 2016)). AllenNLP is an ongoing open-source effort maintained by engineers and researchers at the Allen…

Knowledge Enhanced Contextual Word Representations

Matthew E. Peters, Mark Neumann, Robert L. Logan IV +4 more

cs.CL · September 2019 · cited 744× · arXiv ↗ · more like this ·

Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a general method to embed multiple knowledge bases (KBs) into large scale models, and thereby enhance their representations with structured, human-curated knowledge. For each KB, we first use an integrated entity linker to retrieve relevant entity embeddings, then update contextual word representations via a form of word-to-entity attention. In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that combines a small amount of entity linking supervision with a large amount of raw text. After integrating WordNet and a subset of Wikipedia into BERT, the knowledge enhanced…

Dissecting Contextual Word Embeddings: Architecture and Representation

Matthew E. Peters, Mark Neumann, Luke Zettlemoyer +1 more

cs.CL · August 2018 · cited 503× · arXiv ↗ · more like this ·

Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks. However, many questions remain as to how and why these models are so effective. In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g. LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the representations that are learned. We show there is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks. Additionally, all architectures learn representations that vary with network depth, from exclusively morphological based at the word embedding layer through local syntax based in the lower…

Orb: A Fast, Scalable Neural Network Potential

Mark Neumann, James Gin, Benjamin Rhodes +5 more

cond-mat.mtrl-sci · 29 Oct 2024 · cited 180× · arXiv ↗ · more like this ·

We introduce Orb, a family of universal interatomic potentials for atomistic modelling of materials. Orb models are 3-6 times faster than existing universal potentials, stable under simulation for a range of out of distribution materials and, upon release, represented a 31% reduction in error over other methods on the Matbench Discovery benchmark. We explore several aspects of foundation model development for materials, with a focus on diffusion pretraining. We evaluate Orb as a model for geometry optimization, Monte Carlo and molecular dynamics simulations.

Grammar-based Neural Text-to-SQL Generation

Kevin Lin, Ben Bogin, Mark Neumann +2 more

cs.CL · 30 May 2019 · cited 65× · arXiv ↗ · more like this ·

The sequence-to-sequence paradigm employed by neural text-to-SQL models typically performs token-level decoding and does not consider generating SQL hierarchically from a grammar. Grammar-based decoding has shown significant improvements for other semantic parsing tasks, but SQL and other general programming languages have complexities not present in logical formalisms that make writing hierarchical grammars difficult. We introduce techniques to handle these complexities, showing how to construct a schema-dependent grammar with minimal over-generation. We analyze these techniques on ATIS and Spider, two challenging text-to-SQL datasets, demonstrating that they yield 14--18\% relative reductions in error.

PAWLS: PDF Annotation With Labels and Structure

Mark Neumann, Zejiang Shen, Sam Skjonsberg

cs.CL · 25 Jan 2021 · cited 23× · arXiv ↗ · more like this ·

Adobe's Portable Document Format (PDF) is a popular way of distributing view-only documents with a rich visual markup. This presents a challenge to NLP practitioners who wish to use the information contained within PDF documents for training models or data analysis, because annotating these documents is difficult. In this paper, we present PDF Annotation with Labels and Structure (PAWLS), a new annotation tool designed specifically for the PDF document format. PAWLS is particularly suited for mixed-mode annotation and scenarios in which annotators require extended context to annotate accurately. PAWLS supports span-based textual annotation, N-ary relations and freeform, non-textual bounding boxes, all of which can be exported in convenient formats for training multi-modal machine learning models. A read-only PAWLS server is available at https://pawls.apps.allenai.org/ and the source…

S2ORC: The Semantic Scholar Open Research Corpus

Kyle Lo, Lucy Lu Wang, Mark Neumann +2 more

cs.CL · November 2019 · cited 11× · arXiv ↗ · more like this ·

We introduce S2ORC, a large corpus of 81.1M English-language academic papers spanning many academic disciplines. The corpus consists of rich metadata, paper abstracts, resolved bibliographic references, as well as structured full text for 8.1M open access papers. Full text is annotated with automatically-detected inline mentions of citations, figures, and tables, each linked to their corresponding paper objects. In S2ORC, we aggregate papers from hundreds of academic publishers and digital archives into a unified source, and create the largest publicly-available collection of machine-readable academic text to date. We hope this resource will facilitate research and development of tools and tasks for text mining over academic text.

PySBD: Pragmatic Sentence Boundary Disambiguation

Nipun Sadvilkar, Mark Neumann

cs.CL · 19 Oct 2020 · cited 8× · arXiv ↗ · more like this ·

In this paper, we present a rule-based sentence boundary disambiguation Python package that works out-of-the-box for 22 languages. We aim to provide a realistic segmenter which can provide logical sentences even when the format and domain of the input text is unknown. In our work, we adapt the Golden Rules Set (a language-specific set of sentence boundary exemplars) originally implemented as a ruby gem - pragmatic_segmenter - which we ported to Python with additional improvements and functionality. PySBD passes 97.92% of the Golden Rule Set exemplars for English, an improvement of 25% over the next best open-source Python tool.

Learning to Reason With Adaptive Computation

Mark Neumann, Pontus Stenetorp, Sebastian Riedel

cs.CL · October 2016 · cited 7× · arXiv ↗ · more like this ·

Multi-hop inference is necessary for machine learning systems to successfully solve tasks such as Recognising Textual Entailment and Machine Reading. In this work, we demonstrate the effectiveness of adaptive computation for learning the number of inference steps required for examples of different complexity and that learning the correct number of inference steps is difficult. We introduce the first model involving Adaptive Computation Time which provides a small performance benefit on top of a similar model without an adaptive component as well as enabling considerable insight into the reasoning process of the model.

Writing Code for NLP Research

Matt Gardner, Mark E Neumann, Joël Grus +1 more

Artificial Intelligence · Empirical Methods in Natural Language Processing · 2018 · cited 3× · OpenAlex ↗ · more like this ·

Doing modern NLP research requires writing code. Good code enables fast prototyping, easy debugging, controlled experiments, and accessible visualizations that help researchers understand what a model is doing. Bad code leads to research that is at best hard to reproduce and extend, and at worst simply incorrect. Indeed, there is a growing recognition of the importance of having good tools to assist good research in our field, as the upcoming workshop on open source software for NLP demonstrates. This tutorial aims to share best practices for writing code for NLP research, drawing on the instructors' experience designing the recently-released AllenNLP toolkit, a PyTorch-based library for deep learning NLP research. We will explain how a library with the right abstractions and components enables better code and better science, using models implemented in AllenNLP as examples.…

Beyond the myths exhibition

J. Gothard, Mark Neumann

Education · Murdoch Research Repository (Murdoch University) · 1997 · cited 0× · OpenAlex ↗ · more like this ·

Down syndrome is the most common form of intellectual disability and occurs at a rate of about one in eight hundred live births in Western Australia, as it does throughout the world. Yet it remains largely a mystery to those not touched by it directly. Myths, rather than realities, are how the general public see Down syndrome. The aim of the exhibition was to breakdown some of the commonly held myths about Down syndrome, to highlight the individuality of each person with Down syndrome and to show their potential. The intention was to show people with Down syndrome leading not extraordinary, but 'ordinary' lives.\n\nThrough a series of black and white photographs focusing on the daily lives of people with Down syndrome, this exhibition highlights the richness and variety of their everyday experiences. Young children and their families, teenagers and adults pursuing independent…

Mofasa: A Step Change in Metal-Organic Framework Generation

Vaidotas Simkus, Anders Christensen, Steven Bennett +5 more

cs.LG · December 2025 · arXiv ↗ · more like this ·

Mofasa is an all-atom latent diffusion model with state-of-the-art performance for generating Metal-Organic Frameworks (MOFs). These are highly porous crystalline materials used to harvest water from desert air, capture carbon dioxide, store toxic gases and catalyse chemical reactions. In recognition of their value, the development of MOFs recently received a Nobel Prize in Chemistry. In many ways, MOFs are well-suited for exploiting generative models in chemistry: they are rationally-designable materials with a large combinatorial design space and strong structure-property couplings. And yet, to date, a high performance generative model has been lacking. To fill this gap, we introduce Mofasa, a general-purpose latent diffusion model that jointly samples positions, atom-types and lattice vectors for systems as large as 500 atoms. Mofasa avoids handcrafted assembly algorithms common in…

Orb-v3: atomistic simulation at scale

Benjamin Rhodes, Sander Vandenhaute, Vaidotas Šimkus +4 more

cond-mat.mtrl-sci · 10 Apr 2025 · arXiv ↗ · more like this ·

We introduce Orb-v3, the next generation of the Orb family of universal interatomic potentials. Models in this family expand the performance-speed-memory Pareto frontier, offering near SoTA performance across a range of evaluations with a >10x reduction in latency and > 8x reduction in memory. Our experiments systematically traverse this frontier, charting the trade-off induced by roto-equivariance, conservatism and graph sparsity. Contrary to recent literature, we find that non-equivariant, non-conservative architectures can accurately model physical properties, including those which require higher-order derivatives of the potential energy surface. This model release is guided by the principle that the most valuable foundation models for atomic simulation will excel on all fronts: accuracy, latency and system size scalability. The reward for doing so is a new era of computational…

ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing

Mark Neumann, Daniel King, Iz Beltagy +1 more

cs.CL · February 2019 · arXiv ↗ · more like this ·

Despite recent advances in natural language processing, many statistical models for processing text perform extremely poorly under domain shift. Processing biomedical and clinical text is a critically important application area of natural language processing, for which there are few robust, practical, publicly available models. This paper describes scispaCy, a new tool for practical biomedical/scientific text processing, which heavily leverages the spaCy library. We detail the performance of two packages of models released in scispaCy and demonstrate their robustness on several tasks and datasets. Models and code are available at https://allenai.github.io/scispacy/

Ontology Alignment in the Biomedical Domain Using Entity Definitions and Context

Lucy Lu Wang, Chandra Bhagavatula, Mark Neumann +3 more

cs.CL · 20 Jun 2018 · arXiv ↗ · more like this ·

Ontology alignment is the task of identifying semantically equivalent entities from two given ontologies. Different ontologies have different representations of the same entity, resulting in a need to de-duplicate entities when merging ontologies. We propose a method for enriching entities in an ontology with external definition and context information, and use this additional information for ontology alignment. We develop a neural architecture capable of encoding the additional information when available, and show that the addition of external data results in an F1-score of 0.69 on the Ontology Alignment Evaluation Initiative (OAEI) largebio SNOMED-NCI subtask, comparable with the entity-level matchers in a SOTA system.