arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Adam Pocock

18 papers · 2008–2021

2008: 12009: 02010: 22011: 02012: 32013: 22014: 22015: 12016: 32017: 02018: 02019: 12020: 12021: 2
2008peak 3 in 20122021

Frequent co-authors

Papers

showing 1–18 of 18 RSS

Conditional likelihood maximisation: a unifying framework for information theoretic feature selection

Gavin Brown, Adam Pocock, Mingjie Zhao +1 more

Computer Vision and Pattern Recognition · Research Explorer (The University of Manchester) · · cited 1,057× · OpenAlex ↗ · more like this ·

We present a unifying framework for information theoretic feature selection, bringing almost two decades of research on heuristic filter criteria under a single theoretical interpretation. This is in response to the question: “what are the implicit statistical assumptions of feature selection criteria based on mutual information?”. To answer this, we adopt a different strategy than is usual in the feature selection literature—instead of trying to define a criterion, we derive one, directly from a clearly specified objective function: the conditional likelihood of the training labels. While many hand-designed heuristic criteria try to optimize a definition of feature ‘relevancy ’ and ‘redundancy’, our approach leads to a probabilistic framework which naturally incorporates these concepts. As a result we can unify the numerous criteria published over the last two decades, and show them to…

Beyond Fano's inequality: bounds on the optimal F-score, BER, and cost-sensitive risk and their implications

Mingjie Zhao, Narayanan Unny Edakunni, Adam Pocock +1 more

Artificial Intelligence · Research Explorer (The University of Manchester) · · cited 51× · OpenAlex ↗ · more like this ·

Fano’s inequality lower bounds the probability of transmission error through a communication channel. Applied to classification problems, it provides a lower bound on the Bayes error rate and motivates the widely used Infomax principle. In modern machine learning, we are often interested in more than just the error rate. In medical diagnosis, different errors incur different cost; hence, the overall risk is cost-sensitive. Two other popular criteria are balanced errorrate (BER) and F-score. In this work, we focus on the two-class problem and use a general definition of conditional entropy (including Shannon’s as a special case) to derive upper/lower bounds on the optimal F-score, BER and cost-sensitive risk, extending Fano’s result. As a consequence, we show that Infomax is not suitable for optimizing F-score or cost-sensitive risk, in that it can potentially lead to low F-score and…

Fundamental Nano-Patterns to Characterize and Classify Java Methods

Jeremy Singer, Gavin Brown, Mikel Luján +2 more

Information Systems · Electronic Notes in Theoretical Computer Science · · cited 38× · DOI ↗ · more like this ·

Fundamental nano-patterns are simple, static, binary properties of Java methods, such as ObjectCreator and Recursive. We present a provisional catalogue of 17 such nano-patterns. We report statistical and information theoretic metrics to show the frequency of nano-pattern occurrence in a large corpus of open-source Java projects. We proceed to give two example case studies that demonstrate potential applications for nano-patterns. The first study involves a quantitative comparison of two popular Java benchmarking suites, in terms of their relative object-orientedness and diversity. The second study involves applying machine learning techniques to program comprehension, using method nano-patterns as learning features. In both studies, nano-patterns provide concise summaries of Java methods to enable efficient and effective analysis.

Minimally-Constrained Multilingual Embeddings via Artificial Code-Switching

Michael Wick, Pallika Kanani, Adam Pocock

Artificial Intelligence · Proceedings of the AAAI Conference on Artificial Intelligence · · cited 31× · DOI ↗ · more like this ·

We present a method that consumes a large corpus of multilingual text and produces a single, unified word embedding in which the word vectors generalize across languages. In contrast to current approaches that require language identification, our method is agnostic about the languages with which the documents in the corpus are expressed, and does not rely on parallel corpora to constrain the spaces. Instead we utilize a small set of human provided word translations---which are often freely and readily available. We can encode such word translations as hard constraints in the model's objective functions; however, we find that we can more naturally constrain the space by allowing words in one language to borrow distributional statistics from context words in another language. We achieve this via a process we term artificial code-switching. As the name suggests, we induce code-switching so…

Efficient feature selection using shrinkage estimators

Konstantinos Sechidis, Laura Azzimonti, Adam Pocock +3 more

Computer Vision and Pattern Recognition · Machine Learning · · cited 24× · DOI ↗ · more like this ·

Information theoretic feature selection methods quantify the importance of each feature by estimating mutual information terms to capture: the relevancy, the redundancy and the complementarity. These terms are commonly estimated by maximum likelihood, while an under-explored area of research is how to use shrinkage methods instead. Our work suggests a novel shrinkage method for data-efficient estimation of information theoretic terms. The small sample behaviour makes it particularly suitable for estimation of discrete distributions with large number of categories (bins). Using our novel estimators we derive a framework for generating feature selection criteria that capture any high-order feature interaction for redundancy and complementarity. We perform a thorough empirical study across datasets from diverse sources and using various evaluation measures. Our first finding is that our…

Augur: Data-Parallel Probabilistic Modeling

Jean-Baptiste Tristan, Daniel Huang, Joseph Tassarotti +3 more

Artificial Intelligence · · cited 22× · OpenAlex ↗ · more like this ·

Implementing inference procedures for each new probabilistic model is time-consuming and error-prone. Probabilistic programming addresses this problem by allowing a user to specify the model and then automatically generating the inference procedure. To make this practical it is important to generate high per-formance inference code. In turn, on modern architectures, high performance re-quires parallel execution. In this paper we present Augur, a probabilistic modeling language and compiler for Bayesian networks designed to make effective use of data-parallel architectures such as GPUs. We show that the compiler can generate data-parallel inference code scalable to thousands of GPU cores by making use of the conditional independence relationships in the Bayesian network. 1

A scalable implementation of information theoretic feature selection for high dimensional data

Anthony Kleerekoper, Michael Pappas, Adam Pocock +2 more

Artificial Intelligence · Research Explorer (The University of Manchester) · · cited 6× · DOI ↗ · more like this ·

With the growth of high dimensional data, feature selection is a vital component of machine learning as well as an important stand alone data analytics tool. Without it, the computation cost of big data analytics can become unmanageable and spurious correlations and noise can reduce the accuracy of any results. Feature selection removes irrelevant and redundant information leading to faster, more reliable data analysis. Feature selection techniques based on information theory are among the fastest known and the Manchester AnalyticS Toolkit (MAST) provides an efficient, parallel and scalable implementation of these methods. This paper considers a number of data structures for storing the frequency counters that underpin MAST. We show that preprocessing the data to reduce the number of zero-valued counters in an array structure results in an order of magnitude reduction in both memory…

Informative Priors for Markov Blanket Discovery

Adam Pocock, Mikel Luján, Gavin Brown

Artificial Intelligence · Research Explorer (The University of Manchester) · · cited 5× · OpenAlex ↗ · more like this ·

We present a novel interpretation of information theoretic feature selection as optimization of a discriminative model. We show that this formulation coincides with a group of mutual information based filter heuristics in the literature, and show how our probabilistic framework gives a well-founded extension for informative priors. We then derive a particular sparsity prior that recovers the well-known IAMB algorithm (Tsamardinos & Aliferis, 2003) and extend it to create a novel algorithm, IAMB-IP, that includes domain knowledge priors. In empirical evaluations, we find the new algorithm to improve Markov Blanket recovery even when a misspecified prior was used, in which half the prior knowledge was incorrect. 1

Detecting and Exorcising Statistical Demons from Language Models with Anti-Models of Negative Data

Michael L. Wick, Kate Silverstein, Jean-Baptiste Tristan +2 more

cs.CL · · cited 4× · arXiv ↗ · more like this ·

It's been said that "Language Models are Unsupervised Multitask Learners." Indeed, self-supervised language models trained on "positive" examples of English text generalize in desirable ways to many natural language tasks. But if such models can stray so far from an initial self-supervision objective, a wayward model might generalize in undesirable ways too, say to nonsensical "negative" examples of unnatural language. A key question in this work is: do language models trained on (positive) training data also generalize to (negative) test data? We use this question as a contrivance to assess the extent to which language models learn undesirable properties of text, such as n-grams, that might interfere with the learning of more desirable properties of text, such as syntax. We find that within a model family, as the number of parameters, training epochs, and data set size increase, so…

Augur: a Modeling Language for Data-Parallel Probabilistic Inference

Jean-Baptiste Tristan, Daniel Huang, Joseph Tassarotti +3 more

stat.ML · · cited 4× · arXiv ↗ · more like this ·

It is time-consuming and error-prone to implement inference procedures for each new probabilistic model. Probabilistic programming addresses this problem by allowing a user to specify the model and having a compiler automatically generate an inference procedure for it. For this approach to be practical, it is important to generate inference code that has reasonable performance. In this paper, we present a probabilistic programming language and compiler for Bayesian networks designed to make effective use of data-parallel architectures such as GPUs. Our language is fully integrated within the Scala programming language and benefits from tools such as IDE support, type-checking, and code completion. We show that the compiler can generate data-parallel inference code scalable to thousands of GPU cores by making use of the conditional independence relationships in the Bayesian network.

MSc Project Feature Selection using Information Theoretic Techniques

Adam Pocock

Artificial Intelligence · · cited 2× · OpenAlex ↗ · more like this ·

This document presents a investigation into 3 different areas of feature selection, using information theoretic methods. The first area of research is an investigation into the selection of the first feature in common feature selection algorithms. This step is often overlooked in the construction of feature selection algorithms, with the assumption that the most informative feature is the best first choice. This can be proven to be untrue, and so an investigation into how to select a better suited feature forms the first part of the research. New methods for selecting the first feature are proposed and empirically tested to see if they offer an improvement over the standard method. The second area of research is an investigation into applying the Rényi extension to infor-mation theory to standard feature selection techniques. This requires the development of a Rényi mutual information…

Vate

Daniel Goodman, Adam Pocock, Jason A. Peck +1 more

Artificial Intelligence · · cited 0× · DOI ↗ · more like this ·

Inspired by earlier work on Augur, Vate is a probabilistic programming language for the construction of JVM based probabilistic models with an Object-Oriented interface. As a compiled language it is able to examine the dependency graph of the model to produce optimised code that can be dynamically targeted to different platforms. Using Gibbs Sampling, Metropolis-Hastings and variable marginalisation it can handle a range of model types and is able to efficiently infer values, estimate probabilities, and execute models.

A Unifying Framework for Information Theoretic Feature Selection

Gavin Brown, Adam Pocock, Ming-jie Zhao +1 more

Computer Vision and Pattern Recognition · · cited 0× · OpenAlex ↗ · more like this ·

IFeature selection using mutual information is very popular. IAccepted research practice is to hand-design filter criteria to maximise “relevancy ” and minimise “redundancy”. I In contrast, here we derive a criterion, which naturally includes these concepts. This criterion provably maximises the joint likelihood of the discriminative model on the left. IThis enables us to retrofit numerous heuristics — we find that 20 years of heuristics can be understood within a single theoretical interpretation. The Relevancy/Redundancy mystery... Many successful criteria try to maximise relevancy / minimise redundancy: IMIM- JMIM(Xi) = I(Xi;Y)

Mining Static Features for Squash Prediction in Thread Level Speculation

Paraskevas Yiapanis, Jeremy Singer, Adam Pocock +2 more

Information Systems · · cited 0× · OpenAlex ↗ · more like this ·

Thread-Level Speculation (TLS) is a useful technique that facilitates automatic parallelization via optimistic execution of potentially independent threads. It is particularly valuable where there exist ambiguous dependences that cannot be resolved statically by the compiler. However, TLS bears significant overheads due to thread management which can severely degrade any performance gains when the number of squashed threads is high. This study shows the potential of applying Machine Learning techniques in order to construct models that are able to predict whether a thread will commit successfully or otherwise squash. We apply trace-driven simulation for speculative execution using a set of DaCapo Java benchmarks. Our models are able to predict with up to 92 % accuracy whether a thread will commit or squash.

5 6.4 Win/Draw/Loss results on Hailfinder................ 133

Adam C Pocock

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

6.5 Win/Draw/Loss results for Insurance................. 134 7.1 An example of a negative wI, wI(X; Y) = −0.0214......... 142 7.2 MNIST results, averaged across all digits. Each value is the difference (x 100) in Precision, Recall or F-Measure, against the costinsensitive baseline........................... 149 7.3 MNIST results, digit 4. Each value is the difference (x 100) in Precision, Recall or F-Measure, against the cost-insensitive baseline.149 7.4 MNIST results, averaged across all digits. Each value is the difference (x 100) in Precision, Recall or F-Measure, against the costinsensitive baseline........................... 152 7.5 Summary of text classification datasets................ 153 7.6 Document classification results: F-Measure W/D/L across all labels, with the costly label given w(y) = 10.............. 153 7 5.11 Validation Error curve using MADELON.............. 113 6.1…

Static java program features for intelligent squash prediction. Workshop on Statistical and Machine learning approaches to ARchitecture and compilaTion (SMART10

Jeremy Singer, Paraskevas Yiapanis, Adam Pocock +3 more

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

Abstract. The thread-level speculation paradigm parallelizes sequential applications at run-time, via optimistic execution of potentially inde-pendent threads. This enables unmodified sequential applications to ex-ploit thread-level parallelism on modern multicore architectures. How-ever a high frequency of data dependence violations between speculative threads can severely degrade the performance of thread-level speculation. Thus it is crucial to be able to schedule speculations to avoid excessive data dependence violations. Previous work in this area relies mainly on program profiling or simple heuristics to avoid thread squashes. In this paper, we investigate the use of machine learning to construct squash predictors based on static program features. On a set of standard Java benchmarks, with leave-one-out cross-validation, our approach signifi-cantly improves speculation performance…

Tribuo: Machine Learning with Provenance in Java

Adam Pocock

cs.LG · · arXiv ↗ · more like this ·

Machine Learning models are deployed across a wide range of industries, performing a wide range of tasks. Tracking these models and ensuring they behave appropriately is becoming increasingly difficult as the number of deployed models increases. There are also new regulatory burdens for ML systems which affect human lives, requiring a link between a model and its training data in high-risk situations. Current ML monitoring systems often provide provenance and experiment tracking as a layer on top of an ML library, allowing room for imperfect tracking and skew between the tracked object and the metadata. In this paper we introduce Tribuo, a Java ML library that integrates model training, inference, strong type-safety, runtime checking, and automatic provenance recording into a single framework. All Tribuo's models and evaluations record the full processing pipeline for input data, along…