Author

Mike Innes

5 papers · 2018–2020

2018peak 2 in 20182020

Frequent co-authors

Papers

showing 1–5 of 5 RSS

Flux: Elegant machine learning with Julia

Mike Innes

Artificial Intelligence · The Journal of Open Source Software · 3 May 2018 · cited 367× · DOI ↗ · more like this ·

Flux is library for machine learning (ML), written using the numerical computing language Julia The package allows models to be written using Julia's simple mathematical syntax, and applies automatic differentiation (AD) to seamlessly calculate derivatives and train the model. Meanwhile, it makes heavy use of Julia's language and compiler features to carry out code analysis and make optimisations. For example, Julia's GPU compilation support (Besard, Foket, and De Sutter 2017) can be used to JIT-compile custom GPU kernels for model layers (Innes and others 2017a).

A Differentiable Programming System to Bridge Machine Learning and Scientific Computing

Mike Innes, Alan Edelman, Keno Fischer +4 more

cs.PL · 18 Jul 2019 · cited 226× · arXiv ↗ · more like this ·

Scientific computing is increasingly incorporating the advancements in machine learning and the ability to work with large amounts of data. At the same time, machine learning models are becoming increasingly sophisticated and exhibit many features often seen in scientific computing, stressing the capabilities of machine learning frameworks. Just as the disciplines of scientific computing and machine learning have shared common underlying infrastructure in the form of numerical linear algebra, we now have the opportunity to further share new computational infrastructure, and thus ideas, in the form of Differentiable Programming. We describe Zygote, a Differentiable Programming system that is able to take gradients of general program structures. We implement this system in the Julia programming language. Our system supports almost all language constructs (control flow, recursion…

Interoperating Deep Learning models with ONNX.jl

Ayush Shridhar, Phil Tomson, Mike Innes

Computer Vision and Pattern Recognition · JuliaCon Proceedings · 21 Aug 2020 · cited 13× · DOI ↗ · more like this ·

Flux The framework makes writing layers as simple as writing mathematical formulae, and it's advanced AD, Zygote It makes heavy use of Julia's language and compiler features to carry out code analysis and make optimisations. For example, Julia's GPU compilation support [3] can be used to JIT-compile custom GPU kernels for model layers Flux also supports a number of a hardware options, from CPUs, GPUs and even TPUs via XLA.jl, that compiles Julia code to XLA: an advanced compiler for linear algebra that is capable of greatly optimizing speed and memory usage in large deep learning models. ONNX.jl is an Open Neural Network Exchange backend for the Flux.jl deep learning framework. ONNX.jl supports directly importing high quality ONNX standard models into Flux, thus saving time and reducing the need for additional computation resources. This paper aims at introducing ONNX.jl and explaining…

DiffEqFlux.jl - A Julia Library for Neural Differential Equations

Chris Rackauckas, Mike Innes, Yingbo Ma +3 more

cs.LG · 6 Feb 2019 · arXiv ↗ · more like this ·

DiffEqFlux.jl is a library for fusing neural networks and differential equations. In this work we describe differential equations from the viewpoint of data science and discuss the complementary nature between machine learning models and differential equations. We demonstrate the ability to incorporate DifferentialEquations.jl-defined differential equation problems into a Flux-defined neural network, and vice versa. The advantages of being able to use the entire DifferentialEquations.jl suite for this purpose is demonstrated by counter examples where simple integration strategies fail, but the sophisticated integration strategies provided by the DifferentialEquations.jl library succeed. This is followed by a demonstration of delay differential equations and stochastic differential equations inside of neural networks. We show high-level functionality for defining neural ordinary…

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions

Yingbo Ma, Vaibhav Dixit, Mike Innes +2 more

math.NA · December 2018 · arXiv ↗ · more like this ·

Derivatives of differential equation solutions are commonly for parameter estimation, fitting neural differential equations, and as model diagnostics. However, with a litany of choices and a Cartesian product of potential methods, it can be difficult for practitioners to understand which method is likely to be the most effective on their particular application. In this manuscript we investigate the performance characteristics of Discrete Local Sensitivity Analysis implemented via Automatic Differentiation (DSAAD) against continuous adjoint sensitivity analysis. Non-stiff and stiff biological and pharmacometric models, including a PDE discretization, are used to quantify the performance of sensitivity analysis methods. Our benchmarks show that on small systems of ODEs (approximately $<100$ parameters+ODEs), forward-mode DSAAD is more efficient than both reverse-mode and continuous…