---
title: Neural ODE
type: vocabulary
url: "https://www.envisioning.com/vocab/neural-ode"
summary: A neural network whose layer depth is replaced by a continuous ordinary differential equation.
year: 2018
generality: 0.55
---

# Neural ODE

A neural network whose layer depth is replaced by a continuous ordinary differential equation.
A neural ordinary differential equation, or neural ODE, is a neural network in which the transformation from input to output is parameterized as the solution to an ordinary differential equation rather than as a sequence of discrete layers. Conceptually, where a residual network evolves its hidden state by repeatedly applying a learned residual function (h_{t+1} = h_t + f(h_t)), a neural ODE treats depth as continuous time and solves dh/dt = f(h, t) using an ODE solver, producing the output as the state at some final time. The architecture was formalized by Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud in 2018 (the original Neural ODE paper at NeurIPS) and has since become a load-bearing concept in scientific machine learning, system identification, and continuous-time modeling. It is the theoretical foundation of Liquid Time-Constant networks (LNN) and the broader family of liquid neural networks developed at MIT CSAIL by Ramin Hasani and colleagues, which later spun out into Liquid AI.

Mechanically, a neural ODE has three parts. First, a learned vector field f (typically a small MLP) that specifies how the hidden state evolves. Second, an ODE solver (Euler, Runge–Kutta, or adaptive solvers like Dormand–Prince) that integrates f from the initial state to produce the output state. Third, an adjoint method for memory-efficient gradient computation, which back-propagates through the solver without storing intermediate activations. Training uses standard gradient descent via the adjoint method. At inference, the network is adaptive-cost: a simple input may need few solver steps, a complex input may need more, so computation scales with the input. This is the opposite of how a fixed-depth ResNet behaves — neural ODEs trade wall-clock predictability for input-adaptive computation. The architecture also enables hybrid continuous-discrete models where discrete events (token boundaries, irregularly sampled measurements) interrupt the continuous flow.

The advantage of the neural-ODE frame is that it gives a principled way to model continuous-time processes — physical dynamics, biological systems, irregularly sampled time series — that fixed-depth networks handle poorly. The frame also enables memory-efficient training of very deep effective networks because the adjoint method avoids storing intermediate states. The cost of the frame is that ODE solvers are not always faster than equivalent discrete networks in practice, the adjoint method has numerical-stability issues that can produce wrong gradients on hard problems, and the input-adaptive cost makes inference latency harder to budget for production systems. The frame is also underused: most deep learning research treats depth as discrete, and neural ODEs remain a niche architecture outside scientific machine learning, even though they are theoretically elegant.

Open questions include how to make neural ODEs robust on the kinds of tasks where standard transformers excel (large-scale text, image classification), where they have not matched the accuracy of competitive discrete networks. There is also continued work on the connection between neural ODEs and state-space models (S4, S5, Mamba), which solve similar continuous-time problems in different ways and have been more successful in language modeling. The deeper question — what is the right primitive for deep learning (discrete layers, continuous-time dynamics, sparse activation, or something else) — is genuinely open. The neural-ODE frame is one of the most theoretically well-grounded answers that has yet to fully win on benchmarks.

---
Source: Envisioning — Technology Research Institute (https://www.envisioning.com/vocab/neural-ode)