Neural Differential Equation

• about as obvious as it sounds, you parameterize the dynamics of a differential equation. thats about it lol

For ODEs

$$\frac{d\mathbf{h}(t)}{dt}=f_{\theta }(\mathbf{h}(t),t)$$

where $\theta$ network parameters determine how state changes

What is Being Learned?

just a vector field, at each state which direction and how quickly should the system move.

$$\mathbf{h}(t_1)=\mathbf{h}(t_0)+{\int }_{t_0}^{t_1}{f}_{\theta }(\mathbf{h}(t),t)dt.$$

Eulers Method gives ${\mathbf{h}}_{k+1}={\mathbf{h}}_k+\Delta t{f}_{\theta }({\mathbf{h}}_k,t_k)$, which looks like a Residual Network update. In a neural ODE, solver controls integration steps instead of treating each step as a separately learned layer. See Chen et al. (2018).

Latent State, Inputs, and Observations

For sparse neural or behavioral measurements, a useful formulation is

$$\dot{\mathbf{z}}(t)=f_{\theta }(\mathbf{z}(t),\mathbf{u}(t),t),{\mathbf{y}}_i=g_{\varphi }(\mathbf{z}(t_i))+{\mathbf{ϵ}}_i.$$

• ${\mathbf{y}}_i$: measured EEG or behavioral features at recorded times.
• $\mathbf{z}(t)$: inferred latent state; not directly measured.
• $\mathbf{u}(t)$: observed inputs, such as stimulus features.
• $g_{\varphi }$: observation model, connecting the latent state to a particular sensor.
• ${\mathbf{ϵ}}_i$: observation noise; uncertainty about dynamics is a separate issue.

some encoder can infer an initial-state distribution from observations. The ODE evolves that state; the decoder predicts measurements. This is the basic appeal of Latent ODEs: inference over hidden continuous-time trajectories with observations at irregular times. It does not make the hidden state uniquely recoverable.

• LFADS is a close concept neighbor here. infer initial conditions, a recurrent generator, and optionally time-varying inputs from neural recordings. Its original generator is an RNN, not a neural ODE. The connection is the separation of latent evolution from the observation model. Results on population spiking do not directly establish suitability for sparse scalp EEG. Pandarinath et al. (2018)