Deterministic nodes

RxInfer.jl offers a comprehensive set of stochastic nodes, primarily emphasizing distributions from the exponential family and related compositions, such as Gaussian with controlled variance (GCV) or autoregressive (AR) nodes. The DeltaNode stands out in this package, representing a deterministic transformation of either a single random variable or a group of them. This guide provides insights into the DeltaNode and its functionalities.

Features and Supported Inference Scenarios

The delta node supports several approximation methods for probabilistic inference. The desired approximation method depends on the nodes connected to the delta node. We differentiate the following deterministic transformation scenarios:

1. Gaussian Nodes: For delta nodes linked to strictly multivariate or univariate Gaussian distributions, the recommended methods are Linearization or Unscented transforms.
2. Exponential Family Nodes: For the delta node connected to nodes from the exponential family, the CVI (Conjugate Variational Inference) is the method of choice.
3. Stacking Delta Nodes: For scenarios where delta nodes are stacked, either Linearization or Unscented transforms are suitable.

The table below summarizes the features of the delta node in RxInfer.jl, categorized by the approximation method:

Gaussian Case

In the context of Gaussian distributions, we recommend either the Linearization or Unscented method for delta node approximation. The Linearization method provides a first-order approximation, while the Unscented method delivers a more precise second-order approximation. It's worth noting that while the Unscented method is more accurate, it may require hyperparameters tuning.

For clarity, consider the following example:

using RxInfer

@model function delta_node_example(z)
    x ~ Normal(mean=0.0, var=1.0)
    y := tanh(x)
    z ~ Normal(mean=y, var=1.0)
end

To perform inference on this model, designate the approximation method for the delta node (here, the tanh function) using the @meta specification:

delta_meta = @meta begin
    tanh() -> Linearization()
end

Meta: 
  tanh() -> Linearization()

or

delta_meta = @meta begin
    tanh() -> Unscented()
end

Meta: 
  tanh() -> Unscented{Float64, Float64, Float64, Nothing}(0.001, 2.0, 0.0, nothing)

For a deeper understanding of the Unscented method and its parameters, consult the docstrings.

Given the invertibility of tanh, indicating its inverse function can optimize the inference procedure:

delta_meta = @meta begin
    tanh() -> DeltaMeta(method = Linearization(), inverse = atanh)
end

Meta: 
  tanh() -> DeltaMeta{Linearization, typeof(atanh)}(Linearization(), atanh)

To execute the inference procedure:

infer(model = delta_node_example(), meta=delta_meta, data = (z = 1.0,))

Inference results:
  Posteriors       | available for (y, x)

This methodology is consistent even when the delta node is associated with multiple nodes. For instance:

f(x, g) = x*tanh(g)

f (generic function with 1 method)

@model function delta_node_example(z)
    x ~ Normal(mean=1.0, var=1.0)
    g ~ Normal(mean=1.0, var=1.0)
    y := f(x, g)
    z ~ Normal(mean=y, var=0.1)
end

The corresponding meta specification is:

delta_meta = @meta begin
    f() -> DeltaMeta(method = Linearization())
end

Meta: 
  f() -> DeltaMeta{Linearization, Nothing}(Linearization(), nothing)

or simply

delta_meta = @meta begin
    f() -> Linearization()
end

Meta: 
  f() -> Linearization()

If specific functions outline the backward relation of variables within the f function, you can provide a tuple of inverse functions in the order of the variables:

f_back_x(out, g) = out/tanh(g)
f_back_g(out, x) = atanh(out/x)

f_back_g (generic function with 1 method)

delta_meta = @meta begin
    f() -> DeltaMeta(method = Linearization(), inverse=(f_back_x, f_back_g))
end

Meta: 
  f() -> DeltaMeta{Linearization, Tuple{typeof(Main.f_back_x), typeof(Main.f_back_g)}}(Linearization(), (Main.f_back_x, Main.f_back_g))

Exponential Family Case

When the delta node is associated with nodes from the exponential family (excluding Gaussians), the Linearization and Unscented methods are not applicable. In such cases, the CVI (Conjugate Variational Inference) is available. Here's a modified example:

using RxInfer

@model function delta_node_example1(z)
    x ~ Gamma(shape=1.0, rate=1.0)
    y := tanh(x)
    z ~ Bernoulli(y)
end

The corresponding meta specification can be represented as:

using StableRNGs
using Optimisers

delta_meta = @meta begin
    tanh() -> DeltaMeta(method = CVI(StableRNG(42), 100, 100, Optimisers.Descent(0.01)))
end

Meta: 
  tanh() -> DeltaMeta{ProdCVI{StableRNGs.LehmerRNG, Optimisers.Descent{Float64}, ForwardDiffGrad{0}, true}, Nothing}(ProdCVI{StableRNGs.LehmerRNG, Optimisers.Descent{Float64}, ForwardDiffGrad{0}, true}(StableRNGs.LehmerRNG(state=0x00000000000000000000000000000055), 100, 100, Optimisers.Descent{Float64}(0.01), ForwardDiffGrad{0}(), 1, Val{true}(), true), nothing)

Consult the ProdCVI docstrings for a detailed explanation of these parameters.