Normal distributions

A normal node ties a variable out to a mean and a spread: a variance, a precision, a covariance matrix or a precision matrix. The page covers the univariate and multivariate normals in each of their parametrisations, and the half-normal.

With the spread known, the rules towards out and the mean are exact belief propagation: a normal message on the mean, convolved with the noise, gives a normal message on out, and the same holds the other way. With the spread unknown, belief propagation towards it gives no closed-form message for NormalMeanVariance, and the other normals have no such rule at all. The rules towards the spread are therefore variational: they take marginals and return a Gamma, a Wishart or their inverses. In practice, you choose a factorisation that puts an unknown spread in a cluster of its own, such as q(out, μ) q(τ) or the mean field q(out) q(μ) q(τ).

The nodes are ExponentialFamily's types, declared as nodes here. They all run under DefaultAlgorithm, so a model never names an algorithm for them.

In the interface tables below, a rule takes messages from the interfaces in its target's own cluster, and it takes marginals from the interfaces in other clusters. "Normal" means any univariate or multivariate normal of ExponentialFamily in any parametrisation. "Any" means a marginal with the expectations the rule needs.

Example

The mean-field message towards the mean of a NormalMeanPrecision node takes the marginals of out and of the precision. It is a normal around E[out] with precision E[τ]:

julia> using StandardMessagePassingRules, MessagePassingRulesBase, ExponentialFamily, BayesBase

julia> message = getresult(@call_message_update_rule(
           node = NormalMeanPrecision, target = :μ,
           q = (out = PointMass(2.0), τ = GammaShapeRate(2.0, 1.0)),
       ));

julia> mean(message) ≈ 2.0 && precision(message) ≈ 2.0
true

NormalMeanVariance

\[p(\mathrm{out} \mid μ, v) = \frac{1}{\sqrt{2π v}} \exp\left(-\frac{(\mathrm{out} - μ)^2}{2v}\right)\]

MessagePassingRulesBase.nodespec(NormalMeanVariance)

MessagePassingRulesBase.rule_coverage(NormalMeanVariance)

The belief propagation message towards out, from a normal message on the mean and a known variance, adds the variances:

@call_message_update_rule(
    node = NormalMeanVariance, target = :out,
    m = (μ = NormalMeanVariance(1.0, 1.0), v = PointMass(2.0)),
)

args->begin
        (μ_mean, μ_var) = mean_var(args.m[:μ])
        NormalMeanVariance(μ_mean, μ_var + mean(args.m[:v]))
    end

The variational message takes the marginals of μ and v. It keeps the mean of q(μ), ignores its variance, and uses 1/E[1/v] = 0.5 of the Gamma marginal as the variance:

@call_message_update_rule(
    node = NormalMeanVariance, target = :out,
    q = (μ = NormalMeanVariance(1.0, 1.0), v = GammaShapeRate(3.0, 4.0)),
)

args->NormalMeanVariance(mean(args.q[:μ]), gaussian_variational_variance(args.q[:v]))

Towards out and μ, the node has belief propagation, variational and structured rules. Towards v, belief propagation from normal or known out and μ gives a log-density on the half line, a ContinuousUnivariateLogPdf, which has no closed-form family. The variational rule towards v gives an unchecked GammaInverse with shape -1/2, which is a likelihood rather than a distribution. The average energy covers the mean field and q(out, μ) q(v).

NormalMeanPrecision

\[p(\mathrm{out} \mid μ, τ) = \sqrt{\frac{τ}{2π}} \exp\left(-\frac{τ (\mathrm{out} - μ)^2}{2}\right)\]

MessagePassingRulesBase.nodespec(NormalMeanPrecision)

MessagePassingRulesBase.rule_coverage(NormalMeanPrecision)

With a known precision, the message towards out stays in precision form. The precisions combine as 1 / (1/1 + 1/2) = 2/3:

@call_message_update_rule(
    node = NormalMeanPrecision, target = :out,
    m = (μ = NormalMeanPrecision(1.0, 1.0), τ = PointMass(2.0)),
)

args->begin
        (μ_mean, μ_var) = mean_var(args.m[:μ])
        NormalMeanPrecision(μ_mean, inv(μ_var + inv(mean(args.m[:τ]))))
    end

The variational message towards τ takes the marginals of out and μ, and it is a Gamma:

@call_message_update_rule(
    node = NormalMeanPrecision, target = :τ,
    q = (out = PointMass(3.0), μ = NormalMeanPrecision(1.0, 1.0)),
)

args->begin
        θ = 2 / (var(args.q[:out]) + var(args.q[:μ]) + abs2(mean(args.q[:out]) - mean(args.q[:μ])))
        Gamma(convert(typeof(θ), 1.5), θ)
    end

The rules towards τ take q(out) q(μ) or q(out, μ). There is no belief propagation rule towards τ, so an unknown precision needs a factorisation that separates it from out and μ. Asking for one throws a RuleNotFoundError, which lists the rules that do exist:

try
    @call_message_update_rule(
        node = NormalMeanPrecision, target = :τ,
        m = (out = PointMass(3.0), μ = NormalMeanPrecision(1.0, 1.0)),
    )
catch err
    showerror(stdout, err)
end

RuleNotFoundError: no message rule for NormalMeanPrecision towards :τ under MessagePassingRulesBase.DefaultAlgorithm() takes the inputs (m[:out]::BayesBase.PointMass{Float64}, m[:μ]::ExponentialFamily.NormalMeanPrecision{Float64})
  no rule consumes this set of inputs under this algorithm (no rule of this shape)
  what to try: the factorisation decides which inputs a rule receives (messages from the target's cluster, marginals of the other clusters): choose one that delivers the inputs a rule below takes (in RxInfer, `@constraints`), or define a rule for these inputs
  near misses:
    rule at ~/work/StandardMessagePassingRules.jl/StandardMessagePassingRules.jl/src/rules/normal_mean_precision/precision.jl:1
      ✓ algorithm MessagePassingRulesBase.DefaultAlgorithm
      ✗ q[:out]::Any  not provided
      ✗ q[:μ]::Any  not provided
      ✗ m[:out]::BayesBase.PointMass{Float64}  provided but not consumed
      ✗ m[:μ]::ExponentialFamily.NormalMeanPrecision{Float64}  provided but not consumed
    rule at ~/work/StandardMessagePassingRules.jl/StandardMessagePassingRules.jl/src/rules/normal_mean_precision/precision.jl:10
      ✓ algorithm MessagePassingRulesBase.DefaultAlgorithm
      ✗ q[:out, :μ]::Any  not provided
      ✗ m[:out]::BayesBase.PointMass{Float64}  provided but not consumed
      ✗ m[:μ]::ExponentialFamily.NormalMeanPrecision{Float64}  provided but not consumed

MvNormalMeanCovariance

\[p(\mathrm{out} \mid μ, Σ) = |2π Σ|^{-1/2} \exp\left(-\tfrac{1}{2} (\mathrm{out} - μ)^\top Σ^{-1} (\mathrm{out} - μ)\right)\]

MessagePassingRulesBase.nodespec(MvNormalMeanCovariance)

MessagePassingRulesBase.rule_coverage(MvNormalMeanCovariance)

As in the univariate case, the belief propagation message towards out adds the covariances:

@call_message_update_rule(
    node = MvNormalMeanCovariance, target = :out,
    m = (μ = MvNormalMeanCovariance([1.0, 2.0], diageye(2)), Σ = PointMass([2.0 0.0; 0.0 2.0])),
)

args->begin
        (μ, V) = mean_cov(args.m[:μ])
        MvNormalMeanCovariance(μ, V + mean(args.m[:Σ]))
    end

The rules towards Σ are variational only, and they give an inverse-Wishart likelihood. There is no belief propagation rule towards Σ.

MvNormalMeanPrecision

\[p(\mathrm{out} \mid μ, Λ) = \left|\frac{Λ}{2π}\right|^{1/2} \exp\left(-\tfrac{1}{2} (\mathrm{out} - μ)^\top Λ (\mathrm{out} - μ)\right)\]

MessagePassingRulesBase.rule_coverage(MvNormalMeanPrecision)

The rules towards Λ are variational only, and they give a Wishart likelihood. They apply the context's matrix_correction to its scale, and none by default. There is no belief propagation rule towards Λ.

MvNormalWeightedMeanPrecision

\[p(\mathrm{out} \mid ξ, Λ) = \mathcal{N}(\mathrm{out} \mid Λ^{-1} ξ, Λ^{-1})\]

MessagePassingRulesBase.rule_coverage(MvNormalWeightedMeanPrecision)

Limitations. The node has the message towards out and the joint marginal with known parameters only. There is no rule towards ξ or Λ: they must be known, or have their marginals from elsewhere.

MvNormalMeanScalePrecision

\[p(\mathrm{out} \mid μ, γ) = \mathcal{N}(\mathrm{out} \mid μ, (γ I)^{-1})\]

MessagePassingRulesBase.rule_coverage(MvNormalMeanScalePrecision)

Every message rule takes γ as a marginal, so γ is always in a cluster of its own. The rules towards γ are variational, and they give a GammaShapeRate.

MvNormalMeanScaleMatrixPrecision

\[p(\mathrm{out} \mid μ, γ, G) = \mathcal{N}(\mathrm{out} \mid μ, (γ G)^{-1})\]

MessagePassingRulesBase.rule_coverage(MvNormalMeanScaleMatrixPrecision)

As for MvNormalMeanScalePrecision, every message rule takes γ and G as marginals. The rules towards them are variational.

HalfNormal

\[p(\mathrm{out} \mid v) = \frac{2}{\sqrt{2π v}} \exp\left(-\frac{\mathrm{out}^2}{2v}\right), \quad \mathrm{out} \ge 0\]

MessagePassingRulesBase.rule_coverage(HalfNormal)

Limitations. HalfNormal has one message, a truncated normal towards out from a known variance. There is no rule towards v, and no marginal rule.

HalfNormal

julia> energy = getresult(@call_average_energy(node = HalfNormal, q = (out = PointMass(1.0), v = PointMass(1.0))));

julia> energy ≈ (log(π / 2) + 1) / 2
true