SoftDotMessagePassingRules

The SoftDot node and its rules: a dot product of two unknowns, softened by Gaussian noise so that every message is in closed form. Use it for a Bayesian linear regression, y ~ N(θᵀx, γ⁻¹) with the noise precision γ learnt, or wherever a dot product joins two Gaussian unknowns, which the deterministic dot product node of StandardMessagePassingRules cannot do in closed form. This site is a single page.

SoftDotMessagePassingRules

julia> using MessagePassingRulesBase, ExponentialFamily, BayesBase

julia> q_θ = MvNormalMeanCovariance([1.0, 2.0], [1.0 0.0; 0.0 1.0]);

julia> q_x = MvNormalMeanCovariance([3.0, 1.0], [1.0 0.0; 0.0 1.0]);

julia> result = @call_message_update_rule(node = SoftDot, target = :y, q = (θ = q_θ, x = q_x, γ = GammaShapeRate(2.0, 1.0)));

julia> all(mean_precision(getresult(result)) .≈ (5.0, 2.0))
true

Overview

A deterministic node y = θᵀx with both θ and x unknown has no closed-form messages. SoftDot replaces the constraint by a Gaussian likelihood of precision γ around it. The softening makes it a stochastic node, a density over its variables.

Under variational message passing, the node then sends a normal message towards each of y, θ and x and a gamma message towards γ, all in closed form. It contributes an average energy to the Bethe free energy. A large γ, or a prior on γ concentrated at large values, brings it close to the deterministic dot product.

Definition

\[p(y \mid θ, x, γ) = \mathcal{N}\left(y \mid θ^\top x, γ^{-1}\right) = \sqrt{\frac{γ}{2π}} \exp\left(-\frac{γ}{2} \left(y - θ^\top x\right)^2\right)\]

The output y is normal around the dot product θᵀx, with variance γ⁻¹. y is a scalar; θ and x are both scalars or both vectors of the same length.

Interfaces

using SoftDotMessagePassingRules, MessagePassingRulesBase, BayesBase, ExponentialFamily
MessagePassingRulesBase.nodespec(SoftDot)

The node has four interfaces, two of them with an alias that a model may write instead of the Greek name. The node itself has the alias softdot.

Here q(θ) is the marginal of θ. The rules towards θ and x return a normal in weighted-mean–precision form, univariate or multivariate as the factor is. The rule towards y returns a univariate normal and the rule towards γ a GammaShapeRate. Under the structured factorisation the joint q(y, x) is a multivariate normal over [y; x].

Algorithm

SoftDot runs under DefaultAlgorithm, which takes no parameters; a model need not name it. The node declares no dependencies, so each rule takes the inputs of the default scheme.

Supported rules

MessagePassingRulesBase.rule_coverage(SoftDot)

rule_coverage counts two rules per target, one per factorisation:

• the mean field q(y)q(θ)q(x)q(γ), where every rule reads the marginals of the other interfaces;
• the structured q(y, x)q(θ)q(γ), where the rules towards y and x read the other one's message and the rules towards θ and γ read the joint q(y, x).

The marginal rule q(y, x) computes that joint from the messages on y and x. There are two average energies, one per factorisation. There are no belief propagation rules.

Example

The message towards γ under mean field is Γ(3/2, ⟨(y - θᵀx)²⟩ / 2). With every marginal N(1, 1), the expected square is E[y²] - 2E[y]E[θ]E[x] + E[θ²]E[x²] = 2 - 2 + 4:

@call_message_update_rule(
    node = SoftDot, target = :γ,
    q = (y = NormalMeanVariance(1.0, 1.0), θ = NormalMeanVariance(1.0, 1.0), x = NormalMeanVariance(1.0, 1.0)),
)

args->begin
        (my, Vy) = mean_cov(args.q[:y])
        (mθ, Vθ) = mean_cov(args.q[:θ])
        (mx, Vx) = mean_cov(args.q[:x])
        T = promote_paramfloattype(args.q[:y], args.q[:θ], args.q[:x])
        β = add_outer(Vy, my) / 2
        β -= my * mθ' * mx
        β += (trace_product(Vx, Vθ) + (mθ' * Vx) * mθ + (mx' * Vθ) * mx + (mθ' * mx) * (mx' * mθ)) / 2
        GammaShapeRate(convert(T, 3 / 2), β)
    end

@call_message_update_rule runs the rule and draws the marginals it read. The result is Γ(1.5, 2.0), which getresult extracts:

julia> using MessagePassingRulesBase, SoftDotMessagePassingRules, ExponentialFamily, BayesBase

julia> q = (y = NormalMeanVariance(1.0, 1.0), θ = NormalMeanVariance(1.0, 1.0), x = NormalMeanVariance(1.0, 1.0));

julia> message = getresult(@call_message_update_rule(node = SoftDot, target = :γ, q = q));

julia> shape(message) ≈ 1.5 && rate(message) ≈ 2.0
true

Limitations

• Variational rules only: there is no belief propagation rule, so the node needs a factorisation, mean-field or q(y, x)q(θ)q(γ). It also needs initial marginals where the schedule reads them before they are computed.
• No rules for other factorisations, such as a joint over θ and x or over y and θ.
• y is a scalar: a vector-valued output needs one node per component. The mean-field rules and energy check it, and that θ and x are both univariate or both multivariate, and throw a RuleInputError otherwise.
• The rules take normal marginals and messages on y, θ and x and a gamma-like marginal on γ (anything with a mean, and a mean of the logarithm for the energy).

API

The node and its alias, the name an RxInfer model writes:

SoftDot

p(y | θ, x, γ) = N(y | θᵀx, γ⁻¹),

softdot