Log scales

A message's log scale is a scalar carried beside it. A rule returns a normalised distribution, but what it computes may be an unnormalised function. The log scale is the constant that relates the two:

\[\text{message}(x) = \exp(\text{logscale}) \cdot \hat{p}(x),\]

where $\hat{p}$ is the distribution the rule returns.

Belief propagation is the common case. A message $\mu(x) = \int f(x, y, \dots) \prod_i \mu_i(y_i) \,\mathrm{d}y$ is in general not normalised. In an acyclic graph, the log scales of such messages add up to the log model evidence, which an engine can then read at any edge. A log scale exists only for a message whose integral is finite: an exact message may be improper, with no normalising constant at all (Improper messages). A naive variational message, $\exp \mathbb{E}_q[\log f]$, has no constant with that meaning, so its log scale is undefined.

This page covers the rule's side: what a rule declares, and how it reads the log scales of its inputs. The engine's documentation covers how an engine propagates log scales through products, and what the evidence means.

What a rule declares

The normalised distribution a rule returns does not reveal its log scale. Beta(2, 1) looks the same whether or not a constant was divided out to get it. A message rule therefore states its log scale with the logscale keyword of @define_message_update_rule, in one of five forms:

A marginal rule and an average energy have no log scale, and they do not accept the keyword.

A constant

Zero is the common constant. The convolution of normals integrates to one. A stochastic node's message towards out from point-mass inputs is its own normalised density.

Zero is a fact about the rule, not a safe default. A Bernoulli node with its output observed at 1 sends $p \mapsto p$ towards p. That function integrates to $\tfrac{1}{2}$ over $[0, 1]$, so the rule returns Beta(2, 1) and declares logscale = loghalf.

An Irrational constant, or an integer, takes the precision of whatever it is added to. A Float32 model therefore stays Float32.

A function of the inputs

The function takes some of the slots (algo, ctx, args), named in that order. The node below scales its input, out = a · in. Its message towards in is the normal message on out evaluated at $a x$, which integrates to $1 / |a|$:

\[\mathcal{N}(a x \mid m, v) = \frac{1}{|a|}\, \mathcal{N}\big(x \mid m / a,\, v / a^2\big).\]

using MessagePassingRulesBase, BayesBase, ExponentialFamily

struct Scale end   # out = a · in

@define_factor_node(node = Scale, type = Deterministic, interfaces = [:out, :in, :a])

@define_message_update_rule(
    node = Scale, target = :in, args = (m[:out]::NormalMeanVariance, m[:a]::PointMass),
    logscale = (args) -> -log(abs(mean(args.m[:a]))),
    body = (args) -> begin
        a = mean(args.m[:a])
        NormalMeanVariance(mean(args.m[:out]) / a, var(args.m[:out]) / a^2)
    end,
)

@call_message_update_rule(
    node = Scale, target = :in,
    m = (out = NormalMeanVariance(4.0, 1.0), a = PointMass(2.0)),
)

args->begin
        a = mean(args.m[:a])
        NormalMeanVariance(mean(args.m[:out]) / a, var(args.m[:out]) / a ^ 2)
    end

The result's log scale is $-\log 2$.

From the body, or none

With logscale = from_body, the body returns with_logscale(result, logscale), and whoever runs the rule unwraps it. A rule that omits the keyword gives its message an UndefinedLogScale that names the rule.

julia> using MessagePassingRulesBase

julia> struct Halve end

julia> @define_factor_node(node = Halve, type = Deterministic, interfaces = [:out, :in])

julia> @define_message_update_rule(
           node = Halve, target = :out, args = (m[:in]::Real,),
           logscale = from_body,
           body = (args) -> with_logscale(args.m[:in] / 2, log(2)),
       )

julia> @define_message_update_rule(node = Halve, target = :in, args = (m[:out]::Real,), body = (args) -> 2 * args.m[:out])

julia> result = @call_message_update_rule(node = Halve, target = :out, m = (in = 3.0,));

julia> getresult(result), getlogscale(result) ≈ log(2)
(1.5, true)

julia> getlogscale(@call_message_update_rule(node = Halve, target = :in, m = (out = 1.0,)))
UndefinedLogScale: the message rule for Halve towards :in under DefaultAlgorithm declares no `logscale`

Improper messages

A message's normalising constant is its integral, and the integral may be infinite. The message is then improper, and no log scale exists to declare. An exact belief-propagation message can be so. Take a normal node with a known mean, out ~ N(0, v), and out observed at $y$. The message towards the variance is the likelihood

\[v \mapsto \mathcal{N}(y \mid 0, v) = (2\pi v)^{-1/2} \exp\!\big(-y^2 / (2v)\big),\]

which decays only like $v^{-1/2}$ as $v$ grows, so its integral over $v > 0$ is infinite. The rule declares logscale = improper. Its message's log scale is an UndefinedLogScale whose reason says so, and it propagates as any undefined one does:

julia> using MessagePassingRulesBase

julia> struct Noise end   # out ~ N(0, v)

julia> @define_factor_node(node = Noise, type = Stochastic, interfaces = [:out, :v])

julia> @define_message_update_rule(
           node = Noise, target = :v, args = (m[:out]::Real,),
           logscale = improper,
           body = (args) -> (v -> -(log(2π * v) + args.m[:out]^2 / v) / 2),   # the log-likelihood
       )

julia> getlogscale(@call_message_update_rule(node = Noise, target = :v, m = (out = 1.0,)))
UndefinedLogScale: the message rule for Noise towards :v under DefaultAlgorithm gives an improper message: it has no normalising constant

Omitting the keyword gives an undefined log scale as well, but its reason is that the rule declares none: a log scale that may exist and that nobody derived. improper says that there is none to derive. A product of an improper message with a proper one can still be normalised, a proper prior on $v$ here, but its log scale is undefined too.

improper

Undefined log scales

An undefined log scale records why it is undefined, and it propagates. Adding it to a number, or to another undefined log scale, gives an undefined log scale with the first reason. Nothing errors because a log scale is undefined. The exception is a rule or a user that asks for its value with require_logscale.

UndefinedLogScale(cause::Symbol, detail = nothing)

julia> u = UndefinedLogScale(:initial) + 1.5
UndefinedLogScale: the message is an initial one, not computed by a rule

julia> isdefined_logscale(u), isdefined_logscale(1.5)
(false, true)

UndefinedLogScaleError(logscale::UndefinedLogScale)

require_logscale(logscale) -> Real

julia> require_logscale(-0.5)
-0.5

julia> require_logscale(UndefinedLogScale(:initial))
ERROR: UndefinedLogScaleError: a log scale is needed but not known: the message is an initial one, not computed by a rule

isdefined_logscale(logscale) -> Bool

getlogscale(x)

Declaring a log scale from the body

A body that computes the result and the constant together returns both with with_logscale. The rule declares logscale = from_body, as Halve does above.

with_logscale(result, logscale)
with_logscale(; result, logscale)

@define_message_update_rule(
    node = Mixture, target = :out, algorithm = MixtureBP,
    args = (m[:switch]::Any, m[:inputs...]::Any),
    logscale = from_body, reads_logscale = true,
    body = (args) -> begin
        weights = ...
        with_logscale(MixtureDistribution(collect(args.m[:inputs]), softmax(weights)), logsumexp(weights))
    end,
)

from_body

Reading the inputs' log scales

A rule that needs the log scales its inputs arrived with declares reads_logscale = true. It reads them as args.logscale.m[:x], keyed like args.m. It calls require_logscale on those whose value it needs.

The rule's caller must provide the log scales, or the call is an error. An engine provides them when it tracks log scales, as ReactiveMP does with its activation option logscales = true. A call by hand takes them as logscale = (x = …,).

The Mixture rules are such rules: the message towards the switch is a softmax over the evidence of each component.

julia> struct Evidence end

julia> @define_factor_node(node = Evidence, type = Deterministic, interfaces = [:out, :in])

julia> @define_message_update_rule(
           node = Evidence, target = :out, args = (m[:in]::Real,),
           reads_logscale = true,
           body = (args) -> require_logscale(args.logscale.m[:in]),
       )

julia> getresult(@call_message_update_rule(node = Evidence, target = :out, m = (in = 1.0,), logscale = (in = -2.5,)))
-2.5

RuleLogScales(; m = NamedTuple())