Messages implementation

In the message passing framework, one of the most important concepts is the message. Given a factor graph, messages are arbitrary functions that flow along the edges of the graph and hold information about the part of the graph from which they originate.

Message as a distribution

Often, a message can be represented in the form of a probability distribution, as a probability distribution can be matched with its probability density function. The representation of messages as probability distributions is not only for convenience but also for performance reasons. For example, a univariate Gaussian distribution can be parameterized with two numbers, which significantly reduce the amount of information needed to pass along the edges of the graph.

using StatsPlots, Distributions

plot(Normal(0.0, 1.0), label = "Univariate Gaussian distribution", fillalpha = 0.4, fill = 0)

Variational Message Passing

The message passing technique is useful for finding the posterior distribution over certain parameters in a model, originating from exact Bayesian inference, which is also known as Belief Propagation. However, the message passing technique can also be used to find approximate solutions to posteriors - a technique known as Variational inference. The ReactiveMP.jl package implements Variational Message Passing since it is a more general form than exact inference, and also because the exact solution can be framed as an approximate solution subject to no constraints. Here are visual schematics of the differences between messages in Belief propagation and Variational inference.

Belief-Propagation (or Sum-Product) message

Belief propagation message

Variational message

Variational message with structured factorisation q(x, y)q(z) assumption

Message type

All messages are encoded with the type Message.

Message(data, is_clamped, is_initial, addons)

julia> distribution = Gamma(10.0, 2.0)
Distributions.Gamma{Float64}(α=10.0, θ=2.0)

julia> message = Message(distribution, false, true, nothing)
Message(Distributions.Gamma{Float64}(α=10.0, θ=2.0))

julia> mean(message) 
20.0

julia> getdata(message)
Distributions.Gamma{Float64}(α=10.0, θ=2.0)

julia> is_clamped(message)
false

julia> is_initial(message)
true

From an implementation point a view the Message structure does nothing but hold some data object and redirects most of the statistical related functions to that data object. However, this object is used extensively in Julia's multiple dispatch. Our implementation also uses extra is_initial and is_clamped fields to determine if product of two messages results in is_initial or is_clamped posterior marginal. The final field contains the addons. These contain additional information on top of the functional form of the distribution, such as its scaling or computation history.

getdata(message::Message)

is_clamped(message::Message)

is_initial(message::Message)

getaddons(message::Message)

as_message(::AbstractMessage)

using ReactiveMP, BayesBase, ExponentialFamily

distribution = ExponentialFamily.NormalMeanPrecision(0.0, 1.0)
message      = Message(distribution, false, true, nothing)

Message(ExponentialFamily.NormalMeanPrecision{Float64}(μ=0.0, w=1.0))

mean(message), precision(message)

(0.0, 1.0)

logpdf(message, 1.0)

-1.4189385332046727

is_clamped(message), is_initial(message)

(false, true)

Product of messages

In message passing framework, in order to compute a posterior we must compute a normalized product of two messages. For this purpose the ReactiveMP.jl uses the multiply_messages function, which internally uses the prod function defined in BayesBase.jl with various product strategies. We refer an interested reader to the documentation of the BayesBase.jl package for more information.

multiply_messages(prod_strategy, left::Message, right::Message)

messages_prod_fn(strategy, prod_constraint, form_constraint, form_check_strategy)

Deferred messages