Built-in Functional Forms

This section describes built-in functional forms that can be used for posterior marginal and/or messages form constraints specification. Read more information about constraints specification syntax in the Constraints Specification section.

Background

In message passing framework, in order to compute a posterior over some latent state $q(x)$, it is necessary to compute a normalized product of two messages $q(x) = \frac{\mu_1(x) \mu_2(x)}{\int \mu_1(x) \mu_2(x) \mathrm{d}x}.$ In some situations, when functional forms of $\mu_1(x)$ and $\mu_2(x)$ are know in advance, it is possible to compute the normalized product efficiently and analytically. It is, however, not always the case, since the messages can have arbitrary functional form and it is not always easy to compute the normalization factor.

Functional forms help to circumvent this. They implement a custom callback on the product of two messages, which cannot be computed analytically. Essentially, a functional form constraint defines a functional F, such that $q(x) = F[\mu_1, \mu_2] \approx \frac{\mu_1(x) \mu_2(x)}{\int \mu_1(x) \mu_2(x) \mathrm{d}x}.$

See also Bethe Free Energy section for more information on variational inference and posterior computation.

UnspecifiedFormConstraint

Unspecified functional form constraint is used by default and uses only analytical update rules for computing posterior marginals. Throws an error if a product of two colliding messages cannot be computed analytically.

@constraints begin
    # This is the default setting for all latent variables
    q(x) :: UnspecifiedFormConstraint()
end

PointMassFormConstraint

The most basic form of posterior marginal approximation is the PointMass function. In a few words PointMass represents the delta function. In the context of functional form constraints PointMass approximation corresponds to the MAP estimate. For a given distribution d - PointMass functional form simply finds the argmax of the logpdf of q(x), thus $q(x) = F[\mu_1, \mu_2] = \delta(x - \arg\min_{x} \mu_1(x) \mu_2(x))$. This is especially useful when exact functional form of q(x) is not available or cannot be parametrized efficiently.

@constraints begin
    q(x) :: PointMassFormConstraint()
end

@constraints begin
    q(x) :: PointMassFormConstraint(starting_point = (args...) -> 1.0)
end

PointMassFormConstraint

# This is an example of the `custom_optimizer` interface
function custom_optimizer(::Type{ Univariate }, ::Type{ Continuous }, constraint::PointMassFormConstraint, distribution)
    # should return argmin of the -logpdf(distribution)
end

# This is an example of the `custom_starting_point` interface
function custom_starting_point(::Type{ Univariate }, ::Type{ Continuous }, constraint::PointMassFormConstraint, distribution)
    # built-in optimizer expects an array, even for a univariate distribution
    return [ 0.0 ] 
end

# This is an example of the `custom_boundaries` interface
function custom_boundaries(::Type{ Univariate }, ::Type{ Continuous }, constraint::PointMassFormConstraint, distribution)
    # returns a tuple of `lower` and `upper` boundaries
    return (-Inf, Inf)
end

default_point_mass_form_constraint_optimizer(::Type{<:VariateType}, ::Type{<:ValueSupport}, constraint::PointMassFormConstraint, distribution)

default_point_mass_form_constraint_starting_point(::Type{<:VariateType}, ::Type{<:ValueSupport}, constraint::PointMassFormConstraint, distribution)

default_point_mass_form_constraint_boundaries(::Type{<:VariateType}, ::Type{<:ValueSupport}, constraint::PointMassFormConstraint, distribution)

SampleListFormConstraint

SampleListFormConstraints approximates the resulting posterior marginal (product of two colliding messages) as a list of weighted samples. Hence, it requires one of the arguments to be a proper distribution (or at least the inference backend should be able to sample from it). This setting is controlled with LeftProposal(), RightProposal() or AutoProposal() objects. It also accepts an optional method object, but the only one available sampling method currently is the BayesBase.BootstrapImportanceSampling.

@constraints begin
    q(x) :: SampleListFormConstraint(1000)
    # or
    q(y) :: SampleListFormConstraint(1000, LeftProposal())
end

SampleListFormConstraint(rng, strategy, method)

FixedMarginalFormConstraint

Fixed marginal form constraint replaces the resulting posterior marginal obtained during the inference procedure with the prespecified one. Worth to note that the inference backend still tries to compute real posterior marginal and may fail during this process. Might be useful for debugging purposes. If nothing is passed then the computed posterior marginal is returned (see also UnspecifiedFormConstraint).

@constraints function block_updates(x_posterior = nothing)
    # `nothing` returns the computed posterior marginal
    q(x) :: FixedMarginalFormConstraint(x_posterior)
end

FixedMarginalFormConstraint

It is also possible to control the constraint manually, e.g:

form_constraint = FixedMarginalFormConstraint(nothing)

constraints_specification = @constraints function manual_block_updates(form_constraint)
    q(x) :: form_constraint
end

# later on ...
form_constraint.fixed_value = Gamma(1.0, 1.0)

Gamma{Float64}(α=1.0, θ=1.0)

CompositeFormConstraint

It is possible to create a composite functional form by stacking operators, e.g:

@constraints begin
    q(x) :: SampleListFormConstraint(1000) :: PointMassFormConstraint()
end

Constraints: 
  q(x) :: (SampleListFormConstraint(Random.TaskLocalRNG(), AutoProposal(), BayesBase.BootstrapImportanceSampling()), PointMassFormConstraint())

Custom functional forms

See the ReactiveMP.jl library documentation for more information about defining novel custom functional forms that are compatible with ReactiveMP inference backend.