Mixtures

NormalMixture

A mixture of normals: out is drawn from the component the one-hot switch selects, with means m and precisions p, both groups with one member per component,

\[p(\mathrm{out} \mid \mathrm{switch}, m, p) = \prod_{k=1}^K \mathcal{N}(\mathrm{out} \mid m_k, p_k^{-1})^{\mathrm{switch}_k}.\]

A component is a NormalMeanPrecision, with a Gamma precision, or, when out is a vector, an MvNormalMeanPrecision, with a Wishart precision. The number of components K is the groups' length, not a type parameter. Its interfaces are out, switch and the groups m and p; GaussianMixture is an alias.

A node needs at least two components, as many members in one group as in the other, and a mean-field factorisation; factornode checks all three. The switch's marginal must be a Categorical, a Bernoulli or a one-hot vector PointMass: an integer PointMass switch is an error, which says to one-hot encode it.

Its rules run under its own algorithm, NormalMixtureVMP, and are always variational. Towards out the message is a normal in weighted-mean form, towards switch a Categorical of the components' responsibilities, towards (:m, k) a normal and towards (:p, k) a GammaShapeRate or a Wishart likelihood. It has an average energy, each component's weighted by the switch.

The :out and :switch dependencies list the precisions before the means. An engine subscribes to a target's inputs in the order they are declared, and in variational message passing that order is the update schedule: each component's precision is updated before its mean. Both orders reach the same optimum, but precisions first converges faster, in about 7 iterations against about 13 on a measured mixture model. The order of the members within a group changes nothing, since they do not depend on each other.

Examples

julia> switch = getresult(@call_message_update_rule(
           node = NormalMixture, target = :switch,
           q = (out = PointMass(0.0), m = (PointMass(0.0), PointMass(5.0)), p = (PointMass(1.0), PointMass(1.0))),
       ));

julia> probvec(switch)[1] > 0.99
true

GaussianMixture

An alias of NormalMixture.

NormalMixtureVMP()

NormalMixture's own algorithm. Variational message passing, regardless of the factorisation: every rule takes the marginals of out, switch and the components, never their messages. It is the node's default, so a model need not name it. It stands alone, a direct subtype of AbstractAlgorithm, and inherits nothing from DefaultAlgorithm, which has no mixture rules.

GammaMixture

A mixture of Gamma distributions: out is drawn from the component the one-hot switch selects, a GammaShapeRate with shape a and rate b, both groups with one member per component,

\[p(\mathrm{out} \mid \mathrm{switch}, a, b) = \prod_{k=1}^K \mathrm{Gamma}(\mathrm{out} \mid a_k, b_k)^{\mathrm{switch}_k}.\]

The number of components K is the groups' length. Its interfaces are out, switch and the groups a and b.

A node needs at least two components, as many members in one group as in the other, and a mean-field factorisation; factornode checks all three. The switch's marginal must be a Categorical, a Bernoulli or a one-hot vector PointMass: an integer PointMass switch is an error, which says to one-hot encode it. The rates' marginals must be Gamma distributions.

Its rules run under its own algorithm, GammaMixtureVMP, and are always variational. Towards out the message is a GammaShapeRate, towards switch a Categorical, towards (:a, k) a GammaShapeLikelihood and towards (:b, k) a GammaShapeRate. It has an average energy, each component's weighted by the switch. The :out and :switch dependencies list the rates before the shapes, which is the update schedule; the order within a group changes nothing.

GammaMixtureVMP()

GammaMixture's own algorithm. Variational message passing, regardless of the factorisation: every rule takes the marginals of out, switch and the components, never their messages. It is the node's default, so a model need not name it. It stands alone, a direct subtype of AbstractAlgorithm, and inherits nothing from DefaultAlgorithm, which has no mixture rules.

Mixture

A mixture of arbitrary components: out is the member of the group inputs that the one-hot switch selects,

\[p(\mathrm{out} \mid \mathrm{switch}, \mathrm{inputs}) = \prod_{k=1}^K δ(\mathrm{out} - \mathrm{inputs}_k)^{\mathrm{switch}_k}.\]

Its interfaces are out, switch and the group inputs, one member per component.

Its rules are belief propagation over the incoming messages, whatever the factorisation, under its own algorithm, MixtureBP. Towards out the message is a MixtureDistribution of the inputs' messages, towards switch a Categorical of each component's evidence, and towards (:inputs, k) the message from out, scaled by the probability of k. The switch's message must have a probvec, a Categorical or a Bernoulli; the inputs' messages may be of any type whose product with the message from out has a log scale under the algorithm's product strategy.

The rules read the log scales of their incoming messages (reads_logscale = true), so a graph with a Mixture must track log scales (the engine's activation option logscales = true); an unknown log scale is an error. The node has no average energy: the free energy of a model with one is an error.

MixtureBP(; prod = GenericProd())

Mixture's own algorithm: belief propagation over the incoming messages, regardless of the factorisation. MixtureBP(GenericProd()) is the node's default, so a model need not name it.

Keywords

• prod: the product strategy under which the rule towards switch multiplies the message from out with each input's, for the log scale of their product, which is component k's evidence. Any BayesBase product strategy. Default GenericProd().

It stands alone, a direct subtype of AbstractAlgorithm. Its dependencies and rules are declared on MixtureBP itself, so they apply for every product strategy.