Gaussian-Gaussian conjugate prior (#43)

[affeldt-aist]

Motivation for this change

• add Gaussian-Gaussian conjugate prior

Developed with the assistance of Claude Opus 4.8 and Fable 5 (1M context).

fyi: @gbdrt

Checklist

• added corresponding entries in CHANGELOG_UNRELEASED.md

• added corresponding documentation in the headers

Reference: How to document

Merge policy

As a rule of thumb:

• PRs with several commits that make sense individually and that
  all compile are preferentially merged into master.
• PRs with disorganized commits are very likely to be squash-rebased.

Reminder to reviewers

• Read this Checklist
• Put a milestone if possible
• Check labels

[affeldt-aist]

I made a few final simplifications and I think that the code is ok but
the documentation can be improved. A few points:

What does "complete the square" (two occurrences) mean?

The comment before the lemma normal_pdf_conjugate is talking
about conditional probability but this does not appear
explicitly in the statement.

The comment before the lemma normal_prob_conjugate is talking
about theta but this is not a free variable in the conclusion.

The lemma normal_prob_conjugate is supposed to correspond to the first row of the table
https://en.wikipedia.org/wiki/Conjugate_prior#When_the_likelihood_function_is_a_continuous_distribution

This could be mentioned in the documentation and used to make the documentation more precise by using
similar vocabulary.

@gbdrt ?

[affeldt-aist]

By the way, the documentation now looks like that:

[gbdrt]

Thanks @affeldt-aist!

What does "complete the square" (two occurrences) mean?

It's just the basic manipulation: $a x^2 + b x + c = a (x - mu)^2 + Cst$ (required here to recover the new gaussian pdf). We can probably just drop it in the these comments.

The comment before the lemma normal_pdf_conjugate is talking about conditional probability but this does not appear explicitly in the statement.

Right. We should avoid the Bayesian presentation here. What about something more descriptive?

(**md $\mathcal{N}(t,s)(x)\mathcal{N}(m_0,s_0)(t) =
  \mathcal{N}\left(m_0,\sqrt{s_0^2+s^2}\right)(x)\mathcal{N}(\mu_1,\sigma_1)(t)$
 with $\mu_1$ = [post_mean m0 s0 x s] and $\sigma_1$ = [post_stddev s0 s]. 
*)

The comment before the lemma normal_prob_conjugate is talking about theta but this is not a free variable in the conclusion.

The lemma normal_prob_conjugate is supposed to correspond to the first row of the table https://en.wikipedia.org/wiki/Conjugate_prior#When_the_likelihood_function_is_a_continuous_distribution

This could be mentioned in the documentation and used to make the documentation more precise by using similar vocabulary.

Yes that would be much better. What about?

(**md Gaussian-Gaussian conjugate prior for a single observation $x$ (cf. first
 row of the table of
 [conjugate priors for continuous likelihoods](https://en.wikipedia.org/wiki/Conjugate_prior#When_the_likelihood_function_is_a_continuous_distribution)):
 the prior $\mathcal{N}(m_0,s_0)$ is conjugate to the likelihood
 $\mathcal{N}(\mu,s)$ with parameter $\mu$ (and known variance $s^2$), and the
 posterior hyperparameters are $\mu_1$ = `post_mean m0 s0 x s` and
 $\sigma_1$ = `post_stddev s0 s`; i.e.,
 $V \mapsto \int_{t \in V} \mathcal{N}(t,s)(x)d\mathcal{N}(m_0,s_0)$ normalized
 by its total mass is $\mathcal{N}(\mu_1,\sigma_1)$.
 *)