What I am and am not going to do here

All I’m going to do here is fix the $B$ term in the Fréchet PSRF. I’ll then verify it, and look at where I went wrong previously. I’ll stitch this all back together separately to present the entire Fréchet PSRF, and only then compare this all to the Gelman-Rubin-like diagnostic of Whidden and Matsen (2015).

The key quantity

We have RVs $\Xi$ and $\Upsilon$, and for each we have $n$ samples $\xi_1, \dots, \xi_n$ and $\upsilon_1, \dots, \upsilon_n$. These have sample means $\bar{\xi}$ and $\bar{\upsilon}$. They have (Bessel-corrected) sample variances $s_\xi^2$ and $s_\upsilon^2$. They have (Bessel-corrected) sample covariance $S_{\xi \upsilon}$.

We want to know what this term is equal to.

So, now we do the math. More or less one term at a time. And yes, this time I’m cheating a little because I’ve more or less worked it out already and I want the math to look slightly less horrific.¹ The shape we’re aiming for here is a decomposition into two variance terms and a term about the difference in means (we’d accept a covariance term in the mix, but we’ll be assuming independence so it will go away). Most of the math amounts to either finding the terms that you can push outside a sum, or converting between means (times constants) and sums.

I’m going to try to preserve the larger features of what we’re doing by relegating some of the “everything ends up canceling” type detours to subsections. I’m using gratuitous bracketing to try to keep sum terms clear and to make sure it’s clear the difference between \bar{\xi}^2 and \bar{\xi^2}. (Though we use only the former.)

By Detour 1, the left- and right-most terms here yield,

By Detour 2, under the assumption that $\Xi$ and $\Upsilon$ are independent,

Now we clean up and recognize that we’ve got a quadratic sitting around,

Getting it right, finally

We have now established that,

For the PSRF, we want the squared difference in means, so we rearrange it,

The “usual” $B_{\mathrm{F}}$ generalization is then,

At this point I hesitate to invoke a new name for the right-most term here, but for independent sets of samples I do think we can call it the (sample) average squared distance between the samples.

Recall that the definition of the Fréchet sample variance is

This means we can write out all the terms as sums over parts of matrices as,

$B_{\mathrm{F}}$

To get the $B_{\mathrm{F}}$ term in the Fréchet PSRF, now we move from $\Xi$ and $\Upsilon$ to MCMC replicates ${X}_{k \cdot}$ and ${X}_{\ell \cdot}$.

And this is suitable for plugging directly into what we’d otherwise expect the Fréchet PSRF’s $B_{\mathrm{F}}$ term should look like, without any funky counter-correction terms,

So, what did I do wrong?

The only lesson I’m sure I can extract here is that I should have just done the math for the terms I was actually using.

The first time out, I went, I started with an identity I had written out previously. I then looked at the terms, figured I knew what they meant, and plugged values in. I can’t quite tell if I got mucked up in population vs sample thinking,² or if I didn’t quite get the unbiased estimator I was trying for (by dint of trying to match unbiased variance estimators in the usual PSRF equations). Partly this is because I’m still not quite sure what the mean squared difference terms really are.

My stumbling around in the dark shows the danger of intuition. I was in the right wheelhouse of the solution, because my original term was a bit too small. But I started adding extra terms in and over-corrected, so I ended up having to re-correct the overall $B$ by an odd amount.

All in all, though, most of this isn’t the most importantly wrong. My bad estimator in the first post was close enough to right to get something that would be a recognizable estimator of the PSRF. But it’s safer to operate with firm ground under one’s feet, so it’s good to fix these things.

Mathematical detours

Feel free to treat this like an appendix and ignore it.

Detour 1

Detour 2

Some day I may come back and work out what this is without starting from the assumption of independence. But since we’re trying to work out multi-chain convergence diagnostics, it’s a huge time saver just to plug that in. Since we have independent sample sets, $\xi_i$ is independent of $\upsilon_j$, and we can push the sums and averages around.

Code appendix

As I keep mucking up the math, it proves useful to spot-check that I get it right.

The fix

This checks the key identity. (Small sample sizes are better for seeing small off by 1 or (n - 1)/n type errors).

check_it <- function(x, y) {
    n <- length(x)
    stopifnot(length(y) == n)
    ones <- rep(1, n)

    lhs <- (1 / n^2) *
        sum(sapply(x, function(xi) {
            sapply(y, function(yj) {
                (xi - yj)^2
            })
        }))

    rhs <- ((n - 1) / n * var(x)) +
        ((n - 1) / n * var(y)) +
        (mean(x) - mean(y))^2

    return(lhs - rhs)
}

nrep <- 1000
nsamp <- 10

res <- sapply(seq_len(nrep), function(i) {
    x <- rnorm(nsamp)
    y <- rexp(nsamp)
    check_it(x, y)
})

summary(res)

The whole enchilada

This code checks the math for the entire B term.

set.seed(42)

nrep <- 100
nsamp <- 100

b_classic <- function(x) {
  m <- length(x)
  n <- length(x[[1]])

  stopifnot(class(x) == "list", all(lengths(x) == n))

  chain_means <- sapply(x, mean)
  grand_mean <- mean(chain_means)

  n / (m - 1) * sum((chain_means - grand_mean)^2)
}

b_frechet <- function(x, dist_fun) {
  m <- length(x)
  n <- length(x[[1]])

  stopifnot(
    class(x) == "list",
    all(lengths(x) == n),
    class(dist_fun) == "function"
  )

  d_sq <- as.matrix(dist_fun(do.call(c, x)))^2

  mle_vars <- sapply(1:m, function(k) {
    idx <- ((k - 1) * n + 1):(k * n)
    sum(d_sq[idx, idx]) / (2 * n^2)
  })

  d_sq_bars <- matrix(0, nrow = m, ncol = m)

  for (k in 1:(m - 1)) {
    k_idx <- ((k - 1) * n + 1):(k * n)
    for (l in (k + 1):m) {
      l_idx <- ((l - 1) * n + 1):(l * n)
      d_sq_bars[k, l] <- 1 /
        (n^2) *
        sum(d_sq[k_idx, l_idx]) -
        mle_vars[k] -
        mle_vars[l]
    }
  }

  n / (m * (m - 1)) * sum(d_sq_bars)
}

sim_norms <- function(m, n, means = 1:m, vars = rep(1, m)) {
  lapply(1:m, function(k) {
    rnorm(n, means[k], vars[k])
  })
}

two_chain_b <- sapply(1:nrep, function(idx) {
  x <- sim_norms(2, nsamp)
  c(euclidean = b_classic(x), frechet = b_frechet(x, dist))
})
two_chain_ratios <- two_chain_b["frechet", ] / two_chain_b["euclidean", ]
two_chain_diffs <- two_chain_b["frechet", ] - two_chain_b["euclidean", ]

four_chain_b <- sapply(1:nrep, function(idx) {
  x <- sim_norms(4, nsamp)
  c(euclidean = b_classic(x), frechet = b_frechet(x, dist))
})
four_chain_ratios <- four_chain_b["frechet", ] / four_chain_b["euclidean", ]
four_chain_diffs <- four_chain_b["frechet", ] - four_chain_b["euclidean", ]

eight_chain_b <- sapply(1:nrep, function(idx) {
  x <- sim_norms(8, nsamp)
  c(euclidean = b_classic(x), frechet = b_frechet(x, dist))
})
eight_chain_ratios <- eight_chain_b["frechet", ] / eight_chain_b["euclidean", ]
eight_chain_diffs <- eight_chain_b["frechet", ] - eight_chain_b["euclidean", ]

summary(two_chain_ratios)
summary(four_chain_ratios)
summary(eight_chain_ratios)

summary(two_chain_diffs)
summary(four_chain_diffs)
summary(eight_chain_diffs)

Recap

The Fréchet PSRF is

The pooled Fréchet sample variance is

and the per-chain sample variance is

The biased-from-above $\hat{\sigma}_\mathrm{F}^2$ is

and

where

This is almost entirely as laid out in Part 1 (and references therein) except this last equation, which I got wrong there and spent a lot of time fixing.

Compared to the Gelman-Rubin-like diagnostic

Let’s start again with the easy term,

That is, summing over the entire (squared) distance matrix yields twice the variance.

The harder term is

The sum here is over the (squared) distance matrix of the concatenated chains, so there’s a block structure. The diagonal blocks are clearly variance-only terms, and the off-diagonal blocks are between chain comparison terms (which end up including variance terms too). Last time I tackled this by splitting it up, but having wrangled with these terms more, I’m not sure we want to. But, note that, per Detour 1, our work from Part 3 holds for both these blocks.

So, rewriting this we get,

Now let’s tackle these terms separately. We get the first two via Detour 2

And by Detour 3

Cleaning up, we get,

The factor of 2 in $s_{\text{WM}}^2$ and $\hat{\sigma}_{\text{WM}}^2$ cancel, but I don’t think we need to finish the math because the overall story is the same as we concluded in Part 1. As the sample size $n$ goes to infinity, $B_{\text{WM}}$ goes to 0, so the between-chain component here is asymptotically negligible.

Detours

Treat this like an appendix and ignore it if you like.

Detour 1

Synthesizing some equations from Part 3, we have, for chains $k$ and $\ell$,

Multiplying through by $n^2$, we get

In the case where $k = \ell$, the distance between means is 0, so,

We can also get to this identity by starting from our variance identity,

Detour 2

In parallel,

So in total,

Detour 3

We have that,

But we want to work with all $k$ and $\ell$. We apply symmetry and the fact that for $k = \ell$ the distance between averages is 0, to note that this simply doubles things.

so

and

Code check

set.seed(42)

psrf <- function(x) {
  m <- length(x)
  n <- length(x[[1]])

  stopifnot(class(x) == "list", all(lengths(x) == n))

  chain_means <- sapply(x, mean)
  grand_mean <- mean(chain_means)

  b_n <- sum((chain_means - grand_mean)^2) / (m - 1)

  s_sq <- mean(sapply(x, var))
  hat_sigma_sq <- (1 - 1 / n) * s_sq + b_n

  r_hat <- sqrt((hat_sigma_sq) / s_sq)

  r_hat
}

psrf_frechet <- function(x, dist_fun) {
  m <- length(x)
  n <- length(x[[1]])

  stopifnot(
    class(x) == "list",
    all(lengths(x) == n),
    class(dist_fun) == "function"
  )

  d_sq <- as.matrix(dist_fun(do.call(c, x)))^2

  vars <- sapply(1:m, function(k) {
    idx <- ((k - 1) * n + 1):(k * n)
    sum(d_sq[idx, idx]) / (2 * n * (n - 1))
  })

  d_sq_bars <- matrix(0, nrow = m, ncol = m)

  for (k in 1:(m - 1)) {
    k_idx <- ((k - 1) * n + 1):(k * n)
    for (l in (k + 1):m) {
      l_idx <- ((l - 1) * n + 1):(l * n)
      d_sq_bars[k, l] <- 1 /
        (n^2) *
        sum(d_sq[k_idx, l_idx]) -
        (n - 1) / n * vars[k] -
        (n - 1) / n * vars[l]
    }
  }

  b_n <- sum(d_sq_bars) / (m * (m - 1))

  s_sq <- mean(vars)
  hat_sigma_sq <- (1 - 1 / n) * s_sq + b_n

  r_hat <- sqrt(hat_sigma_sq / s_sq)

  r_hat
}

wm_stat <- function(x, dist_fun) {
  m <- length(x)
  n <- length(x[[1]])

  stopifnot(
    class(x) == "list",
    all(lengths(x) == n),
    class(dist_fun) == "function"
  )

  d_sq <- as.matrix(dist_fun(do.call(c, x)))^2

  vars <- sapply(1:m, function(k) {
    idx <- ((k - 1) * n + 1):(k * n)
    sum(d_sq[idx, idx]) / (n * (n - 1))
  })

  b_n <- 1 / ((m - 1) * m * n^3) * sum(d_sq)

  s_sq <- mean(vars)
  hat_sigma_sq <- (1 - 1 / n) * s_sq + b_n

  r_hat <- sqrt((hat_sigma_sq) / s_sq)

  r_hat
}

sim_norms <- function(m, n, means = 1:m, vars = rep(1, m)) {
  lapply(1:m, function(k) {
    rnorm(n, means[k], vars[k])
  })
}


nrep <- 100
nsamp <- 100

four_chain_psrf <- t(sapply(1:nrep, function(idx) {
  x <- sim_norms(4, nsamp)
  c(
    euclidean = psrf(x),
    frechet = psrf_frechet(x, dist),
    wm_2015 = wm_stat(x, dist)
  )
}))

r <- range(four_chain_psrf)

plot(
  four_chain_psrf[, 1:2],
  xlim = r,
  ylim = r,
  xlab = "Standard PSRF",
  ylab = "Frechet comparisons"
)
abline(a = 0, b = 1, col = "red")
legend(
  "topleft",
  legend = c("Fréchet", "WM2015"),
  fill = c("black", "blue"),
  border = NA,
  bty = "n"
)
points(four_chain_psrf[, c(1, 3)], col = "blue")

Disclaimer

This post shows both the benefits and dangers of intuition. I managed to guess something close enough to the nature of the error I made last time to fix it. But wrong enough to cascade and require other weird re-correcting. I’m leaving it up because math is messy and sometimes we’re all stumbling around in the dark. But I’d suggest skipping to part 3.

What did I test and how wrong was I?

It’s always best to start relatively simple, so I started out by testing $B$. Yes, $s^2$ would have been simpler, but that’s already been through peer review, so we’ll leave it alone.¹ But this is the heart of what we need to be right, and it reduces oh so slightly the number of places an error could creep in compared to jumping straight to $\hat{R}$ or $\hat{\sigma}^2$.

How was I testing $B$? With the code shown below, just implementing the math in my previous post. So, each “chain” is a bunch of IID normals, and chain $k$ has mean $k$. The comparison, as mentioned last time, is to the regular PSRF for real-valued random variables. If the generalization is right, in this case the math should collapse and the answers should be identical.

The good news: my $B_{\mathrm{F}}$ was close to the standard $B$.

The bad news: my $B_{\mathrm{F}}$ was stochastically wrong, and wrongness varied in both $m$ and $n$.

Fixing it: plan of attack

Now, assuming that I implemented the math right in code, that is, that this is actually a math error, where to look? The likely culprit would seem to be that I’ve got a biased estimator of $\mathbb{E}[d(X_{k \cdot}, X_{\ell \cdot})^2]$. This doesn’t exactly explain the $m$-dependency, which could be a separate error. But if it’s, say, off by a Bessel correction that would explain why it seems to converge as $n \to \infty$. It’s also the only bit of math in $B$ defining a random variable that hasn’t previously been vetted.

Fixing it: no plan survives contact with the enemy

Previously, I said that if we have $n$ samples of a real-valued variable $\xi$ and $n$ independent samples of a real-valued variable $\upsilon$, then

Going back to Euclidean land, that’s saying

At this point, I will admit, I took a leap of faith, and assumed that the right answer looked Bessel-corrected. That is, I assumed the right estimator is

The good news is that this did banish the stochastic part of my being wrong, but it didn’t make it go away. The bad news is now I need to try to figure out why this is The Right Thing To Do.

Going back to the math, which is the same equation we are borne back ceaselessly unto, we have, without hats

Staring at this isn’t proving all that illuminating, so I probably should stare at the Bessel correction’s proof instead. But I’m going to save that for another time and see if I can’t flounder my way to figuring out the right answer first, and then prove it later.

What I can say now is that, with this correction, but no other math changed, $B_{\mathrm{F}} / B$:

• Looked like it stopped being stochastic
• Looked like it became constant in $m$
• Produced some suspiciously informative numbers as a function of $n$

In particular, it looks like with the corrected $\hat{\mathbb{E}}[(\xi - \upsilon)^2]$, $B_{\mathrm{F}} / B = n / (n - 1)$.

The right $B_{\mathrm{F}}$, I hope

My numerical experiments suggest that, to within decent numerical tolerances, The Right Generalization of $B$ is

where

Next steps

Armed with something that at least checks out computationally, we need to understand why it’s right mathematically. I can allow myself to hope that the average squared distance bit is just a Bessel correction. But then why did the leading factor of $n$ become a factor of $n - 1$? Well, time (or someone smarter) will tell eventually.

Code appendix

set.seed(42)

nrep <- 100
nsamp <- 100

b_classic <- function(x) {
  m <- length(x)
  n <- length(x[[1]])

  stopifnot(class(x) == "list", all(lengths(x) == n))

  chain_means <- sapply(x, mean)
  grand_mean <- mean(chain_means)

  n / (m - 1) * sum((chain_means - grand_mean)^2)
}

b_frechet <- function(x, dist_fun) {
  m <- length(x)
  n <- length(x[[1]])

  stopifnot(
    class(x) == "list",
    all(lengths(x) == n),
    class(dist_fun) == "function"
  )

  d_sq <- as.matrix(dist_fun(do.call(c, x)))^2

  vars <- sapply(1:m, function(k) {
    idx <- ((k - 1) * n + 1):(k * n)
    sum(d_sq[idx, idx]) / (2 * n * (n - 1))
  })

  d_sq_bars <- matrix(0, nrow = m, ncol = m)

  for (k in 1:(m - 1)) {
    k_idx <- ((k - 1) * n + 1):(k * n)
    for (l in (k + 1):m) {
      l_idx <- ((l - 1) * n + 1):(l * n)
      d_sq_bars[k, l] <- 1 /
        (n * (n - 1)) *
        sum(d_sq[k_idx, l_idx]) -
        vars[k] -
        vars[l]
    }
  }

  (n - 1) / (m * (m - 1)) * sum(d_sq_bars)
}

sim_norms <- function(m, n, means = 1:m, vars = rep(1, m)) {
  lapply(1:m, function(k) {
    rnorm(n, means[k], vars[k])
  })
}

two_chain_b <- sapply(1:nrep, function(idx) {
  x <- sim_norms(2, nsamp)
  c(euclidean = b_classic(x), frechet = b_frechet(x, dist))
})
two_chain_ratios <- two_chain_b["frechet", ] / two_chain_b["euclidean", ]
two_chain_diffs <- two_chain_b["frechet", ] - two_chain_b["euclidean", ]

four_chain_b <- sapply(1:nrep, function(idx) {
  x <- sim_norms(4, nsamp)
  c(euclidean = b_classic(x), frechet = b_frechet(x, dist))
})
four_chain_ratios <- four_chain_b["frechet", ] / four_chain_b["euclidean", ]
four_chain_diffs <- four_chain_b["frechet", ] - four_chain_b["euclidean", ]

eight_chain_b <- sapply(1:nrep, function(idx) {
  x <- sim_norms(8, nsamp)
  c(euclidean = b_classic(x), frechet = b_frechet(x, dist))
})
eight_chain_ratios <- eight_chain_b["frechet", ] / eight_chain_b["euclidean", ]
eight_chain_diffs <- eight_chain_b["frechet", ] - eight_chain_b["euclidean", ]

summary(two_chain_ratios)
summary(four_chain_ratios)
summary(eight_chain_ratios)

summary(two_chain_diffs)
summary(four_chain_diffs)
summary(eight_chain_diffs)

I was, when I started writing this, pretty sure that at some point five or six years ago I wrote out some math that shows this is a valid Fréchet generalization of the PSRF in much the same way one can generalize the effective sample size. But the more I look at all this math, the more I think that I didn’t actually manage to do so. Or, perhaps, I got extraordinarily lucky then and goofed this time around.

Disclaimer

I didn’t quite get the math right here. The big picture is fine, but some of the details are wrong. I’m leaving it up because it’s mostly right, because I don’t want to re-write all the sidebars here, and because it’s honest to show that mistakes happen when doing math (and science).

Check out Part 3 for the fix, or Part 4 for the final correct version and comparison to Whidden and Matsen.

PSRF

Assume you have $m$ MCMC runs each with $n$ samples. The form of the PSRF that you usually see written out² is a ratio of standard deviations

where $s^2$ is the average within-chain sample variance and $\hat{\sigma^2}$ is

which is constructed to be biased upwards. Note that while this looks like it is constructed such that $\hat{\sigma^2} \to s^2$ as $n \to \infty$, the definition of $B$ has a cancelling $n$ in it

So, when all independent MCMC runs are sampling the same density, $B \to 0$ as $n \to \infty$,³ so $\hat{R} \to 1$.

To $B$ or not to $B$

It may prove helpful later⁴ to have some idea what $B$ really is, if we’re going to start trying to generalize it and otherwise muck with math in its general vicinity.

To start, let us adopt a mixture distribution view of the situation. We have $m$ component distributions with equal weight $1/m$ and a categorical variable for which component $k$ a given one of the $mn$ total samples comes from. Let $Z$ be the component-defining categorical variable.

Now we have to be careful of randomness. The sample mean of chain $k$, $\bar{X}_{k \cdot}$, is a random variable, as is the grand mean $\bar{X}$. They are also estimates

and

In this light, $B/n$, give or take a Bessel correction, can be written

Now, for the sake of sanity, let’s drop the hats. Clearly, $B/n$ is a sample-based estimate of something, and if we look at the RHS here, with or without hats, we can understand what.

So, $B/n$ is a term estimating the variance in the mixture component’s means. It feels worth noting, since we’ve gone to the trouble to write this all out, that this is one of two components in the law of total variance

For mixture distributions, this tells us that the total (marginal) variance is bigger than the average variance by a factor of how far apart the means of the components are.

Whidden and Matsen (2015)

Index chains such that $x_{ki}$ is the $i$th sample from chain $k$. Let $ d(x,y)$ be a distance function, such as the SPR distance, as Whidden and Matsen (2015) used. They proposed to quantify convergence with

where $s_{\text{WM}}^2$ is the average of the $m$ per-chain quantities

and where $\hat{\sigma^2}_{\text{WM}}$ is calculated analogously to $\hat{\sigma^2}$ but using

My memory is made the fool

When I started writing this, I thought I was going to unpack Whidden and Matsen’s equation rather directly. After all, I did it before, right? Instead, it turned out to be easier to derive the ingredients of a Fréchet-generalized PSRF from scratch, which we’ll do in this section, and in the next section we’ll compare them.

I like to think of Fréchet generalizations as what you get when you substitute (squared) arbitrary distance functions for (squared) Euclidean distances. For a distribution on real numbers, the mean is the value minimizing the squared Euclidian distance to all other values, weighting by the distibution’s PDF or PMF. The sample mean is the point minimizing the squared Euclidean distance to the sampled values. The variance is the average squared Euclidean distance from the mean to the rest of the distribution, weighted by the PDF or PMF. Give or take a Bessel correction, the sample variance is the average squared Euclidean distance from the sample mean to the sampled values. Remove the qualifier “Euclidean” and you’re in Fréchet land.

Now let’s look at the two key terms in the PSRF and see what that looks like in Fréchet land.

The easy one: $s^2$

The Fréchet sample variance of a single MCMC run is, per Supplementary Equation 15 of a paper of mine on the ESS of phylogenetic trees

The harder one: $B$

The biased-from-above $\sigma^2$ term, or equivalently $B$, is a bit more complex. Several initial attempts here led me to failure, which I won’t bore the universe with.⁵ To get there, we take a somewhat more circuitous path, including some helpfully unnumbered equations⁶ from the supplement to my aforementioned tree ESS paper (see the section entitled “The frechetCorrelationESS”).

One of these equations states that, for (real-valued) random variables $\xi$ and $\upsilon$,

In the tree ESS paper, we used this equation to estimate the covariance, because we could separately estimate the variances, and because we could chant “MCMCCLT” and call the last term 0. But for our purposes here, we have samples from independent MCMC runs, so $\xi$ and $\upsilon$ are $X_{k \cdot}$ and $X_{\ell \cdot}$, and thus their covariance is 0.⁷ We can still get the variances separately, so if we rearrange the equation, make the Fréchet substitution of $d(\xi,\upsilon)^2$ for $(\xi - \upsilon)^2$, and use our chain-based variables, we get

The practical use of this equation is estimating the LHS by using the estimates we get for the terms on the RHS. What estimates, you ask? The variances are as above. The remaining term should be estimable as the average of all $X_{k i}$, $X_{\ell j}$ comparisons.⁸ Putting that together, giving it a convenient shorthand for later, and not putting hats over terms that deserve it because I can’t make \widehat work in markdown LaTeX, we get

Now, right about now it looks like we’ve solved the Wrong Problem. We’ve got a difference between chain means pairwise, but that’s not what $B/n$ is. But if we look at the equations that gave us the Fréchet variance, we can turn this into what we want. For $n$ samples of a variable $X$,

Instead, we consider the $m$ sample means of the chains, and we get

And there we have it, the LHS is $B/n$ and the RHS we just showed how to compute.

Wherefore art thou Fréchet

Now we can interpret Whidden and Matsen (2015) in a Fréchet context.

The $1 / (n (n - 1))$ term in both $s_{\text{F}, k}^2$ and $s_{\text{WM}, k}^2$ is identical, but the sum for the Fréchet variance is restricted to the upper (or lower) triangular portion of the matrix. As this is a distance matrix, the diagonal is $\mathbf{0}$, and the two triangular portions are equal. Thus

To understand $B_{\text{WM}}$, we need to do some rewriting of what we wrote above.

We’ll start by trying to get a birds eye view of where we’re going. We’ll use some dubious notation to revisit and slightly tweak an earlier definition of $d(\bar{X}_{k \cdot}, \bar{X}_{\ell \cdot})^2$, yielding

Plugging this into our compact definition of $B_{\mathrm{F}}$, we get

The variance terms each show up $(m - 1)$ times, one each for each $k \neq \ell$ pair. Condensing those terms and then cleaning up the leading fractions accordingly, we get

At this point, we can perhaps finally see where this is going if we think in terms of the matrix of (squared) distances between all $mn$ samples in all chains. This is a block matrix, including both within-chain ($k = \ell$) terms and between-chain ($k \neq \ell$) terms. $B_{\mathrm{F}}$ is the sum of a term proportional to the sum of the upper (or lower, if you’d rather) diagonal blocks and a term proportional to the sum of the diagonal blocks, but we’re subtracting the diagonal terms. To be properly clear, let’s put this in more explicit sum form. So, now we go backwards, and plug in our actual definitions from when we defined $d(\bar{X}_{k \cdot}, \bar{X}_{\ell \cdot})^2$, yielding

Cleaning up, we get

Now we can make that first term a sum over all off-diagonal blocks, and compensate with a factor of $1/2$ in the leading fraction because distance matrices are symmetric.

We’re almost done. $B_{\mathrm{F}}$ is a sum over the entire matrix, diagonals and all, so let’s add them back into our leading term (and balance the equation as needed).

And now we clean up.

At this point, we may have forgotten that

So, we have

Alternately

TL;DR

This has gotten excessively long for a blog post, so let’s wrap it up. Where are we?

If the math I did here is right, and that is not a foregone conclusion, then Whidden and Matsen’s proposed Gelman-Rubin-like diagnostic:

• Is Gelmen-Rubin-like. We can see that it’s related to a Fréchet generalization of the PSRF.
• Is not a Fréchet generalization of the PSRF. Its $B$ term downweights the between-chain variance and upweights the within-chain variation. Asymptotically in chain length $n$ the within-chain bit goes to zero and $B_{\text{WM}} \to 2/n \times B_{\mathrm{F}}$. So we might expect that it is relatively insensitive to between-chain variation.

What still needs to be done? Obviously I need to check my math. But that’s a lot of math to check, so a test is in order. If I got the math right, and I implement that math right in code, then for a real-valued variable, if I use the Euclidean distance, I ought to be able to recover the non-Fréchet PSRF with what I claim is the Fréchet PSRF.

But, again, this has gone on long enough, so I’ll leave myself in suspense until I get around to doing some implementation and checking. I’ll add a link to a follow-up when I invariably find some of my math errors in this computational testing.

The de-facto model of choice here is a negative binomial branching process, so named because every infection causes a number of secondary infections according to a negative binomial distribution.² In many ways this is nice: it’s a highly flexible model that includes Geometric branching processes as a special case and Poisson branching processes as a limiting case. And the math works out nicely to yield useful quantities in closed form. The probability mass function on final chain sizes exists in closed form, as do the mean and variance of that distribution.

On the other hand, the negative binomial isn’t always exactly the most scrutable distribution. Sometimes we can get around this by thinking about the negative binomial not in its marginal form, but in its representation as a mixture of Poissons. Other times, we still end up stuck with vague notions about relatively higher variance for some parameter values than others. This is, I think, a common enough feature of continuous mixture distributions: (when) the math works out (it works out) nicely, but discrete mixtures are easier to comprehend. So, with that in mind, what about a discrete mixture of Poisson distributions for transmission?

Our model

Let there be a vector of reproduction numbers $\mathbf{R}$, and associated probabilities $\mathbf{p}$, such that an individual has reproduction number $R_i$ with $p_i$. Conditional on $R$, then, an individual infects a $\mathrm{Poisson}(R)$ number of people.

Where we need to go

Following along with Lloyd-Smith et al. (2005) and Blumberg and Lloyd-Smith (2013),³

For chain size $j$, we have

which involves evaluating at $s = 0$ the $(j - 1)$th derivative of

The term $Q(s)$ is the probability generating function of the branching process, which can be represented as

where $f(u)$ is the probability distribution function for individual-level $R$.

Some (hopefully mostly correct) math

Now we need to backtrack through these steps and hope I don’t go and bodge either the Dirac deltas or derivatives too badly. To start, I think we get

This does reassuringly align with the homogenous Poisson case, where there is one $R = R_1$ and one $p_1 = 1$.

Then we can tackle $T_j(s)$

Then, for sanity’s sake letting $x := \sum_i e^{-R_i(1 - s)} p_i R_i$, we can apply the product rule to get,

This is ugly and going to get uglier. WolframAlpha provides the following,

Clearly there’s a pattern here. The denominator in the $k$-th order derivative is $j^{k + 1}$. The term $j^k \log^k(x)$ gets introduced in the $k$-th order, and for order $\ell > k$ becomes scaled by each successive $k$. The sign in front of this term alternate, starting positive, and thus matching the order. Generalizing, we get,⁴

Despite this being ugly, we’re on the home stretch now. We have

Finally, removing $x$ and putting back our mixture,

This is a horrific PMF. In good news, for a fixed mixture parameter set $\mathbf{R}, \mathbf{p}$, we can pre-compute the mixture term. Other than that, we hope we never have to evaluate it for particularly large $j$.

Let us track the frequency of a split across $m$ chains (each containing $n$ samples), as $f_i$. The average across all chains is $\bar{f}$. (These are our per-chain and pooled estimates of the split’s probability.) The SDSF is either,

or $\sqrt{m / (m - 1)}$ times that if you want to use the Bessel correction and the unbiased sample SD.

I’ve written in more depth about split frequencies here¹, and as with many things in phylogenetics, you can find it in the MrBayes manual.

But what do we do with the SDSF?

A phylogenetic posterior distribution has many splits.² If we want to assess topological convergence, we then have many SDSF to compare. Commonly, we average them and compare to the arbitrary thresholds of 0.01 and 0.05. There are many examples of where that’s maybe not great.³ Some people suggest using the maximum, and this is indeed better. Often times rare splits are removed because they tend to have low SDSF and thus drag the average down.

Defining bad, better

For a pair of chains, Fabreti and Hoehna (2021)⁴ proposed an approach based on the sampling distribution of the absolute difference in split frequencies. I have proposed looking at the distribution of the signed difference. Either way, the distribution depends on the sample size, which turns out to be a potentially useful degree of freedom.⁵

Fabreti and Hoehna suggested that we define “bad” in terms of absolute differences larger than the 95th percentile of the sampling distribution. As to the sample size, they fixed it to a threshold ESS, essentially the target ESS that we want our chains to exceed. Thus, as we run our chains longer, all the absolute differences should come under this threshold. They specifically suggested computing the exact sampling distribution, and provided this pre-computed for several choices of sample size as it is somewhat computationally intensive for even moderate sample sizes.

I proposed defining “bad” in terms of a difference in frequencies which falls outside a confidence interval.⁶ I took the sample size to be the effective sample size (of the tree, defining and computing which was the actual point of the paper). If splits were independent⁷ and we could perfectly estimate the relevant ESS⁸, we’d expect this to mean that we’d always have some splits failing the diagnostic.

The sampling distribution of the SDSF

Now, neither of those is the SDSF, so why are we talking about them? Partly to warm up to the idea of sampling distributions involving split frequencies. Partly to motivate why we might want something different. Both the signed and absolute difference in split frequencies are natural summaries for comparing two chains. Neither of them readily generalizes to having three or more chains. The SDSF does, however, and so it may be a useful starting point to think about generalizations we can use beyond pairwise comparisons.

We could instead think about saying that $m$ chains’ split frequencies are overly divergent if the SDSF is too large, measured by the sampling distribution. So, how do we get that?

Exact computation is right out. For two chains and Fabreti and Hoehna’s focal ESS of 625, exact computation requires computing all $625^2$ pairwise probabilities. That’s nearly 400,000, which is a lot, but not as bad as $625^4$ for 4 chains, which is over 150 billion.

We’re left with approximation, so let’s do some math and get to something we know how to approximate.⁹ For the moment, let us forget that we’re working with MCMC samples, and just think about $m$ samples of size $n$. Under the null (that all chains are sampling the same split frequency), the distribution of counts of times we see the split in any chain is identical, and all chains are independent. Substituting $\bar{f}$ for the true mean, the count in chain $i$ is $n f_i$ which has mean $n \bar{f}$. What is the variance? If the samples in each chain are independent (as we typically assume in convergence diagnostics), it is $n \bar{f} (1 - \bar{f})$. Therefore, we can turn to our old friend the central limit theorem to assert that,

is asymptotically Normal(0,1).

Armed with this,

We can see that the term in the summand is the square of a variable which is asymptotically Normal. Therefore the sum is asymptotically $\chi^2$ with $m - 1$ degrees of freedom.¹⁰ We lose a degree of freedom because the average, along with any $m - 1$ of the squared deviations, determines the remaining one.

However, we’re not quite done yet, because we actually have the square root of a scaled $\chi^2$ variable. The scaling is easy enough. A $\chi^2(m)$ distribution is a Gamma(shape = $m/2$, scale = $2$) distribution, and we can scale those by, well, scaling the scale parameter. So the term inside the square root is a Gamma($m / 2$, $2 (\bar{f} (1 - \bar{f})) / mn$). The square root of a Gamma-distributed variable turns out to be known as a Nakagami distribution.

Thus, for $m$ sets of $n$ independent samples, the distribution of the SDSF is asymptotically Nakagami with shape parameter $(m - 1) / 2$, and spread parameter¹¹ $(\bar{f} (1 - \bar{f})) / n$. Or, if you want the Bessel version, shape parameter $(m - 1) / 2$ and spread parameter $(\bar{f} (1 - \bar{f}) (m - 1)) / mn$.

Reckoning with MCMC

The actual amount that $f_i$ varies from $\bar{f}$ depends not on the number of samples we take, but the effective sample size of (that split in) chain $i$. If we simply wish to generalize the approach of Fabreti and Hoehna to more than two chains, this isn’t a deal breaker, we simply put the target ESS (e.g. 625) in for $n$. Then, as our actual ESS gets large enough, all of the SDSF will eventually decrease below the 95th percentile of their specified Nakagami distributions.

On the other hand, if we want to use this to generalize confidence interval-based detection of abnormally deviant splits, we have to reckon with the fact that each chain has its own effective sample size. That is, there is not really one $n$ but there are actually $m$, one $n_i$ per chain.

Mathematically, we can do this just fine. The derivation is very similar, as the logic is otherwise identical.

This is the square root of the sum of $m$ independent scaled $\chi^2(1)$ distributions. A $\chi^2(1)$ distribution is a Gamma($1/2$, $2$) distribution, and while each scaling is different, each term in the sum is Gamma($1/2$, $2 \bar{f} (1 - \bar{f}) / mn_i$). Unfortunately, while Gammas with the same scale parameter can be summed, Gamma distributions with the same shape parameter but different scales cannot.

Are we out of luck? Sort of. Still, we’ve come this far, and there are at least a handful of options which present themselves.

Turtles all the way down

We are already approximating things, so what’s one more, we might ask? The potentially least-approximate approximation is brute force simulation. But as we would have to, for some large-ish number of replicates, simulate $m$ Gammas for each split, this quickly becomes intractable. So, we could do something more approximate,¹² but it if we don’t find an astonishingly good one it seems unlikely to produce a particularly performant result.¹³

What would be more approximate? We could use $\min(n_i)$ for all $n_i$ and proceed with the equal sample sizes case to aim for a lenient test. We could use $\max(n_i)$ instead for a strict test. If the ESS doesn’t vary too much across chains, then we might not do terribly by using the mean instead.¹⁴ Lastly, we could try to match variances. That is, each term inside the sum has variance $2 \left(\bar{f} (1 - \bar{f}) / mn_i \right)^2$, and, since chains are independent,¹⁵ the variance of the sum is the sum of the variances. So, if we assume the sum looks Gamma-ish, with shape parameter $(m - 1)/2$, the scale parameter is $\sqrt{2 \sigma^2 / (m - 1)}$.

I don’t get the sense that any of these are particularly good options unless the ESS are pretty consistent across the chains.

Reframe the problem

We could also reframe the problem. We’ve been asking about the sampling distribution of the SDSF. But what if we asked about the sampling distribution of a related but rescaled and thus slightly different quantity?

If we rescale the squared deviations of split frequencies, we can remove the sample size dependency. Thus, as in the equal sample sizes case, we get a tractable Nakagami distribution. One might note that, in essence, we’ve cheated, and changed the summary statistic to counts of splits. The important thing to note is that these aren’t the actual counts of times we’ve seen any given split in any given chain, though. Since the $n_i$ are the per-chain ESS, these are hypothetical counts.

One potential advantage here is that, since all splits will end up with the same (asymptotic) distribution of the rSDSF, we can look at the overall performance of the chain in a single summary statistic.

Fin

How well any of this works, if indeed it works at all, I leave for future work. I should note, following along from the final cheat to working with split counts, instead of frequencies, that there’s a somewhat conceptually simpler option presenting itself here. If we correct for the fact that each chain has a different effective sample size, we can simply use a chi-squared test on the counts of samples with (and without) the split in each chain. We could also use a chi-squared test in the Fabretti Hoehna style, fixing the target ESS and letting the observed test statistics decrease below it as we add more samples.

I like a good forward-time generative model, and the other day I stumbled across some pieces that make me think it could actually be pretty easy to get tractable³ birth-death phylodynamic models with multi-births. But to get there, we need some background.

The short-ish version

Deriving the likelihood function for a phylogenetic birth-death model more or less amounts to figuring out how two quantities change through time. I’ve called them $D(t)$ and $E(t)$ before, but others call them $q(t)$ and $p_0(t)$. I’ll stick with D and E, and note that the lack of subscripts implies that we’re restricting ourselves to models where all lineages alive at some time $t$ are exchangeable. So, time-dependent birth-death models yes, clade-dependent models no.

I like to think of $D(t)$ as the probability of straight lines in the observed phylogeny. Slightly more rigorously, it’s the probability of a single lineage surviving from time $t$ (measured in time before the present) to 0 (the present).⁴ For a lineage surviving from some time $t_o$ to some more recent $t_y$, the probability is $D(t_o) / D(t_y)$. Thus, the birth-death likelihood of a tree can be expressed as a product of $D(t_o) / D(t_y)$ terms, plus terms for the observed births and samples taken. In turn, $D(t)$ depends on $E(t)$, the probability that a lineage alive at time $t$ leaves no sampled descendants in our tree.

To actually get anywhere with a birth-death model derivation, we start by thinking about $\mathrm{d}E/\mathrm{d}t$ and how to integrate it. Then we think about $\mathrm{d}D/\mathrm{d}t$ and how to integrate it. Then we sprinkle in the probabilities of births and other events we can see, namely births and sampling events. We can ignore sampling, which can get complicated, and we’ll get to the births later.

What are we talking about changing?

Before we look at how each of these changes if we allow for multiple births, we need to be clear what we mean. In a typical birth-death model, there’s a birth rate $\lambda(t)$ at which one lineage gives birth to another. I’m suggesting our model instead says that there’s a rate $\lambda(t)$ at which one lineage gives birth to some number of additional lineages. That is, to simulate forward, when your process says that the next event is a birth in lineage $\ell$, you draw the number of children from some distribution, $\nu \sim \mathrm{Pr(\nu)}$.

$E(t)$

I prefer to think about $E(t + \Delta t)$ to $\mathrm{d}E/\mathrm{d}t$. That is, in a sufficiently small window of time, how does the extinction probability change? Then you can rearrange and divide through and get the differential needed.

Now I’m going to add novel notation in the possibly misguided hopes that it will help abstract away from model details. Let’s call $\mathcal{E}$ be the event that the lineage goes extinct, $\mathcal{N}$ the event that nothing happens to it, $\mathcal{B}$⁵ the event that it experiences a birth event. Then let $\mathrm{Pr}(\mathcal{E}| t, \Delta t)$ the probability that it goes extinct between $t$ and $t + \Delta t$.

Extinction probabilities are propagated according to equations of the form

These correspond to the three things that can happen in the interval: absolutely nothing, it goes extinct, or there’s a birth. If nothing happens, then it still has to go extinct eventually, hence the $E(t)$ in that term. If it goes extinct, end of story, extinction achieved. And if there’s a birth, both lineages have to then go extinct, hence the $E(t)^2$.

If we want to allow multi-births, only this last term changes, because now there are many possibilities: a birth event with 1 child, 2 children, and so on. All of these will eventually have to go extinct, so the equation becomes

So, if we want this to work and be tractable, we need a $\mathrm{Pr}(\nu)$ that plays nicely with this functional form and produces an analytical solution. Because if it doesn’t (even approximately), we can’t get at one of the most important quantities in our likelihood.

Also, let’s be careful, because I have now done the math assuming that at a birth event one lineage survives, and some number $\nu$ of additional lineages arise. We could assume that the original one dies and then $1 + \nu$ lineages are born, but this formulation should play better with PMFs of most distributions, so if we want to think that way, we should do the math this way, and then reinterpret later.

A distribution is just a sequence with partial sums we know⁶

What would play nicely with a sum of this form? The geometric distribution seems a likely candidate, with parameter $\theta \in [0,1]$ and random variable $\nu$ the PMF is

Where the Geometric goes, typically goes the Negative Binomial, of which it is a special case. Adding parameter $r$, and shopping around for the right parameterization we get,

There’s also the Poisson distribution (which allows any $\theta > 0$),

A final candidate is the Logarithmic distribution, for $\theta \in (0,1)$

It was the Logarithmic I was staring at when this whole idea really took shape. It shows up in compound Poisson contexts, which is a reason to be hopeful it might work here. That it naturally lives on $\nu \geq 1$ instead of $\nu \geq 0$ may or may not have much utility, though it would be nice to not have to work with conditional PMFs. The Geometric can, depending on how you define write it, live on either space; I’ve written it above for $\nu \geq 1$. The Negative Binomial rears its head in a lot of places, though often as a result of mucking with probability distributions. In this case, the Geometric looks more plausible, since it’s already off by a $1 - \theta$ from the form we want, and the Negative Binomial just keeps going that way.

$D(t)$

Propagating $D(t)$ looks similar. In the typical case,

This corresponds to two possibilities. Either nothing happens in the interval, so the straight line we see in the tree was what really happened in that time, or there’s a birth where all the other lineages go extinct, and the straight line we see is simply all that’s left. I think we can interpret the 2 as ${2 \choose 1}$, that is, pick one of the two lineages to survive, as we can’t distinguish them.

The generalization would then be,

Does this tell us anything more about distributions that might work for $\mathrm{Pr}(\nu)$? Maybe. The ${\nu \choose 1}$ term looks a bit awkward being added to the Geometric, and turns the $1 / \nu!$ in the Poisson into a $1 / (\nu - 1)!$. Both of those tilt the sequences in ways that seem less than promising to make the math work out. On the other hand, it simply cancels the Logarithmic’s $1 / \nu$ in the denominator, leaving a cascading sum of $(E(t) \theta)^\nu$ terms, which ought to sum up nicely.⁷

$\lambda(t)$

I was about to start doing the math,⁷ when I realized I had missed something. In a typical birth-death model, the contribution of the births observed at times $\mathbf{t}$ to the likelihood is very straightforwardly the product of the birth rate at all times of observed births

But we have undone the stipulations that make it this straightforward. Now it’s not that every birth leads to two lineages, both of which we see. Now every birth even leads to some number of children, some (usually) lesser number of which we see. That last bit smuggles in the extinction probability. Say we see a trifurcation in the tree: were there two offspring lineages, and we’re seeing all, or might there have been eight?⁸

Let there be $\nu$ lineages descending from the birth event in the tree, and $\nu_\mathcal{E}$ be the number of unobserved extinct lineages, then the likelihood ought to be

Attempting mathematics

With all that said and done, and having written far more than I expected to by this point, I am now going to try actually making this work. I don’t think I’m going to actually finish the math, but I’m going to squint at things and see if I think there’s hope for finishing it up later. I think the thing to do is start with the sum that looks the hardest, and work our way towards easier ones. Without showing my work, I’ll say I tried the Logarithmic and gave up. The $1 / \nu$ in the PMF reacted poorly with the combinatorial coefficient in the probabilities of birth events, leaving a stray random variable lying about.

$\lambda(t)$ revisited

Replacing $E(t)$ with $E$ to reduce clutter, we want the following to work out to something nice

Define $p := E (1 - \theta)$

Note that

Ergo

We can now recognize the summand as the PMF of a $\mathrm{NegativeBinomial}(\nu_\mathcal{E}; \nu + 1, p)$ random variable, which means it evaluates to 1. Before we even think about cleaning it up, we need to see that the other terms work out.

$D(t)$ revisited

We need this sum to work out too

Define $(1 - p) := E (1 - \theta)$

We can expand the summand to include $\nu = 0$ where it is 0,

And now we can recognize the sum as the expectation of a $\mathrm{Geometric}(\nu; p)$-distributed random variable,⁹ but the 0-including kind of Geometric, so we can simplify to

$E(t)$ revisited

The last, though perhaps most important, thing that we need to work out nicely is

Using $p := E (1 - \theta)$ again

This is everyone’s favorite geometric sequence, with a sum of $p / (1 - p)$, yielding

So where are we?

I haven’t checked my math sufficiently to be sure I’m right, but I think even with some inevitable slips, there will be an analytical form for all three of the terms we’ve looked at. Which means that there are analytical solutions to $D(t + \Delta t)$ and $E(t + \Delta t)$, and a derivation of an analytically tractable multi-birth-death model is possible. The details of actually making the math work out will depend on the model itself, and I’d be shocked if something didn’t get ugly.¹⁰ But it looks possible to me.

This model could be useful in phylodynamics when sampling is sufficiently good to actually see superspreading in a tree, or perhaps when sampling isn’t quite as good, but there’s still more heterogeneity in offspring distribution than a usual time-varying birth-death model.¹¹ Or it could be useless. But it might be fun to find out.

The good

The good news, such as it is, is that nonparametric bootstrapping of divergence times does seem to work. I compared it to the parametric bootstrap, which is a similarly widely-scoped hammer for the problem. As you can see, they both achieve basically the correct coverage in a simple constant-rate birth-death model scenario.

In general, the nonparametric bootstrap is probably a bit faster than the parametric bootstrap. Univariate likelihood profiling could perhaps be faster still, but, well, it’s univariate³. As tested, I used TESS, which is very fast for simulating birth-death processes, for the parametric bootstrap, and R’s built-in sample() for the nonparametric bootstrap. TESS numerically integrates the CDF of divergence times and then draws from that CDF. This is, minimally, one more step than simply drawing the indices of the samples for the nonparametric bootstrap. The number of random numbers needed in both case is the same (other simulation-based approaches would need more). I can’t entirely explain why, in practice, with 250 bootstrap replicates the nonparametric bootstrap was nearly two times faster than the parametric. It could just be how much effort has gone into R’s sample() over the years, or that drawing from a numerically-approximated CDF inherently requires more comparisons than drawing uniform random integers.

The bad

Nonparametric bootstrapping of divergence times, as an approach, is actually rather limited in application scope. That might sound weird, given how few assumptions it would seem to rely on. But the assumptions it does rest on are not small.

The explicit assumption is that branching times are IID. I won’t spend too much ink on this, as it is the other, implicit, assumption that is the real death blow. I do think we can push past models with tips only at the present⁴, and serially-sampled birth death models are important. I do not have a strong intuition on whether times still be IID if we think there are, say, arbitrary waiting time distributions for births and deaths. Quite possibly?

The implicit assumption here is that branching times are all we care about. After all, that’s all we’re bootstrapping. But when birth and/or death rates change along the tree, it’s the tree that matters, not just the divergence times. That means trait-dependent models are out, or even just models that let the birth and death rates vary from branch to branch. Such SSE family models comprise a huge portion of the usefulness of phylogenetic birth-death models⁵ in practice.

It’s also worth noting that the nonparametric bootstrap is still an asymptotic approach. So it’s not liable to be particularly helpful if you only have a tree with ten tips.

The ugly

Nonparametric bootstrapping of divergence times is completely unnecessary in practice. Roughly, I’d say, the nonparametric bootstrap is useful when it allows us to do something we could not otherwise do satisfactorily. Many times this is about our ability to remove assumptions from the model. Other times this is simply that we have no other way to address uncertainty about our estimates.

I can’t see ways that this allows us to remove assumptions from our modeling. In other statistical arenas, for example, it allows us to quantify uncertainty about sample means, or to compare multiple means, without assuming a distribution. But in phylogenetic contexts, while we may not adore our branching models, we’re far more interested in their parameters (we care a lot about things that increase the branching rate) than their sample statistics. And we can’t even address all sample statistics with this approach: we could talk about the mean branching time (which I cannot, for the life of me, fathom a utility for) but not the number of branching events (which, under the right circumstances, we might just care about⁶).

And for the class of models to which bootstrapping divergence times is applicable, I don’t think we have any trouble getting at uncertainty. Lots of our models have tractable likelihoods, so we can always use Bayesian inference to get at (multivariate) parameter uncertainty. And I’d argue that most of our models are generative, so even if inference isn’t straightforward, we can simulate from them and use a parametric bootstrap.

TL;DR

Nonparametric bootstrapping of branching times is doable, under strong assumptions, but not particularly useful, unless you happen to have a purely time-dependent model which is not generative and which lacks a tractable likelihood.

Code appendix

Herein lies the code required to produce the figure above. Caveat emptor.

library(TESS)

set.seed(42)

lambda <- 1.0
mu <- 0.6
nsim <- 250
nboot <- 100
widths <- seq(0.05, 0.95, 0.05)

fitCR <- function(bt) {
  fn <- function(par) {
    -tess.likelihood(bt, exp(par[1]), exp(par[2]))
  }
  r <- log((length(bt) + 1) / 2) / max(bt)
  opts <- optim(log(c(r + 0.1, 0.1)), fn)
  opts$par <- exp(opts$par)
  return(opts)
}

npBootCR <- function(bt, nboot, widths) {
  rates <- sapply(seq.int(nboot), function(idx) {
    fit <- fitCR(sample(bt, replace = TRUE))
    return(c(b = fit$par[1], d = fit$par[2]))
  })
  half_alphas <- (1 - widths) / 2
  apply(rates, 1, function(samps) {
    sapply(half_alphas, function(alpha_2) {
      quantile(samps, c(alpha_2, 1 - alpha_2))
    }, simplify = FALSE)
  })
}

parBootCR <- function(bt, nboot, widths) {
  opts <- fitCR(bt)
  point <- opts$par

  sims <- tess.sim.taxa(n = nboot, nTaxa = length(bt) + 1, lambda = point[1], mu = point[2], max = 100)
  rates <- sapply(seq.int(nboot), function(idx) {
    fit <- fitCR(ape::branching.times(sims[[idx]]))
    return(c(b = fit$par[1], d = fit$par[2]))
  })
  half_alphas <- (1 - widths) / 2
  apply(rates, 1, function(samps) {
    sapply(half_alphas, function(alpha_2) {
      quantile(samps, c(alpha_2, 1 - alpha_2))
    }, simplify = FALSE)
  })
}

extractCoverage <- function(reps, widths, true_par = c(b = lambda, d = mu)) {
  res <- lapply(c("b", "d"), function(par) {
    covered <- sapply(reps, function(sim) {
      sapply(sim[[par]], function(low_high) {
        return(as.integer(low_high[1] < true_par[par] && low_high[2] > true_par[par]))
      })
    })
    dimnames(covered) <- list(widths, NULL)
    return(covered)
  })
  names(res) <- c("b", "d")
  return(res)
}

trees <- tess.sim.age(n = nsim, age = 10, lambda = lambda, mu = mu)
times <- sapply(trees, ape::branching.times)

system.time({
  np_cis <- lapply(times, npBootCR, nboot = nboot, widths = widths)
})

system.time({
  par_cis <- lapply(times, parBootCR, nboot = nboot, widths = widths)
})

np_coverage <- extractCoverage(np_cis, widths = widths)
np_birth_coverage <- cbind(widths, rowMeans(np_coverage$b))
np_death_coverage <- cbind(widths, rowMeans(np_coverage$d))


par_coverage <- extractCoverage(par_cis, widths = widths)
par_birth_coverage <- cbind(widths, rowMeans(par_coverage$b))
par_death_coverage <- cbind(widths, rowMeans(par_coverage$d))

plot(np_birth_coverage, col = "#ffa035", pch = 1, xlab = "Nominal quantile", ylab = "Coverage")
points(np_death_coverage, col = "#00a7b0", pch = 1)
points(par_birth_coverage, col = "#ffa035", pch = 4)
points(par_death_coverage, col = "#00a7b0", pch = 4)
abline(a = 0, b = 1, col = "#66666690")
legend(
  "topleft",
  legend = c("Birth rate", "Death rate", "Nonparametric bootstrap", "Parametric bootstrap"),
  col = c("#ffa035", "#00a7b0", "black", "black"),
  pch = c(16, 16, 1, 4),
  border = NA,
  bty = "n"
)

The scientific question

A question that comes up in phylogenetic contexts a lot is “do these two loci share a phylogeny?” In particular, I’m talking about cases where we’re interested in underlying differences in topology, and not simply branch lengths³. The last time this question came up for me, I was trying to identify recombination breakpoints in some viral genomes. But this question can also come up if we’re feeling lazy and don’t really want to use species tree methods and hope we can just concatenate things. Or even if we’ve given up and know we need the multispecies coalescent but want to reduce the number of gene trees involved.

From a statistical perspective, we can frame this as a hypothesis test, or as model selection. For simplicity’s sake, let’s consider only two loci, so our models are that there is one tree, or that there are two trees. If we want to do hypothesis testing, we need some (asymptotic) distribution of a test statistic, like in a likelihood ratio test (LRT), where (twice) the difference in maximized log-likelihoods between the models follows (approximately, asymptotically) a chi-squared distribution. (Or we need to undertake am expensive brute-force simulation-based approach⁴.) If we want to do model selection, and we don’t want to wait on marginal likelihoods⁵, we might like to do something like AIC or BIC, to penalize the two-tree model for complexity. Either way, we need to know how much more complex the two-tree model is. That is, how many more parameters does it have? The easy part: it has twice as many branches. The hard part: it has twice as many trees.

My not-so-clever methodology

In brief, my notion to address this question was to swap around input and output in a LRT. Roughly speaking, a LRT takes in as input the degrees of freedom (the difference in the number of free parameters between the more complex and the nested simpler model), an observed difference in log-likelihoods between two models, and outputs a probabilistic judgement on whether the more complex model is justifiably better. It does this via the asymptotic approximation that the distribution of the LRT statistic is chi-squared. What I had in mind was to feed in many observed differences in log-likelihood, a known judgement on whether the complex model is justifiably better (it would not be), and output the degrees of freedom.

What is Normal, anyways?

If that doesn’t make sense⁶, let’s try an example. And for the moment, let’s stick with Normal distributions⁷. Imagine that I tell you I will simulate independently $m$ times from the following experiment. I draw $kn$ observations, where $n$ is large, of a Normal($\mu$, $\sigma^2$) distribution. Then I fit two models, and give you their likelihoods. The simpler (null) model is that all $kn$ draws come from the same Normal, whose mean and variance I estimate. The more complex model is, for some $k$, that there are $k$ sets of $n$ draws, each from their own Normal distribution whose mean and variance I estimate.

You might choose to record, per experiment, twice the difference in log-likelihoods of the two models. Then, the $m$ values are independent draws from a distribution which is asymptotically Chi-squared, with $2(k - 1)$ degrees of freedom (where the 2 is because you know that each component comes with a mean and a variance to estimate). This means that, in theory, you can fit $k$ to the $m$ LRT statistics and tell me what it is.

Now, the question, does it work? The answer would seem to be yes. When I simulate 1000 times with $k = 6$, the best-fitting Chi-squared distribution has 10 degrees of freedom, which is the right answer.

The results for trees

My plan for trees was pretty similar. For each experiment, at a given tree length and number of tips, I would simulate one true tree. I would then simulate an alignment with $2n$ sites in it, and fit two models with maximum likelihood, a one-tree model and a two-tree model. To keep things simple, I’d use Jukes-Cantor so that the difference in free parameter count between the models would be the tree topology and the number of branches.

I decided to do this for 4 through 10 tip trees, because IQ-TREE can infer trees that size nearly instantaneously so 200 replicates at each tree size for a few tree lengths would be feasible before I tried to scale up.

However, when I summarized these simulations, to my surprise I found that the degrees of freedom was simply the number of branches in the tree. This meant that I was inferring that the number of free parameters a tree was worth was… 0. Because all the free parameters were going into the branch lengths on the second tree.

What went wrong?

So, why did this go off the rails?

The LRT is an asymptotic result, and as such I chose to simulate large alignments of 100,000 sites. For the shortest simulated tree length (0.25) for the largest tree (10 tips, 17 branches) the average branch has 1470.6 substitutions on it. That’s probably large enough for the asymptotics, but unfortunately it’s also a guarantee that we get the right tree topology every time⁸. In other words, the effect of the tree topology isn’t showing up in our simulations because, in the region we’re simulating, there is no effect of the tree topology.

Really, then, the problem is that I tried to use an asymptotic tool to answer a non-asymptotic question. The effect of phylogenetic uncertainty lives in a world where we don’t have an effectively infinite amount of data about the topology. Given how skeptical I often am about asymptotic results, I feel like I should have seen this coming in advance. Alas.

Many have previously proposed to measure resiliency of projects or organizations via the “Bus Factor.” Bus Factors take values in the positive integers $\mathbb{Z}^{+}$.

The Standard Bus Factor (SBF) is, loosely, the number of people who must be removed from their roles befall before a crucial system cannot be maintained (or similar, such as a project needing to be suspended). The model of resiliency underlying the SBF is of non-exclusive “or.” That is, the system will work if individual 1 is present or individual 2 is present or… and so forth. Thus, resiliency increases with increasing SBF.

A less typical Alternative Bus Factor (ABF) definition runs in the opposite direction, indexing the number of individuals of which the removal of any one is fatal. The model of resiliency underlying the ABF is “and.” In this model, a system only works if individual 1 is present and individual 2 is present and… and so forth. In this case, increasingly large values indicate decreasing resiliency, with more crucial links which can be taken out.

New approaches

It is not necessary, either mathematically or conceptually, to retain both Bus Factors. Under the ABF model, failure occurs if one out of any number of other individuals is “bussed.” This suggests that we might instead measure and-based Bus Factors in terms of fractions. Thus, $\frac{1}{3}$ would indicate that if one out of any three key people were bussed, failure would occur, and as the fraction decreases, so does stability.

Thus we propose replacing the SBF and ABF with the Unified Bus Factor (UBF), which takes values in $\frac{1}{\mathbb{Z}^{+}} \cup \mathbb{Z}^{+}$. The UBF increases with increasing stability, and encompasses both the SBF and ABF. A UBF of 2 indicates that either of two people are sufficient for resiliency, while a value of 1/2 indicates that both are required. As the SBF is undefined for values less than 1, we further suggest that the UBF can in general be safely used implicitly, requiring only (ill-advised) usage of the ABF to be qualified.

Discussion and Conclusions

The SBF and ABF both presuppose at most one type of individual, the “or” type and the “and” type. This is not necessarily the case for all systems of interest. For example, a baseball team requires individuals capable of filling a number of different roles: catcher, pitcher, first base, and so forth. Without one for every role, the team cannot play, but a team has more than one pitcher available, and thus can play even without one. This “and”-based unification of multiple “or” types, we leave for future work.

Despite its limitations, the SBF has proven a useful tool for communicating about system resiliency. We therefore hope that the UBF will provide a helpful extension for enumerating least-stable systems in clear and compact forms.