X-shift invariant prior for logistic regresion

what prior makes the posterior invariant to shifts in the covariates?

Consider the univariate logistic regression

\[\begin{align} \psi = b_0 + b_1 x \\ y \sim \text{Binomial}(1, \sigma(\psi))\\ \begin{bmatrix}b_0 \\ b_1\end{bmatrix} \sim g \end{align}\]

We want \(p(\mathbf{b}| y, X) = p(\mathbf{b}| y, \tilde X)\) where \(\tilde X = X + \mathbf{c}\mathbf{1}^T\) for some \(\mathbf{c}\in \mathbb{R}^p\)

In a meeting with Matthew on 2023-01-10 I mistakenly thought that \(\nabla^2_\mathbf{b}\log p(y | \mathbf{b}, X)\) does not change, but that is clearly not the case.

\[\begin{align} \nabla^2_\mathbf{b}\log p(y | \mathbf{b}, X) &= X^T D X \\ D &= \text{diag}(\sigma(\psi_i)(1 - \sigma(\psi_i))) \\ \psi &= \mathbf{b}^T \mathbf{x}_i \end{align}\]

Instead it’s that \(\left[\nabla^2_\mathbf{b}\log p(y | \mathbf{b}, X)\right]^{-1}_{11}\) for shifts in \(\mathbf{x}\). In the example below we see that the MLE and stderr are unchanged for the effect, all the change is absorbed into the intercept.

Let \(X = [\mathbf{1}, \mathbf{x}]\) and \(\tilde X = [\mathbf{1}, \mathbf{x}+ \mathbf{1}c]\). Note that the linear predictions at the MLE do not change since we can map parameters of the first regression problem to parameters of the second \((b_0, b_1) \mapsto (b_0 + b_1 c, b_1)\). Then \(D = \tilde D\)

Write \(\delta = \mathbf{1}^T D \mathbf{1}\), \(\gamma = \mathbf{x}^T D \mathbf{x}\) and \(a = \mathbf{x}^T D \mathbf{1}\)

Code

x <- rnorm(1000)
y <- rbinom(1000, 1, 1/(1 + exp(-x)))

x2 <- x + 10

fit <- glm(y ~ x, family='binomial')
fit2 <- glm(y ~ x2, family='binomial')

summary(fit)$coef

              Estimate Std. Error    z value     Pr(>|z|)
(Intercept) 0.03216293 0.07176779  0.4481527 6.540430e-01
x           1.19888202 0.09080725 13.2024919 8.488004e-40

Code

summary(fit2)$coef

              Estimate Std. Error   z value     Pr(>|z|)
(Intercept) -11.956657 0.90645402 -13.19058 9.941294e-40
x2            1.198882 0.09080725  13.20249 8.488004e-40

\[\begin{align} (\tilde X ^T D \tilde X)^{-1} &= \begin{bmatrix}\delta & a + \delta \mu \\ a + \delta \mu & \delta \mu + \gamma + 2 a \mu + \mu^2 \delta \end{bmatrix}^{-1} &= \frac{1}{\delta\gamma - a^2} \begin{bmatrix}\delta \mu + \gamma + 2 a \mu + \mu^2 \delta & - a - \delta \mu \\ - a - \delta \mu & \delta \end{bmatrix} \end{align}\]