Fisher Information Matrix when Changing Variables

Copied from this link

For normal \(X\sim N(\mu,\sigma^2)\), information matrix is

\[\mathcal{I}_1 = \left( \begin{matrix} \frac{1}{\sigma^2} & 0 \\ 0 & \frac{1}{2\sigma^4} \end{matrix} \right) \]

For curved normal \(X\sim N(\mu,\mu^2)\)

\[\mathcal{I}_2=\frac{3}{\mu^2}.$$So, your observation that determinants being equal is not universal, but that is not the whole story. Generally, if $\mathcal{I}_g$ is the information matrix under the reparametrization $$g(\theta)=(g_1(\theta),...,g_k(\theta))',$$ then, it is not difficult to see that the information matrix for the original parameters is $$I(\theta)=G'I_g(g(\theta))G$$ where $G$ is the Jacobian of the transformation $g=g(\theta)$. For Bernoulli example $(\theta_0,\theta_1)=(p,1-p)$ and $g(p)=(p,1-p)$. So, the Jacobian is $(1,-1)'$ and thus $$\mathcal{I}(p) = \left( \begin{matrix} 1& -1 \end{matrix} \right)\left( \begin{matrix} \frac{1}{p} & 0 \\ 0 & \frac{1}{1-p} \end{matrix} \right) \left( \begin{matrix} 1 \\ -1 \end{matrix} \right)=\frac{1}{p(1-p)}\]

For curved normal example,

\[\mathcal{I}_2 = \left( \begin{matrix} 1& 2\mu \end{matrix} \right)\left( \begin{matrix} \frac{1}{\mu^2} & 0 \\ 0 & \frac{1}{2\mu^4} \end{matrix} \right) \left( \begin{matrix} 1 \\ 2\mu \end{matrix} \right)=\frac{3}{\mu^2}. \]

I think now you can easily relate the determinants.

Follow-up after the comment

If I understood you correctly, the FIM is valid as long as you extend the parameters in meaningful way: the likelihood under new parametrization should be a valid density. Hence, I called the Bernoulli example a unfortunate one.

I think the link you provided has a serious flaw in the derivation of the FIM for categorical variables, as we have \(E(x_i^2)=\theta_i(1-\theta_i)\neq \theta_i\) and \(E(x_ix_j)=\theta_i\theta_j\neq 0\). Expectection of the negative Hessian gives \(\mathrm{diag}\{1/\theta_i\}\), but not for the covariance of the score vectors. If you neglect the constraints, the information matrix equality doesn't hold.