We study the commuting graph of $n\times n$ matrices over the field of $p$-adics $\mathbb{Q}_p$, whose vertices are non-scalar $n\times n$ matrices with entries in $\mathbb{Q}_p$ and whose edges connect pairs of matrices that commute under matrix multiplication. We prove that this graph is connected if and only if $n\geq 3$, with $n$ neither prime nor a power of $p$. We also prove that in the case of $p=2$ and $n=2q$ for $q$ a prime with $q\geq 7$, the commuting graph has the maximum possible diameter of $6$; these are the first known such examples independent of the axiom of choice. We also find choices of $p$ and $n$ yielding diameter $4$ and diameter $5$ commuting graphs, and prove general bounds depending on $p$ and $n$.