In the theory of matrix-valued orthogonal polynomials, there exists a longstanding problem known as the Matrix Bochner Problem: the classification of all $N \times N$ weight matrices $W(x)$ such that the associated orthogonal polynomials are eigenfunctions of a second-order differential operator. In [4], Casper and Yakimov made an important breakthrough in this area, proving that, under certain hypotheses, every solution to this problem can be obtained as a bispectral Darboux transformation of a direct sum of classical scalar weights. In the present paper, we construct three families of weight matrices $W(x)$ of size $N \times N$, associated with Hermite, Laguerre, and Jacobi weights, which can be considered 'singular' solutions to the Matrix Bochner Problem because they cannot be obtained as a Darboux transformation of classical scalar weights.