Interesting, even NVIDIA is moving away from the transformer architecture similar to my baby hobby project.

NVIDIA is using Mamba architecture with MoE, very similar to the experimental AI model I been doing except that some parts of the architecture is absent from NVIDIA's but seemingly unique from the hobby project.