Many experts seem to be underutilized, possible optimization potential.
#24
by Dampfinchen - opened
Hey there,
I would like to bring your attention to this information Kalomaze on X found out: https://x.com/kalomaze/status/1918238263330148487
It seems some experts are severly underutilized. He used a ton of diverse data to see if experts trigger, some other poster claimed they threw 1.4 million tokens at it and still noticed that some experts are not active at all.