the first point is that the Sommerfeld model showed us that only a small fraction of the electron, namely those within k_BT of E_F can contribute to conduction. These electrons approximately have the same velocity, namely the Fermi velocity
To the 2nd point: note that the drift velocity is the net average velocity of the electron gas. The individual electrons can still move extremely rapidly