It seems to me that at this time, written words were basically instructions for reading out loud. I would guess that while Caesar was exceptionally educated for his time and that indeed the men who were so amazed could not read at all let alone silently, Caesar was not the only scholar who could read silently.
And did the ability to read silently reflect a fundamentally different understanding of text vs spoken language? If written text was indeed meant to be spoken, would that not imply that such text tended to be simpler and that only after people understood that text could be read more slowly, and retraced by the reader that more complex texts be produced.
(I would bet my bottom dollar that Tacitus or Pliny (even the Younger) could read without even moving their lips.)