Sounds about right for a v1, depending on how you define difficulty.
I would probably do the opposite: count how many words are not frequent.
And I would not make it a hard cutoff, but rather weight by rarity.
The challenge is dealing with tokens like prices or emails that won't be in your list (OOV).
And there is also the sequence level. Very long sentences of easy words are probably not easy.
For a working solution, you can try ModelFront. It's for translation risk, which does tend to correlate strongly with sentence difficulty, modulo length.