Grammatical categories - tense and remoteness in this case - are expressed morphosyntactically, whereas in your English example the idea of a recent event is expressed by the adverb "yesterday" (i.e. lexically).
On the other hand, one could theoretically argue for a certain tense gram (form) purely based on semantics. In this case, one could talk of the following tense forms:
I went to town yesterday. (hesternal)
I went to town a week ago. (pre-hesternal)
I went to town last year. (remote)
I went to town a hundred years ago. (ancient past)
etc. ad absurdum.
cf. Kibort 2008
In order to be regarded as a (grammaticalised) tense, the expression of location in time has to be integrated into the grammatical system of the language [emphasis mine - Alex B.]. In contrast, a lexicalised expression of the location in time indicates its integration into the lexicon of the language, but does not entail any necessary consequences for the language's grammatical structure. Grammaticalisation, as opposed to lexicalisation, of the location in time, correlates with two parameters: obligatory expression and morphological boundness. The very rough rule is that a tense is grammaticalised if its morphological expression is obligatory even if the information carried by the exponent is redundant.
Kibort, Anna. "Tense." Grammatical Features. 7 January 2008. http://www.grammaticalfeatures.net/features/tense.html.
