This is a bit more complicated than just the branch. Both semantics and pragmatics will deal with aspects of this issue but you also need to pick the right school and the right people to read. People who have done most work in this area are people working in cognitive linguistics and certain branches of functional linguistics.
The ultimate book that will deal with all of these topics in the way you mention is Towards Cognitive Semantics by Leonard Talmy.
It is quite a tome, though. Perhaps a gentler introduction to this kind of thinking about language is Lakoff and Johnson's Metaphors we Live By even though it does not deal with all elements of what you're interested in, it outlines the agenda for that kind of thinking about language and meaning. (Read the 2003 edition - it has an important new chapter in it).
From the functional perspective, perhaps Talmy Givon's work on tense, aspect and modality is the most relevant. I think it is complementary to the cognitive work but it is a very different kind of approach.