The answers provided are not very helpful.
Common ways that words gain new meanings are through metaphor and metonymy. Words also shift by changing word classes. A common situation is nouns being used as verbs, such as google.
An example of metaphor-induced change (I wrote a paper on this): "over" never used to be usable in talking about time, such as "over the past few days". In the 1800s though, there were poetic uses, such as saying that a major event cast a shadow over the following decade. That is using a common schema of visualizing time as a space. If it is a space then something can cast a shadow over it. If enough people speak in this metaphoric sense about time, using the word "over", then eventually people can get used to in normal grammar, and it will find it's way into more phrases, like "over a period", "over time", and "over the years".
Whenever a new concept arises in a culture, metaphor is typically used to speak about such concepts. For example, a computer mouse is only called such, because it is kind of the shape and size of a mouse and traditionally had a cord attached that that resembles a mouse's tail.
In some cases, the form or function of a thing can change over time, gradually, while the word(s) associated with it follow along. Think about the earliest telephones compared to the modern idea of a phone. I'm sure better examples of this could be provided.
Metonymy is where one thing is commonly associated with another and used in place of it, such as saying that "George flew to Brazil." Did George actually fly, or did he get in a plane which flew to Brazil? It would be considered metonymy, if the temporal sense of "over", had developed from a common association with the spatial sense, having to do with travelling over distances, which of course takes time as well. However I do not believe "over time", developed this way.
I do suspect that one way "over" began being used in a temporal sense has to do with graphs used in economics, stock markets and science. Considering that graphs would typically put time on the X-axis and chart changes in a dependent variable on the Y-axis, the resulting graph could be seen going up or down over time. I don't know if this should be called metaphor or metonymy or something else, but it is an example of the "inside joke" phenomenon mentioned by curiousdanni. Innovative language in one realm, such as sports and games (low blow, rain check, pawn), the military (fubar, deadline, blockbuster), leaks into common usage. Such terms usually have a specific meanings in there origin but broaden or shift greatly when it enters common parlance. The shifts again take place through metaphor, metonymy, or something else.
Another common trend is for words to shift in meaning from concrete and tangible to abstract. If you look at the etymology of any abstract word in English, there is a good chance it has origins is some everyday physical thing. For example, "government" comes from the verb, "govern" which ultimately originates from the Greek word kybernan, which means "to steer or pilot a ship". Metaphor strikes again.
The fact of the matter is, the only way people learn (and thus perpetuate) words is by observing how others use it and making up some unconscious rule in their head about what the word means and how to use it, based on the contexts that they hear it in. But human minds are not like computers. They are very fuzzy and like to generalize things and consider the whole rather than the parts. In the process of learning words it is very easy for one person to store slightly different rules in their head compared to the people they have learned it from. Thus they will stretch or shift the boundaries whenever they use the word. And of course people enjoy stretching meanings deliberately too.