Classical Latin, as I understand things, barely has a definite article at all: ille is the nearest equivalent, and even this word is closer to English that than the. But Spanish, French and Italian are chock full of el/le/il/etc. What on earth could cause a language, even over the course of centuries, to undergo such a drastic structural change?
Languages evolve in many ways! Proto-Indo-European had no articles at all, but they evolved independently in several different branches: you can still see the similarity between English "the" and "that", which is almost exactly the same as how ille turned into el/il/etc.
It looks a bit more likely, too, when you realize this evolution only had to happen once in Romance. This semantic bleaching started in Vulgar Latin around the first few centuries CE; it was well-established by the time Vulgar Latin started splitting into the ancestors of French and Spanish and Italian and the like. So they all inherited it from the same source, rather than all having to develop it independently.
such a drastic structural change
The change is not drastic at all! It is a simple case of semantic bleaching (this is where the meaning of a word gets weaker. So you can kind of see how the is a "weaker version" of that).
Also it's not a structural change, since wherever ille and all its forms may be used, it's the same whether it was early on and meant that or it was later on and it meant the.
This had already happened in Late Latin. By that time, ille was pretty close to meaning the.
I agree with the earlier answers, but they essentially state that it was a possible occurrence and how it happened (ille, illa), not why it happened.
Could it be (I am only venturing a hypothesis), that there was an influence of another language, typically Greek?
While "standard" Italian has these forms:
(the street, the house)
Naples and other Southern regions have:
which is suprisingly close to Ancient Greek "hos" and "he" (with the Dorian variation of the eta into alpha). Furthermore, those articles "o" an "a" have a glottal attack that may resemble the Ancient Greek prononciation.
Could it be that, e.g. native Greek speakers, e.g. from the military or tradespeople, could have borrowed this article from Greek when speaking whatever Latin pidgin they were speaking?
After all, we saw this phenomenon of cross-pollinization much later with the Italian lingua franca of the Mediterranean (which e.g. adopted Arabic habit of doubling words such as poco poco).