In real language, having affixes means that inevitably there will develop phonological rules which change the pronunciation of roots and affixes. The only way to avoid that is to not have affixes, and that's not something that can be controlled in real languages.
First, it is inevitable that a given morpheme (word) will be produced differently depending on what comes before or after. This is basic coarticulation 101. Second, if morpheme distribution isn't random, certain sequences are likely to be more common, and may get conventionalized. Third, especially when separate words get mashed together for form bi-morphemic words (prefix+root or root+suffix), i.e. when historically separate "the pig" becomes a single word /ð+pig/, you are more likely to develop conventional phonological rules that lighten the perceptual and production load of sub-optimal phonetic forms (all phonetic forms are ultimately sub-optimal by some standard).
The problem that you seem to be alluding to is that sometimes, if there are not many differences marked with affixes and not a lot of roots that the affixes combine with, you may not be able to parse the roots and affixes and figure out the phonological rules where /ban+ton/ → [baden]. English has very few productive affixes. Bantu languages on the other hand have very many, and there is no problem with learning those systems because there is so much data that tells the child what the roots and affixes are. A given root can have millions of inflectionally-related forms, and you clearly cannot memorize all of the forms, so you either get rid of morphological differences, or you keep the phonological rules easy to learn.
Suppose you have a language with 500 roots and 20 affixes, but you only have 2,000 root+affix combinations, and in that set there are only 10 examples where the root+affix combination diverges on the surface from the simple morpheme concatenation. You can memorize those special cases; you can also decide not to memorize those combinations, so you get analogical leveling. The hardest solution to learn, in this case, might be "also learn the rules". If the rule is trivial like "t→s/V__V", perhaps that much can be learned. An alternative (known as analogical leveling) is that the child thinks "I must have heard that wrong, it should be [feton] and they don't have that rule anymore. Children don't faithfully reproduce what the adults around them say.