Your criteria for deeming something to be "lexical stress" kicks the can down the road, by stipulating that stress location in insight / incite is "part of the lexical item". Usually we say that something is "part of the lexical item" if it has to be stored in the lexicon, and is not the result of applying rules that give different surface forms. There is a well-known grammatically-based difference between nouns and verbs in English regarding the placement of stress, and insight / incite is a good example of that.
The sound pattern of English is a classic analysis of English phonology which completely predicts stress (making it entirely non-lexical), at the cost of some abstractness (for example positing a consonant cluster in vanilla as opposed to none in Pamela). There are other somewhat less-abstract analyses, for example Hayes' analysis that says that certain things are "ignored" (extrametrical), which reduces to enclosing part of the string in parentheses and "not seeing" that part of the string when you apply the stress rule. Ultimately, there is some lexical marking, even if you don't mark stress itself in the lexicon.
At the level of data, the pertinent question is what creates the surface data that makes one think that there is "lexical stress". There really is no single answer other than to say "because it's not possible to predict", but we also have to include some condition like "based on any reasonable analysis of underlying forms". For example, many dialects of Arabic have a Latin-like stress system which stresses the last heavy syllable within a final window of three syllables (otherwise, stress is on the antepenult). But some (many) of those dialects also split up final consonant clusters with epenthetic [i]. The interaction of these rules is the surface stress contrast [ˈkatabat] 'she wrote' vs [kaˈtabit] 'I wrote' from /katabt/. There are ample grounds for distinguishing the endings /t/ vs /at/, and it is perfectly reasonable to maintain that Arabic stress is predictable, just not totally trivial. It is not trivial to the point that it can be left off of (good) transcriptions, but it is not memorized.
Latin had a contrast between long and short vowels, which was lost in the Romance languages, but still resulted in some positional contrasts in Romance (under the reasonable assumption that the modern languages do not covertly maintain historical vowel length). The broad answer is that via historical changes, a transparent stress system can become opaque due to changes in the conditioning factors. But the specific reason for lexical stress in Slavic is different from the specific reason in Romance.
The opacity of English stress is very much connected to competing lexical sources, in that Germanic vs. Romance vocabulary had different surface stress patterns. Place names famously exemplify the principle that you need to know the etymology to know the stress – people from outside the Pacific Northwest tend to use "general rules" to stress words like Yakima, and there is local variation in more-obscure names like Swinomish. Algonkian place names follow different rules, which people from Maine know, but I don't know (except from having read the stress literature). English stress is not entirely memorized, but it is at least highly memorized.
Whether or not one should deem Japanese to have "lexical stress" is a matter of substantial controversy – the alternative is that it is a restricted tone language. The Tokyo system is pretty restricted, but some dialects (e.g. Ibukijima) are, at least in terms of surface contrasts, similar to West African tone languages. I have seen highly-restricted tone languages being described as having "lexical stress" (certain Bantu languages such as Safwa). There isn't a particularly good argument one way or the other for declaring that Safwa has lexical stress vs. restricted tone.