Modal verbs exist in many languages; but they are often defective. English is an extreme example where they seem to only have present tense forms; and have no gerund, participle, or infinitive; some also lack a past tense. I was wondering how does that happen? I know that other Germanic languages have modal verbs that are less defective than those in English; for example, the German "kann" has a past tense and an infinitive; unlike its English cognate "can". Did those verbs ever have some of the missing forms in modern English centuries ago? If so what were those forms? Or are some of the forms missing in English but present in German innovations of German?
The simple answer is that they cease to be "verbs" and become some other category entirely. In the formal study of syntax, we could say they have category T ("tense") instead of category V ("verb"), or more simply we could say that they're now "modals" rather than "verbs". The purpose of these labels is to talk about shared behaviors, after all, and modals and verbs don't behave very similarly at all.
As for how this change happens, syntactic reanalysis and specifically grammaticalization is a common way for languages to change over time. Those terms should lead you to good resources on it.
In many cases, both the present and past forms of English modals are still around: can~could, will~would, shall~should, may~might. (The big exception is "must"; "mote" has died out except in a couple fossilized phrases.) We still see this alternation when the "present" modals are used in a past context: "I think he will do this" but "I thought he would do this".
Other forms have died out, but can be seen in older varieties of English, when they still conjugated like verbs: "canst" (second person singular of "can") and its ilk appear in Shakespeare, for example. They generally look exactly like you'd expect (predictable endings stuck onto the verb stem). A few have taken on a life of their own, like "cunning" (originally a participle from "can"), and are now used without any apparent connection to the verbs that birthed them.
This is the result of historical devolopment.
The Germanic language (all of them) have a bunch of verbs where the past tense acquired a present tense meaning, called Präteritopräsentia in the German literature. This class of verbs was originally not restricted to modal verbs, the German verb wissen "to know" is a member of it, and the English verb to dare also belonged to it (You can still say He daren't do this without third-person s on dare). The verb will was originally not a member of this class but in a class of its own even going back to Protoindogermanic (compare the irregular Latin verb velle).
Now, in English the only surviving verbs of this class are all modals, and almost all modals belong to this class, with the exception of will that was later assimilated into this class. While German completed the conjugation paradigm for its präteritopresentia, the English ones stayed defective in some sense (no participles, no future tense, and the past tense can easily be reanalysed as just another verb with specialised meaning).
The lack of non-finite forms for English modal verbs seems to be associated with their role in English grammar as a special kind of auxiliary verbs.
These verbs share the historical morphological property of being what are called "preterite-present" Germanic verbs. There are attested non-finite forms, including infinitives, for some preterite-present verbs. But it seems like the absence of non-finite forms was an early characteristic feature of those preterite-present verbs that were used as modals.
According to the paper "The evolution of surviving English preterite-present verbs (āgan, cunnan, *durran, *magan, *mōtan, *sculan): a corpus-based study" by Magdalena Tomaszewska, several of the ancestors of English modal verbs such as *mōtan and *sculan (the ancestors of modern English must and shall/should) are already unattested in the infinitive form in Old English (Table 1, page 3). (A previous version of this answer incorrectly stated that they were attested—I was confused by the practice in dictionaries of citing these verbs by their hypothetical infinitive.)
However, can was attested in its infinitive form cunnan. The explanation seems to be that can and other preterite presents that were used in the infinitive did not have a modal meaning when so used: for example, cunnan was also used to mean "to know" with a noun as its object.
Thus, "Grammaticalization of modals in Dutch: uncontingent change" (Griet Coupé and Ans van Kemenade, 2009, Chapter 15 in Historical Syntax and Linguistic Theory, editors Paola Crisma and Giuseppe Longobardi (eds) (2008) Oxford University Press) summarizes which forms are found in early Germanic languages as follows:
We conclude that Old Germanic preterite presents, when they have modal meaning and take an infinitival complement, are always finite. Thus, they cannot feature as the middle member of a three-verb cluster. (page 4)
The development of non-finite forms of modal verbs in some modern Germanic languages such as Dutch is described by Coupé and Van Kemenade as a later innovation (page 8).
As for why the grammaticalization of these verbs as modals goes along with them lacking non-finite forms, that I am not sure about.
Frans Plank, in "The Modals Story Retold" (1984) writes "the modals would belong to a word-class for which all these verbal paradigmatic categories as such are irrelevant in the first place, rendering the concepts of paradigmatic gaps and zero exponents strictly speaking meaningless" and suggests that they "have to be conceived of as finiteness markers of their clauses or verb phrases" (page 328).
Speaking of these modal verbs as defective is derived from the attempt of fitting everything nicely into square boxes. Deriving them from Proto-Indo-European is no small task not only because they do not fit nicely into these boxes. In some cases, the roots themselves are not entirely certain. To derive them from first principles is to work our way backwards, however.
Since "can" is in question, intuitively speaking, German kann ("can") could have a weak preterite in kannte ("knew"). "I knew to" has perfect semantics which could has not. Since subjunctive and past tense have merged, could is rather imperfect. German still distinguishes pret. konnte from Subjunctive könnte, but the latter is equally expressed by kann ja, könnte ja*, which could be a reanalyzed cognate from present subjunctive *-ī on the respective stem. The reconstructed root for can and know would be the same Proto-Indo-European *ǵneh₃- "to know" in some view, but the matter seems to be unresolved. can < *kunnaną < *ǵn̥-né-h₃-ti has an n-infix and a sylabified nasal, correctly reflected in u (cp. un- < *n-); know < *knēaną on the other hand is optimistically deemed cognate, "but the exact mode of derivation is unclear", "''apparently from *ǵneh₁-, of unknown origin''", without Gothic comparisons. (Wiktionary1, 2, 3).
This leaves room for idle speculation, which is on its own not reliable. Eg. zero-grade *ǵn̥h₁ is not unknown, cp. *kunją "kin, family", *knōsl "race, descent", see also Ger. Bekannte "friends", bekennen "confess" (be alied to). Though it is true that etmylogies for these roots are wanting, linguists seem to agree that they are not related, which brings us back to the question about modal verbs, although the laryngeals are not supported on the Germanic side.
The other ones are shall, will, must, may, † tharf for canonical examples, further, ought to, used to, dare (Wikipedia), and I dare add to try for good measure. Every single one of them is questionable.
- shall / shoul < *skal (infinitive *skulaną) < PIE *skel-4 is per se not convincing, and Baltic cognates in close proximity to Germanic don't change that.
English has of course to scold. Related Ger. beschuldigen "to blame, implicate", Schuld "guilt, debt" < PIE *skél-ti-s ~ *skl̥-téy-s (Kroonen apud Wiktionary) is fairly important for legal history, making Schulz one of the most prominent names. However, the verbing may very well be backformation from the adjective, schuldig, Old High German sculdīg, inasmuch as participles cover both grounds. Pfeifer glosses "sculdīg `schuldend, verpflichtend, verpflichtet, zugehörig, geeignet’ (8. Jh)"5, ie. obliged: cp. German part. obliegend, ind. auferlegen, aufschlagen. This pseudo legalized layer of Germanic by extension, is poorly understood because it would have been writen almost exusively in Latin if at all.
As far as the evidence goes, the word might just come from the confluence of several idioms, eg. weregeld" blood money": cp. gold, gulden, Geld, or better heilen" (judicial "to restitute"). Guilt is etymologically uncertain, mind. However, much more striking are the isoglosses gel, wa (ca. "isn't it?"), the former allegedly short for gilt es, akin to guilt, the later from wahr "true, very", possibly influenced by wad "what" (as in Singapor English), because Germanic w- seems to have been loaned as gu- into French, cp. guarantee, warantee, quarantine, and quid, Quittung, quod erat demonstrandum. Also of interest, cuckold, cockewald, -wald, in a conceivable legal sense "plaintiff" would point to a common root *(h₂)welh₁-, cp. Anwalt "lawyer", but the first element remains unknown since cuck is compared to the cuckoo call and judged sound-immitative, so that diminutival suffix -ulus wouldn't even seem that strange. Jt's a cold case, since sigmatic aorists have no place tbis side of the schizma.
- will, would, want < ...
Incidently, *h₂welh₁- is reconstructed as the root of will. See my so far unanswered question for more (Origin of "will" in Germanic, wouldn't it be subjunctive?). More recently I argued that to want also belongs here. Again, mainstream communis opinio finds *h₂welh₁- and *wenH- unrelated. Oh Well! Even though, a subpletive paradigm may be gleaned.
It is fairly obvious in my humble opinion that may and must have been related. This must be difficult to workout because of several situational interpretations with as broad a meaning as "to go", to which cp. miktieren, Latin meō "go about, wander", moveo "move".
Besides the n-infix, productive negation as per wouldn't throws things out of proportion. mustn't and muss nicht are notorious false friends.
The root noun might on the other hand is chiefly nort-west European when triangulating the evidence (cf. JIES '22 Vol. 1). It would be wrong to reconstruct this for PIE.
The other ones wont fare much better, I'm affraid.