Erudite English has an interesting practice where the plural form of loanwords may follow the inflectional grammar of the source language. Thus "campi" as well as "campuses", "minima" as well as "minimums", "these kanji were" as well as "these kanjis were". Though this is likely a conscious, artificial scholarly invention, many of these word-forms have been absorbed into the living language, to the point where some plurals all but only occur in the Latin style (algae/??algas, larvae/??larvas, desiderata/??desideratums, nuclei/??nucleuses).
Since speakers are normally unfamiliar with the source grammar, this practice sometimes generates putatively "foreign" plurals that wouldn't be part of the source language: "virii" for what in Latin would be *vira, or "octopi" rather than Greek octopodes (though some people do use the latter). The so-called "fake" or "spurious" plurals might be argued to be evidence of productivity, and thus of an English mini-grammar (in the construction-grammar sense). The implication is that "Latin-style plurals" (at least) are now part of the natural English language, if peripherally (with productive rules like "if a word end in -us, its erudite plural is -i").
I've heard that German, a language with four cases, used to go a step further; I don't know anything about old German practices, but apparently erudite texts used to inflect Latin loans in the corresponding cases whenever appropriate in a German context. It seems that Catholic liturgy still declines Jesus, for example: „daß ihr einerlei gesinnt seid untereinander nach Jesu Christo…“ (where the German preposition nach, which governs the dative, causes the Latin Jesus Christus to be declined in the Latin dative).
Are there more examples of languages where loanwords keep their source-language inflection, even if only in erudite or literary usage?