There is a table of IPA performances by Peter Ladefoged here: they happen to be in AIFF format, but you can presumably work around that. Note however that only the vowels are produced with just the letter in question, and in the case of consonants you have a choice of "pa" or "apa", but not just "p". This is mainly because there's virtually nothing recordable with isolated stops, and formant transitions to a vowel are essential to identifying a consonant. The clearest case of the problem with trying to glue together isolated segments would be the attempt to glue together "tip" from the components "t", "ɪ" and "p". You can get a steady-state recording of "ɪ" (the snippet of "ɪ" on that page is 458 msc. long, which is abnormally long for speech). But "t" and "p" simply involve closing the lips or raising the tongue, and there is no actual sound during their production, so all you have is silence (the ability to discern "p" versus "t" comes from the formant transition effect that these articulations have on adjacent sonorants). This, "tick, tip, tit, pit, kit..." would all sound the same using this technique, namely [ɪ].
If you limit yourself to word without stops, you could piece together "s", "æ" and "ʃ" with steady-state recordings of those segments. If you do this, you also have to decide about the margins: do you leave an acoustic gap between "s" and "æ", or do you trim the samples so that you pick just the center of a performance? A no-tech way to decide this is to articulate the sounds "s" "æ" "ʃ" in rapid succession, and if this is satisfactory, you can go from there. If you aren't happy with that ("doesn't sound natural") you can trim the segments to eliminate the gaps and also make the durations more realistic, say vowels=100 msc, fricative=200msc. Piecing the parts together, I think you will find the result to be even more unnatural.
There is no repository of isolated fricatives out there, so you would have to create it. A simple way to determine if it's worth your time is to either record your own samples or copy the Ladefoged recordings and edit out the surrounding vowel, then paste the parts together. If that does not dissuade you, then you might try extending the set of consonants to include nasals and voiced stops, which you can hyperarticulate (strong voicing, sustained duration) and use as the basis for extracting appropriate centers. I think that the Ladefoged samples of "b", "m" etc. would provide a decent basis, since the articulations are longer than usual and voicing / nasality is stronger (more controlled).
One other impediment is that even if you have a transcription of a language using just IPA letters, the "exact" values employed in the transcription are often rather divergent from the actual sounds of the language. Supposing you had the word "cotton" in an IPA-transcribed database which gives you [katən] – that's one of the possible IPA transcriptions. But in American English it is usually pronounced something like [kʰɑʔn̩]. Unfortunately, all existing (professional) recordings of IPA letters omit the modifier-type letters such as "ʰ" and "̩ ", and you won't be able to find all of the aspirated, ejective, affricated and so on type modifications. And you will probably have to manually edit the transcriptions to apply allophonic details that are typically omitted, or to "correct" somewhat arbitrary choices in letter selection (the vowel of "hot" is not well-matched by either IPA "a" not "ɑ", so people typically pick one without denying the possibility of the other choice).
One fairly simple way to overcome the problem of formant transitions is to use "diphone synthesis". This roughly involves recording all two-phoneme sequences (from real words), selecting an area around the transition, and then gluing them together, so "sunshine" could be generated from a stock of parts of "sʌ"+"ʌn+"nʃ"+... However, I think all methods of synthesis require a serious dose of programming ability plus knowledge of acoustics. There was a question about IPA and synthesis here, you could slog through this thread on xkcd, and this is also on the topic of IPA synthesis. And let us not forget the virtues (and vices) of Google synthesis, where you can play real and non-real words in English (and get credible results as long as it's phonotactically possible in English). This is Finnish gibberish, which sounds like Finnish. Swahili gibberish doesn't sound like Swahili; Icelandic gibberish is a bit better but not good. There is no synthesis for quite number of languages.