The distinction is not as systematic as you might think. Transliteration presupposes a standard writing system such as Cyrillic, Devanagari, Fidel or Latin, and maps one system to another. Writing systems include not only representations of pronounceable stuff, it includes numerals and punctuation. A transliteration from Arabic to Latin would therefore change Arabic numerals like ٣ to "Latin Arabic" (not Roman IV, but 4).
Let's take the word ብቅዓት in Tigrinya. This can be transliterated as bɨk'ɨʕatɨ, which is a simple mapping of 1 Tigrinya letter to a corresponding CV sequence in a different alphabet. This alphabet is augmented Latin, which include letters like ɨ, ʕ because standard a-z is inadequate. You do find the Latin alphabet being augmented to make up for deficiencies in what it can represent – and Cyrillic, Devanagari and Arabic are likewise augmented to include ө, ڳ etc. An alternative transliteration of this word is bɨk’ʕat, which uses a context-sensitive mapping of letters, where ቅ could be transliterated into k' or kɨ, depending on what letter precedes or follows. As it happens, this is because of pronunciation. Yet another transliteration is bekat – this uses nothing but ordinary Latin letters. A transcription is a representation of how the word is pronounced – [bʁ̆a̰t], at least in the Asmara dialect.
Transliteration and transcription are both conventionalized to some extent. My transcription of the word follows IPA conventions, which is the most popular for general use. There is an area-specific transcription convention for indigenous languages of the Pacific Northwest, employing letters like ƛ̵, š, c, x̌, and there are somewhat-competing phonetic alphabets like the Americanist and Uralic Phonetic Alphabet.
In general, transcriptions are just about the pronunciation of words, with consideration for the system of contrasts in the language, whereas transliterations are conventionalized in a very language-specific fashion. The transliteration ṭ could represent Tigrinya ጥ = [t'ɨ], Arabic ط = [t̴] or [tˁ], Tifinagh ⵟ (same) or Devanagari ट = [ʈ]. In theory, transcription is not defined in a different way for each language, thus [ʃ] and [e] mean the same thing in all uses of the IPA, except that in practice there is huge variation in how words are transcribed (especially in English). But this variation generally reflects either different phonological analyses, or else actual dialect differences.
When a language is unwritten, that defines one choice – you have to transcribe, you can't transliterate. At some point, a transcription system can become an orthographic system (e.g. the mostly-standard Lushootseed spelling system or the old Shona spelling system). The original Dogri system (Takri) got supplanted via transliteration where a Takri letter was changed into a Devanagari letter, and now it can be transliterated by convention into Latin letters. But those spellings don't really tell you how words are pronounced, for that you need actual pronunciation from a speaker of the language (or a letter-to-pronunciation guide, as may be fund in some grammar books).
Transcription is mainly used when you care more about how words are pronounced. Transliteration is used when you want readers to be able to reconstruct the original text (in Arabic or Tigrinya) but don't assume that all readers know how to read Arabic script.
