After some searching, I'm still unsure about what function the soft sign (Ь) performs in Russian. I have read that it indicates declension, palatisation, and iotation in different contexts, but with limited understanding of what the latter two concepts mean.
I suppose the function of the sign would also explain why it is variously represented as y or ' in Latin script.
WARNING: The question is sooo many-sided, it is very wide and can be split into at least 3 different questions. I'll answer it all, don't tell me later that you haven't been warned the answer would be long.
First of all, this letter has no sound of its own. The main function of the soft sign <Ь> in Russian is to change the sound of the consonant letter which stands before it. In Russian, most consonant sounds can be paired into couples of non-palatalized ("hard") vs. palatalized ("soft"). In English no consonant sound has such a pair; most of them are non-palatalized and for English native speakers it is usually a hard part of studying Russian pronunciation to learn how to pronounce those sounds palatalized.
Palatalization (P) can easily be explained, but is very hard to produce with the tongue (for those who have no such thing as P in their native language). The main idea behind it is that palatalized consonants (PC) are bi-focal: beside the main focus of articulation, the tongue also makes the second focus touching the palate with its back, hence the name. For example, Russian <C> is approximately like the English [s], with the tip of the tongue nearly touching somewhere near the upper teeth, but in the Russian <CЬ> [sʲ] the tongue also touches the palate with its back adding a [j]-like air to [s]. Still, [sʲ] is one sound, you can drawl, sustain it for a long time and all that time it will sound the same. (Not all palatalized consonants can be sustained, of course, just as not all consonants can be sustained.)
Another trick is that the P of a consonant sound can be shown not only with the soft sign, but also with vowel letters. They are also paired into the ones that cause P (they are called "iotizing" — that is, 'adding the [j] air') and the ones that don't (called "non-iotizing"). The pairs are like this:
Non-iotizing vowel letters: a э ы о у
| | | | |
Iotizing vowel letters: я е и ё ю
[1]
The idea behind the iotizing vowels is that when they stand after a consonant letter they palatalize it and then they are read as the corresponding non-iotizing letters. For example:
мал [mal] - small
мял [mʲal] - (he) crumpled
[2]
In the beginning of words and after another vowel letter the iotizing letters are pronounced as [j] + the corresponding non-iotizing letter:
ясно [ˈjasnə] - clear
бояться [bɐ'jatsə] - to be afraid
(yeah, here the soft sign doesn't affect any sound)
[3]
But what if we need to write a word in which a consonant is followed by a distinct sound [j] + a vowel, like a 3-sound word [bʲju] - 'I beat'? We cannot write it as бьйу, because (another trick is coming up!)
the combination <Й> + a non-iotizing vowel letter are (hahaha!)
forbidden in spelling of Russian words which were not borrowed from another language. [4]
If we write it as бю, then according to [2] it will be read as [bʲu], a 2-sound word... The only correct way to write that word is бью, because
after <Ь> the iotized vowel letters
are read as [j] + the corresponding
non-iotizing letters:
льёт [lʲjot] - he/she pours
пью [pʲju] - I drink
And at last, differentiating the gender of nouns with the help of <Ь>. In Russian, most feminine nouns end in [a], but there are lots of them that end in a consonant. In the latter case a soft sign must be written at the end of the word (ahaha!) irrespective of whether the final consonant is actually palatalized or not!
(masc.) мел [mʲel] - chalk
(fem.) мель [mʲelʲ] - shoal
(fem.) мышь [mɨʂ] - mouse (ends in a
non-P consonant!)
(fem.) ложь [loʂ] - a lie (ends in a
non-P consonant!)
This is actually everything you asked about! Naturally, there are other issues with <Ь>, like differentiating some verb forms where it doesn't in any way affect the pronunciation, or using it before non-iotizing letters in borrowed words, not to mention its historical usage problems, but that is another tale.
From the point of view of the spelling, Ь simply means "the previous consonant should be palatalized". See https://en.wikipedia.org/wiki/Palatalization_(phonetics) to learn what palatalization is, it's a lengthy topic (P.S. Yellow Sky has a nice overview). Most consonsants in Russian come in pairs: palatalized/non-palatalized (also called soft/hard). Additionally, before vowels, Ь means "y" (Воробьев => Vorobyov)
When it comes to morphology, -Ь is often found in cases where we have a null ending in words whose stems end in a palatalized consonant, example: путь. Also found in 2nd declension in nominative/accusative cases (of feminine gender) for historical reasons: ночь (consonant Ч is always palatalized, so no need to have Ь as a palatalization marker, but we still do).
-ь is also found, for example, as a morphological unit, in imperative word forms: брось! i.e. it tells that C is palatalized, witch matches with the palatalizedness of С in the original infinitive: бросить
In some cases, like -ешь in verbs (for 2nd person), the spelling is just historical, and Ь there doesn't mean anything anymore.
So, in a nutshell, you can think of Ь as yet another palatalizing vowel (similar to Е and И), except it's not actually pronounced, yet still has influence on surrounding sounds.
