Shakespeare used apostrophes the exact same way we do—to signal that a sound is omitted.1
There's nothing vague about his usage. As Draconis explains, "wand'ring" makes it clear that the vowel spelled by the missing "e" between "d" and "r" is to be omitted.2 Usually, as here (but not with "o'" for "of"), this results in an entire syllable being dropped, which changes the scansion of the line, which is of course very important in metric verse.
In formal written English today, we mostly use apostrophes only in fixed contractions—e.g., "don't" for "do not" and "she's" for "she is". But they're used exactly the same way. "Don't" signals that the "o" sound in "not" (and therefore a whole syllable) is omitted.3
It may seem that there are so few of these contractions that we just learn each one as a separate word, and no longer have an "apostrophe rule". But a moment's reflection proves that can't be true. You can contract "is" or "has" onto almost any noun: "John's coming later", "The dog's got a bone", etc.
And, outside formal writing, people still use apostrophes in other locations, like writing "drinkin' and thinkin'" to signal the normal colloquial pronunciation instead of the one with a "g" at the end4—exactly as Shakespeare does with "o'". In every case, it works the same way for us as it did for Shakespeare.
In poetry (and sometimes song lyrics and other similar things) where scansion is important, people even still use apostrophes in exactly the same locations Shakespeare did. The only reason this doesn't come up in plays as often as in Shakespeare's day is that most plays aren't written to strict meter anymore.
There are a few cases that may seem strange, such as frequent "'d" for "ed" on the ends of words. What's going on there? Well, think of "I learned of a learned man"—the first "learned" is one syllable, the second is two.5 This distinction was more widespread in Shakespeare's day, so many "-ed" forms that can only be monosyllabic to us were ambiguous to him. So he spelled them with "'d" to force the monosyllabic pronunciation, exactly as with "wand'ring".
1. There are, of course, other uses of the apostrophe—possessive "William's" doesn't omit anything, nor does the disambiguating plural marker in "p's and q's", nor do borrowed words that were spelled with apostrophes in their native language, except when that native language used apostrophes for omission. But that's also the same today.
2. Of course you need to know enough about English orthography and pronunciation to know how to omit the sound properly—in this case, that "wandering" can be pronounced as both "wan-dring" and "wan-der-ing", so the apostrophe tells you to select the former. But that's not unique to apostrophes; all English spelling works that way.
3. And again, the rest is up to your existing knowledge. In fact, notice how much more "don't" changes from "do not", compared to "wand'ring" from "wandering". Which one would you call more "vague"?
4. Of course there's not actually a /g/ sound at the end. But native English speakers think there's a /g/ sound at the end, so when you're writing plays, poems, lyrics, etc., using the apostrophe to signal a native speaker to "drop the g" (or just writing "eye dialect" in a novel), the signal works.
5. British spelling today usually distinguishes them as "learnt" vs. "learned", while a few generations ago they instead used "learned" vs. "learnèd". But Americans spell them indistinguishably, which makes for a better example here.