Using it to represent phones is of course bonkers, it would make much more likely for an unitary language to be split apart. When we are dealing with phonemes that problem is inexistent in my opinion, yet this use is also criticized in my textbook.
"Anything bad" is a really broad category. There are various objections to phonemic spelling. (I wonder a bit why your textbook did not mention any of them.) Justin B Rye has a web page giving what I consider to be a good overview of possible objections to spelling reform, along with potential counter-arguments.
First, a note: there is a difference between making an orthography "more" accurate at representing phonemes and making an orthography correspond 100% to the phonemes (and all other possible phonological distinctions, like contrastive stress and tone) of a language: as far as I know, almost no writing system in the world reaches the latter extreme, but many writing systems are more phonemic than standard English orthography.
You can find some people who maintain that the standard orthography of present-day English orthography is in some way "ideal" or "optimal". Unless this is taken as an unfalsifiable statement of opinion, I think it's pretty hard to defend this in its most extreme form: I don't think there's any convincing way to argue for the optimality of every inconsistency along the lines of "deceive, deceit, deception" vs. "receive, receipt, reception" and every isolated quirky spelling like "ptarmigan" or "bass".
However, the less extreme claim that English spelling can be considered "near-optimal" in terms of some coherent, definable set of criteria is not so obviously false. Although I think that this claim has been advanced more because of its provocativeness than anything else, there are linguistic sources like Chomsky and Halle's Sound Pattern of English that argue that English spelling actually has some kind of underlying systematicity that we wouldn't get with a purely phonemic spelling system. It is well known that some aspects of English spelling show morphological relationships; e.g. the words observe and observation both start with the same letters <observ> even though they start differently in pronunciation (for me and many other speakers, /əbˈzərv/ vs. /ˌɑbsərv/). (A somewhat common term that seems to be based on a similar viewpoint is "deep" orthographies, as opposed to "shallow" orthographies.) If you think that it's important for a spelling system to show these kinds of relationships, then this is one area where "something bad" would happen if we shifted to using phonemic spelling.
See the following Faculty of Language blog post for some more discussion of the concept of morphology-based orthography and whether standard English spelling is actually very good at it: http://facultyoflanguage.blogspot.com/2016/09/chomsky-was-wrong.html
Aside from highly theoretical arguments like this, I think the most common objection is that different dialects of English have different phonologies, not just different phonetic realizations of the same set of phonemes. In principle, this is not such an insurmountable objection as it's often made out to be, because it is possible to design "diaphonic" orthographies that represent multiple sound systems at once, or some compromise between them. But practically, it would be difficult to design and standardize a single diaphonemic orthography for English that would be found acceptable by a majority of speakers from every region.
I think the appropriate question is, would things be net better or worse if..., and not, would anything bad happen. Regularizing the spelling would mean that you couldn't tell from spelling that amoeba is from Greek, and would make the semantic relationship between hymn, hymnal less obvious. Spell-checkers would have to be totally rewritten. A somewhat more earth-shattering consequence would be that (if we take 2020 as the year for imposing spelling reform), texts written before 2020 would need to be orthographically translated or consigned to the same historical dustbin that Middle and Old English are in.
A "phonetic" spelling would not be viable, not even one that represented each phoneme in a standard way (as contrasted to the current myriad ways of spelling [i], [ai] and so on). That is because the phonemic form of words is not uniform across dialects of English. At the minimum, there would have to be a couple dozen national standards for spelling, if we want a pronunciation-based spelling system.
More modest reforms could be somewhat viable. For example, writing [i] as "ee", [ai] as "igh", [ɪ, ɛ] as "i,e" could be a first step. Igh think thuh apropreeuht question is, wood things bee net betuhr or wurs if wee startuhd wrighting lighk this, and whuht wood wee gain in dooing soa.