I'm currently completing a Masters of Linguistics, specializing in Computational Linguistics, after a Software Engineering undergrad so I think my experience might be relevant to you.
I think you may be a bit disappointed by the actual math involved in linguistics. Setting aside Computational Linguistics, the only real math I see is statistical analysis used in academic papers (and not something you'll generally need for your course work). Of course, semantics requires some basic set theory and algebraic logic but as a computer programmer by training and [formerly] by trade, it was all fairly basic. My [limited] understanding is that the cutting-edge work in syntax, semantics, and phonology come from examination and theorizing of frameworks involved in language usage.
Computational Linguistics is where you're going to see more advanced math; being a cross-disciplinary field with Computer Science. Machine learning and AI use various mathematical models and statistics. Information Retrieval heavily uses linear algebra and statistics (with some algorithms even using graph theory). Speech recognition and speech production all involve a certain amount of signal processing which you may find interesting. The problem with Computational Linguistics though is that it is typically part of the Computer Science department and this might limit the type of projects you'll get to do and the types of classes you'll get to take.
However, I'm more concerned that you don't seem to have any set goal other than "I did a degree in math and this linguistics stuff looks interesting. I wonder if I can combine the two?" I wasn't much better off when I decided to pursue linguistics but it sounds like you need to do a bit of leg work before you deciding that linguistics is right for you. Linguistics is a very broad and diverse field of study. Your first step should be finding what part you're interested in. Try to audit some introductory linguistics courses and/or skim an introductory textbook. Think about what made you interested in linguistics in the first place.
Once you decide what area you want to pursue then you can start deciding what courses would help prepare you. In general, the math used in linguistics is very applied (as opposed to the more theoretical stuff you're probably used to) so you might be better off taking some engineering or computer science courses. A course on signal processing would be very helpful if you're interested in speech recognition but useless if you're interested in semantics. Similarly, if your interests lie in question-answer systems then some courses in AI would be helpful but would get you nowhere if you're interested in phonetics. Without an area-of-interest it is very difficult to answer your question.