I don't know the details, but here's what he said in an interview in Glot, a newsletter for linguists.
Originally, I was in mathematics, but for no particularly good reason — I really hadn’t thought much about what I was getting myself into. I was taking some language courses and enjoyed them and at the same time I got more and more turned off by mathematics. Eventually I thumbed through the University of Chicago time schedule and saw that there was something listed as linguistics. So I sat in on a linguistics course taught by Eric Hamp, which I greatly enjoyed. Then I got a mathematics scholarship to study in Germany, at the University of Münster. But instead of doing very much mathematics, I took all sorts of language courses, including a Dutch course. During that year, I got more and more turned off by mathematics. After I got back to Chicago, I wanted to take a language course just for the fun of it. Japanese was offered at a convenient time, so I took it and I fell in love with the language right away. I also started looking around in the library for linguistic books. I came upon Syntactic Structures and it really turned me on. Not long after that I saw the announcement for the new linguistics graduate program that they were starting at MIT. I applied, got accepted, went there, became a linguist and I have been enjoying life much more.ever since then.
Linguistics is fun!
Languages are weird and wonderful things. As long as you are perceptive enough there is plenty to keep you happy and busy. In that respect, I think linguistics is way to hell more interesting than mathematics. There’s a wonderful quotation from Bertolt Brecht at the beginning of Feyerabend’s Against Method: Ordnung gibt es meistens wo nichts ist. Sie ist eine Mangelerscheinung. “There is order mainly where there is nothing. It’s a phenomenon of absence.” And, well, Ordnung gibt es in der Mathematik.
I guess you could say he loved mathematics but he loved data more.