It turns out, "parts of speech" are one of those formalisms that's taught in all the schools, but isn't always useful when you start looking closer.
Fundamentally, "part of speech" is a word's role in the syntax. "Walk" and "run" and "go" and "travel" all act pretty much the same, syntactically, so it makes sense to say "all of these things are Verbs, and this is how Verbs act". And then you find words like "eat" that act almost the same, but not quite, so you separate them into "Intransitive Verbs" and "Transitive Verbs", and write some rules about Intransitive Verbs and some about Transitive Verbs, and some about all Verbs in general.
But this stops being useful when words act, syntactically, like nothing else in the language.
The "infinitive to" appears in only one construction. It takes the bare form of a verb, and turns it into either a non-finite verb or a verbal noun, depending who you ask. And there's no other word in English that can do the same thing. It looks just like the "preposition to", but it doesn't act as a preposition: you can't say *he wants for go now, or *he wants behind go now.
So, historically, the solution was to call it a "particle"—where "particle" means "we don't know what category to put this in, but it needs a category". (In English at least: in a few languages, "particles" are a legitimate category of words that all act the same.) But as far as actually doing syntactic analysis, I'd just keep it in a class all of its own, which is what most POS-taggers will do: the Penn treebank, for example, gives it the special tag
TO shared with no other word.