In present-day English ought may behave like a 'modal' verb, taking an unmarked infinitive, only in negatives and questions, not in ordinary declaratives:
Ought I take that seriously?
You ought not take that seriously. BUT
You ought to take that seriously.
But even in negatives and questions the unmarked infinitive is not always required; for many speakers a marked infinitive is acceptable, too:
Ought I to take that seriously?
You ought not to take that seriously.
ADDED:
Even more likely in US speech is the "melded" form oughta:
Oughta we take that seriously?
You oughta not take that seriously.
And this tolerates what looks like lexicalization but is probably better understood as modal stacking:
Do we oughta take that seriously?
You didn't oughta take that seriously.
In any case, this is a question of declining importance: use of ought has decreased markedly in the last couple of generations.