What are the main differences between machine learning and classic approaches to natural language processing problems?

Question

What types of NLP problems are best suited to machine learning and which are best solved by more classical approaches (e.g. syntactic and semantic analysis?)

Answer 1

The major dichotomy in NLP is that of rules-based approaches vs statistical approaches.

(Machine learning is used in some statistical approaches. Given the timeframe of the development of the field, the earliest statistical approaches, for example those of SMT pioneers, surely can or will be seen as classical.)

Rules-based approaches are basically the obvious or naive approach - hand-tweaked heuristics. They require linguist experts for every language launched, they are hard to bring to full coverage, and as coverage increases so does complexity. They are inherently static (which is a drawback in a world where the probability of the parse of the photo caption Nice truck attack changes in real time).

Statistical approaches are better suited to open-ended problems, but they require initial investment and plenty of data, and people who understand the principles of linguistics and statistics (and software). They can update as the data update or with humans in the loop. But as they always produce results they can produce surprising results, for one they trained on imperfect data, and not all machine learning systems make explainable or even deterministic decisions.

The attractiveness of statistical approaches has followed the steady growth in problems, data, CPU and connectivity. In practice today's production systems use a (often messy) hybrid of rules-based and statistical approaches. For example a search feature may be use rules-based filtering, but statistical ranking, with ad hoc boosts and blacklists.

So the decision of which approach to use is only partly a function of the problem type. Just as important are the goals, scope, data and so on. At another extreme there is even the human-backed approach - a human pretending to be a machine pretending to be a human - which has high accuracy, high latency and high cash burn.

Answer 2

Machine learning tends to produce results, that look impressive but cannot be tracked by humans. A statistical machine learning algorithm may have really good scores at some classification task, but when you analyse how it comes to its conclusion, it uses a% of feature_1, b% of feature_2 and so on to base its decision on.

Classical approaches select features humans can recognise and describe easily, and base their decision on only one or a few of them. Machines use several dozens of features that are easy to compute (e.g. frequency counts, a hard task for a human) and base their decisions on many of them.