What are the most relevant ways to store expressions in NLP?

Question

I am trying to understand the best ways to store expressions for NLP from an exercise I received

Consider the following two expressions, which have the same value. Which one will typically be more relevant in NLP? Why?

"Monty Python"[6:12]
["Monty", "Python"][1]

Natural Language Processing with Python, Ch1 from Steven Bird, Ewan Klein and Edward Loper

In my opinion it is the second one because the first ones refers only to letters whereas only words are meaning and can be meaningful for NLP. But this is only a preconceived opinion and I was therefore wondering what would be a more canonical answer.

Answer 1

Yes, from the perspective of that manual the second answer is the right one.

2.1 Lists

What is a text? At one level, it is a sequence of symbols on a page such as this one. At another level, it is a sequence of chapters, made up of a sequence of sections, where each section is a sequence of paragraphs, and so on. However, for our purposes, we will think of a text as nothing more than a sequence of words and punctuation.

A sequence of words, not a sequence of characters. (Each punctuation mark is normally tokenized as a separate word in the NLTK, hence "words and punctuation".)

More broadly, it certainly is within the realm of NLP to decide how to tokenize a list of characters, and the NLTK even gives you different options. In fact, as you work with texts you will probably get them as a single string of characters, meaning it's up to you to apply a tokenizer. But this manual is, as you guessed, asking you to look at it from the place where more interesting theory applies — after tokenization.