1

What is a numerical estimate for the “RAM” of the human brain required to actually compute resolutions of the semantic content of sentences?

For example, consider there is an algorithm that expends 1 kilowatt of energy per morpheme parsed (as a completely random and arbitrary estimate, maybe a calorie of energy instead).

Perhaps we can estimate the parse complexity of a sentence regarding if we think there are transformation or rewrite rules from surface structure to some underlying one. Maybe some sentences are more energy-expensive than others, for some reason.

I would like to know an actual argument for the anticipated “computational complexity” of resolving the semantics of sentences, especially as compared to the syntax. What is the comparative rate of growth, in terms like O(n^2), “polynomial time”, etc., of building a human-quality semantic representation of discourse, compared to just parsing unstructured data (sound, text) into a syntactic parse that resolves constituency structures but does not grasp at a lot of shared “meaning” that the sentences have?

For reference, here is a semantic graph / network parser from Stanford CoreNLP: https://nlp.stanford.edu/nlp/javadoc/javanlp-3.5.0/edu/stanford/nlp/semgraph/SemanticGraph.html https://link.springer.com/chapter/10.1007/978-3-319-10377-8_10

I am motivated by a desire to know, if there is some sort of exponential growth in the number of computations or steps necessary to resolve an actual semantic understanding of a sentence, then perhaps, a large language model’s algorithm is less efficient than the human algorithm, and maybe it can be shown more concretely that current LLMs aren’t mathematically able to access good semantic parses, because their algorithm is too inefficient, or lacking something - or, maybe they are. I would like to put some numbers to that. Thank you very much.

1
  • To get a Fermi estimate of the computational size, you will need to look at the approximate RAM of the neocortex, Broca's region, and Wernick's region, because these are the three regions of the brain most responsible for language.
    – Fomalhaut
    Jun 25, 2023 at 21:21

2 Answers 2

9

There is a popular yet false presumption about brain structure and language that enables this question, among others. The simple answer is, we have absolutely no idea. The medical folks could tell us how they know how many calories per hour you consume chopping wood versus sleeping. There are no realistic estimates of how many calories are consumed proving Fermat's last theorem versus mentally reciting "Om mani padme hum". Because we built computers from the ground up and used to string magnets on wires, we know about "bits" as a minimal unit of storage, but we don't even know how "a piece of information" is stored in the brain, therefore we cannot say what the minimum unit of brain storage is, nor how a specific snippet of information is stored, nor do we know whether "data" and "procedures" are physically-distinct sorts of units.

Rather than give up on the enterprise, some people assume that human brains are sort of like known computers. The question then changes from "how many brain cells are required to compute language" to "what kind of storage and computation is necessary for a computer that (a) processes human language or (b) processes human language using the techniques used by the human brain. Addressing question (b) rapidly crashes into unknowability since we don't know how the brain actually computes language – that is the question that we set out to answer.

If we only consider the question from perspective (a), the one that addresses "what the brain produces" and not "how, in reality", we still have to have some model of an assumed underlying pseudo-machine, such as a Turing machine. An analogous question would then be "how many steps are required to access two decimal digits and store the answer somewhere", using a physical Turing machine (I assume someone has built one somewhere). Also, "how many positions on the tape are required to store the program" and "how many positions on the other tape are required to store the data", and maybe "can you construct a machine that has just one tape". Now, we can also ask the same questions about real inorganic computers, and there is no single answer, because some computers can add decimal digits, and some cannot. Instructions, registers and storage are the the atoms of computers, but those atoms ultimately are decomposed into smaller physical things. The "read" and "write" heads on a Turing machine are just taken to be primitives, but clearly IO systems are themselves marvelously complicated sub-computers.

Given that we have a currently unresolved level of ignorance about real brain operations, the standard practice in the field seems to be to assume a generic almost century-old model of abstract computing, and ask about "complexity" from that perspective. There is a body of mathematical work that addresses "complexity" from that perspective. The interesting question, from my perspective, is what other notions of "complex" exist out there. I suspect that all contemporary modeling of the notion of complexity is based on modern computer models, so we effectively can't break out of the Turing machine box.

0

Some possible avenues for consideration:

Power consumption or wattage of brain, from a physiological angle (ways to directly measure energy consumed, if at all possible - thermokinetics of bodily processes, or any metabolic or anatomical indicators of effort, like heart rate, blood pressure, eye-tracking, pupil-dilation, EKG, FMRI, FMRI with AI…

The human brain is said to have “working memory”, and it’s unclear to me how similar or different this is to a computer idea basically of “local memory” (RAM), or just “memory”, in a Turing machine - but it seems like a really important and relevant point. Maybe we can imagine - and learn a lot, from psycho- and neuro-linguistics experiments, what we think the “bandwidth” or processing power of human linguistic attention is (LLMs also have “attention”, funnily enough, though again, they are quite different, though intriguing at minimum to compare). Some experiments flooding humans with language (like speed-readers, or rapid speech), to try to see how much they can keep up with, and consider a smallest possible representation of some local system - a controllable, tiny “language”, regarding the current state of a simple object or system, like, “the armchair is now resting in its side; now its upside down;” etc.

It might be possible to move from a crudely simple such “language” which necessarily has to get parsed from the form in which its encoded, into what we assume is some abstract representation of the essential information, to subtly more complex ones, to see how human performance changes, rises or drops, across conditions, and compared to computer programs. This sounds like a promising avenue.

A common if cliche estimate of human working memory is “Miller's magic number", around 5-9 “digits” at any one time, but that should be verified or investigated extensively, as it seems likely that different kinds of information streams have different algorithms and processing pipelines in the brain (vision, sound, color, physical movements and responses to environmental stimuli, the reflexes of the autonomous nervous system controlling organs, etc.)

The human brain is said to stand out for being energy-efficient. It is supposedly a 20-watt “device”. A MacBook Pro is apparently around 87 watts, for comparison.

The mathematics of semantic parsing will depend on an almost tautological level on what kind of mathematical (or non-mathematical) system you think semantic parsing is.

It can at least be fruitful to explore the question on minimum, then increasingly augmented, models of (real-time) semantic comprehension - especially compare to a computer, how do human performance and computer compare for various cognitive tasks, or a battery of them, as you test hypotheses for the, or an, “algorithm” for that task? If we explore “computational economy” as a “strong hypothesis”, we maybe can even use machine learning to find the most efficient algorithm we know of, for a given task, and then see how it compares to human performance: does human performance compare to, lag behind, or excel beyond the fastest-time algorithm we know, if we hold energy constraints equal for human vs. computer? How do the two co-vary, across varying conditions, like different kinds of tasks, or algorithms?

Some semantic models, such as a graph one (below) used by Stanford CoreNLP, can vary because graph-algorithms - and all algorithms - can vary in what their computational complexity is. For example, finding the shortest path between two nodes is claimed to be a problem in the order of “polynomial time”.

Maybe we have to model it as three components, the pre-existing system, the algorithm that actually runs, and the input message. One reason a human may be more efficient when “running” than a computer is how much world-knowledge it already has, which is important when considering or attempting to run a computer model for the mind. Some networks can grow in exponential time regarding an input, but transformers can apparently execute in linear time (that’s said without knowledge of how quality drops, as input gets longer, even if response time is a constant function of length). Another factor adding information is context: the energy required to parse something may drop if perhaps conditional probabilities from surrounding info allow you to select one from what is actually a number of possible resolutions, of something (this is also true for phonology and syntax, for example).

Trivial bounds on the two systems - GPT’s, billions of parameters, human brains, trillions of neurons… much more to be added, here. (E.g., number of connections). We can imagine the energy consumption of GPT-4 is monolithic compared to a human brain. Depending on the situation or task, it can be faster or slower than a human.

It’s said GPT-3 has around 200 billion “parameters”, which are sort of meant to act like neurons. I think the human brain has somewhere in the trillions, 1-100, I can’t remember. It was apparently proven by a study at Google that the relationship between network width and depth (nodes per layer vs. layer) is ultimately mathematically equivalent. This sounds very comparable to the idea of writing a language as a gigantic context-free grammar, perhaps - you can have a lot of little rules, or a little complex rules which stack onto each other; the result may (perhaps) be the same.

Base system (elements and what are the rules, what can you do with them), and what is the optimum being sought? (The “cost” function of an evolutionary communication / language game, as a computer simulation or model)

Maybe there are actually 2 big parameters at play here, I do not know. One is the “intensional” richness of the language. Two is the energy expended. The first leads to questions of how world-representations can be transformed into a “transportation” medium (for communication). The second question is how they should (if conserving energy while relaying messages is part of winning the “game”, an incentive, of the model.)

Ambiguity could be profoundly energy-efficient, but there may be many other factors at play, to explore

The more ambiguity the messages have, then their ambiguity may increase exponentially, the longer the messages are. But there would be many interesting sub-cases to explore or consider.

The question remains, what is the minimum kind of semantic representation which is adequate, for human coping and functioning, in their environment? I now feel the best way forward would indeed be to start with tiny languages and explore, perhaps under computer world-models, their adequacy or inadequacy as a communicative system (under some game-theory pressures, as AI learns to play games via reinforcement learning).

LLM’s have mastered aspects of syntax while being essentially non-examples of even basic semantic comprehension. They fail on the most basic semantic puzzles imaginable if you choose one at random, instead of selecting text it can convincingly reproduce the right answer to as relevant text. If their basic mechanism of action is some sort of multi-level/multi-order clustering, why does that do well on syntax but not well at all on semantics? The fundamental question remains open. Is semantics more complex, or is it just different, like requiring a different algorithm altogether, a different embedding, or even being philosophically inaccessible since LLMs don’t have access to bodies (the embodiment hypothesis) or the outer world?

LLMs like GPT-3 or -4 often generate grammatically convincing language, while fundamentally lacking basic human-level semantic comprehension of incoming messages, of their input. Perhaps we can estimate, if GPT is something like a 200-billion-nodes-in-total decision-tree or compositional arranger, that that extremely inefficient (probably) architecture requires that much to do well on syntax. Maybe an argument that semantics has multiple reasons why it is more computationally higher-order than syntax or phonology can help set a bound on what the size of the current model-type would have to be to adequately represent semantic world knowledge. Perhaps it can inspire new methods and techniques for more efficient algorithms, on the other hand - if it is better understand what transformers really are doing, if that itself can be tuned or optimized. My belief is “symbolic regression”, a deterministic technique for finding optima to satisfy some equation, holds potential.

More research is needed to fully understand the computational complexity of human language processing and how it compares to the capabilities of large language models. We can develop more sophisticated models of semantics that take into account the context in which a sentence is used and the world knowledge of the listener or reader. We can also develop more efficient algorithms for processing language, for example by using parallel processing or by exploiting the structure of the language.

1: Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2), 81.

2: Howarth C, Gleeson P, Attwell D. Updated energy budgets for neural computation in the neocortex and cerebellum. J Cereb Blood Flow Metab. 2012 Jul;32(7):1222-32. doi: 10.1038/jcbfm.2012.35.

3: Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., & McClosky, D. (2014). The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations (pp. 55-60).

4: Jurafsky, D., & Martin, J. H. (2019). Speech and language processing. Prentice Hall.

5: Resnik, P. (1999). Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language. Journal of Artificial Intelligence Research, 11, 95-130.

6: Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

7: Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog.

8: Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:190 9: Herbert Simon, “What Computers Can’t Do”.

10: Hypercomputation: https://en.wikipedia.org/wiki/Hypercomputation

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.