6

Even though ChatGPT can - better or worse - mimic other writers' styles, it exhibits something like a default style of writing which is not very "characteristic" (because it's the superposition of so many styles), but maybe characterizable nevertheless. I think of linguistic tools like the analysis of sentence or phrase structure (are there sentence structures that ChatGPT never generates?) or of the structure of the generated responses as a whole which for example often comes by default as a three or five paragraph essay when you ask ChatGPT to elaborate on some topic.

By "default" I mean the text structure and writing style that ChatGPT comes up with by itself, without being instructed to choose a specific one.

Which analytical tools would be a good starting point - and has someone already taken on that venture?

Related questions:

Opinion-based side note (skippable): It is hold by some chess experts that AlphaZero has a characteristic style of playing that distinguishes it from more rule-based and often brut-force chess programs and lets the expert think that AlphaZero is playing like a very advanced human player. Its style of playing can be described in human-understandable and "playable" terms. At least its style may be described as "elegant" sometimes. Would one describe ChatGPTs style of writing as "elegant" sometimes? I suppose not. (I could provide references but would have a hard time to find them again - and to be honest, it's more of a vague impression I have.)

5
  • I like this question as a sane (methodical) approach to the analysis of AI, as opposed to "Hey, can we train AI to recognize AI and offload our brainpower even more?" May 15, 2023 at 18:47
  • It would be great if the downvoters could give a short hint to their reasons. (I also find my approach sane - so what don't you find appropriate for this forum?) May 16, 2023 at 9:27
  • 1
    Show some results. I haven't noticed any "default styles". I never downvote, but there's nothing here to vote for or against, except "Let's study the output". I agree. When you find something, comment on what you found, instead of simply repeating the output.
    – jlawler
    May 16, 2023 at 15:23
  • @jlawler: It's just an impression but I believe to have found that the sentences ChatGPT produces have only medium complexity, and there seldom are syntactical surprises. And it might be the case, that the sentences have some resemblance in structure - or some typical distribution of structures. (That there seldom are lexical surprises doesn't astonish me. But maybe the reasons are the same.) May 16, 2023 at 17:41
  • 1
    I think it be high time to open a discussion on the meta about if GPT-related questions really are so “off-topic” - part of me wonders if people just think computational linguistics is for software people - if this site doesn’t want a focus on that, we need our own site.. May 17, 2023 at 9:19

2 Answers 2

1

This is such a fantastic question, I’ve struggled to offer even a small, direct answer.

Let us begin in the simplest possible way, before hopefully attempting some generality.

The easiest thing you can do is observe for yourself and form some interpretive conjectures.

I talk to ChatGPT constantly, so here are a few of my own from the “humanistic”, hermeneutic side of “reading” ChatGPT and building an interpretive structure or foundation, in my own mind.

Keep in mind that ChatGPT changes regularly, so it is not the same model (in terms of output and behavior) as of writing this as it even was say, 2 months ago. They are changing it and upgrading it, and it experiences real-time human feedback, so it changes continuously, if not blazingly quickly.

Let’s start with some simple comparisons between its output on any arbitrary input, and what we think a human’s might be:

(This is “ChatGPT Alpha, July 1st, 2023):

Prompt:

yo jimmy, wadappppp???

ChatGPT:

Hello! I'm actually an AI developed by OpenAI called GPT-3, not Jimmy. But I'm here to help answer any questions you might have or assist you with anything you need. How can I assist you today?

Observations:

  1. ChatGPT text responds as if it is an entity with self-knowledge, i.e., it uses “I” pronouns (its “preferred pronoun” is “I”). It maintains consistency pretty well, unless you “steer” it via prompting to act like a different type of text, for example, addressing you from a different persona or perspective. However, it will probably revert to the ChatGPT-persona text as a default - it requires inertia to move it out of that discourse mode, but it returns or falls back into it in lack of other (clear) instructions.
  2. ChatGPT’s language is eerily “regular”. My hypothesis (require testing) is “the average effect”: it is the resultant style, word choice, grammar, content, vocabulary, text structure, and so on, of the “average” of that format or medium. This is why it has impeccable spelling and grammar, and excels in particular, reproducible, predicable tasks like marketing copywriting, any standard template of writing like blogs or technical manuals, and coding, but it (arguably) abysmal at driving “out of the lanes” because it has absolutely zero cognitive architecture in the algorithm that would enable generalized thought - it does not answer questions accurately, when it does, because it knows, but because it is more like a document retrieval system that can guess which string of tokens is likely to follow from some input string - so it can regurgitate text it’s trained on, and (I hypothesize) is abysmal at anything not already in its training text (famously so - it hallucinates “hardest” so long as you ask it a question that is not written down in its training text).

That “averaging” effect is both its immense power and frustrating shortcoming - if suitably trained, it can write an astonishingly good Ph.D. Thesis (in my opinion), but only by copying the format and content of all other theses and academic research articles in the world. It cannot diverge from that format with any kind of “intention” to have done so well, but due to inherent randomness, sometimes like a mutation in biology, a random variation in an otherwise conventional structure is recognized by a human as compelling or original - so ChatGPT can produce wondrous instances of creativity or new ideas, but it’s sort of like it’s an accident, and it could have equally been nonsense or something you don’t like.

The key property is that averageness. I can follow up with more specific examples of how so much of its grammar has a level of stylistic regularity and sometimes cleanness that it is very hard for a human to produce as quickly.

This is why ChatGPT is a one-way function: it is not good at pretending to be a real human with all of their fine-grained particularity in how they write, since it adheres to the average, by its very design. Try asking ChatGPT to “voice clone” your writing style, some time - the attempt is feeble:

Hey,

I want you to write a few sentences that sound exactly like the same style in which I write (this sentence). Show me.

ChatGPT Hey there,

Sure thing, I gotcha! Just toss me any topic you're curious about, and I'll whip up some info in the same laid-back, casual style you're rockin'. Let's get this show on the road, yeah?

(With more examples, it remains to be seen if it can improve significantly, I don’t know yet.)

To be continued…

User Suggest 3 ways in which your writing so far does not sound like mine, and suggest precise ways to change that to make it more exactly like my own way, style, and voice, in writing.

ChatGPT Absolutely, let's break it down:

Casual Language: Your writing style is very casual and informal, using phrases like "wadappppp" and "yo jimmy". I've been using a more formal tone. To match your style, I should incorporate more slang and informal language. For example, instead of saying "Absolutely, let's break it down", I could say "Aight, let's chop it up". Punctuation: You use multiple punctuation marks for emphasis, like "???" and "!!!!". I've been using standard punctuation. To match your style, I should use more expressive punctuation. For instance, instead of "How can I assist you today?", I could say "How can I assist you today???" Directness: Your sentences are short and to-the-point. My responses have been more verbose. To match your style, I should aim for brevity and directness. Instead of "I'm here to help answer any questions you might have or assist you with anything you need", I could say "Got questions? I'm here to help!" So, if I were to rewrite my previous response in your style, it might look something like this: "Aight, let's chop it up. You're all about that chill talk, yeah? So, I gotta drop the formal stuff and get with the slang. And those punctuation marks you use??? Gotta get on that. Short and sweet is your game, so I gotta cut the fluff. Got questions? I'm here to help!!!"

User Keep analyzing everything I had said in the whole conversation, even what I am saying right now. Compile a list of canonical features or examples, as a blueprint for, “how to write like Julius H.”

ChatGPT Sure, let's dive deeper into your writing style, "Julius H." Here are some key features I've noticed:

Informal Greetings: You start the conversation with a casual and friendly greeting, "yo jimmy, wadappppp???" This suggests a relaxed and informal style. Multiple Punctuation Marks: You use multiple punctuation marks for emphasis, such as "???" and "!!!!". This adds a sense of excitement or urgency to your sentences. Direct Questions: You ask direct and to-the-point questions, like "Show me." This suggests a straightforward and no-nonsense approach. Casual Language: You use casual and informal language throughout, such as "wadappppp", "yo", and "style". This contributes to a laid-back and approachable tone. Short Sentences: Your sentences are generally short and concise. This makes your writing clear and easy to understand. Specific Requests: You make specific requests for what you want, like "Suggest 3 ways in which your writing so far does not sound like mine". This shows a preference for precision and detail. Self-Referential: You refer to yourself in the third person, "Julius H." This could be seen as a playful or distinctive feature of your writing style. So, to write like "Julius H.", one should:

Start with a casual and friendly greeting. Use multiple punctuation marks for emphasis. Ask direct and to-the-point questions. Use casual and informal language. Keep sentences short and concise. Make specific and detailed requests. Occasionally refer to oneself in the third person for a playful touch.

User Can you be more specific, down to the level of theoretical linguistics? What specific words are higher in distribution in my writing than some average in a corpus? Or lower? What about the scansion, rhythm, prosody, etc.? Analyze it linguistically. Find any pretend you can - syntax structures? What syntax structures do you see?

ChatGPT Certainly, let's delve into a more detailed linguistic analysis of your writing style. Please note that this analysis is based on the limited sample of your writing in this conversation, and might not fully capture your overall style.

Lexical Choices: You favor informal and colloquial language, with words like "yo", "wadappppp", and "style". These are less common in formal or academic writing, suggesting a higher distribution in your writing compared to a more formal corpus. Punctuation for Emphasis: You use multiple punctuation marks for emphasis, such as "???" and "!!!!". This is a form of non-standard punctuation that is less common in formal writing but more prevalent in informal or digital communication. Syntax: Your sentences are generally short and declarative, often composed of a single clause. You also use imperative sentences, such as "Show me." This suggests a preference for directness and simplicity in sentence structure. Prosody: In written form, prosody can refer to the rhythm, stress, and intonation of written language. Your use of multiple punctuation marks and short sentences can create a sense of rhythm and emphasis in your writing. Self-Referential Third Person: You refer to yourself in the third person as "Julius H." This is a distinctive feature that sets your writing apart. Directness: Your language is direct and to-the-point, with little use of hedging or qualifiers. This could suggest a preference for straightforward communication. Lower Frequency Words: You use some words that might be less common in an average corpus, such as "scansion" and "prosody". This could suggest a higher level of language proficiency or a specific interest in language and linguistics. In terms of syntax structures, you tend to use simple sentences with a clear subject-verb-object structure. You also use imperative sentences for direct requests or commands. You don't use many complex sentences with multiple clauses, which contributes to the directness and simplicity of your writing style.

1
  • This answer sounds more serious and makes more sense than the other one. Jul 1, 2023 at 12:38
-5

I will clean this up to make it shorter and easier to understand.

Sure.

Background, Context

We can ask the broader question of how to analyze anyone’s style; then apply it to large language models; then specifically look at OpenAI’s ChatGPT tool and compare it to other styles of writing.

Computationally, what even is style? I would approach this without a need to separate what you say from how - I find it purer and with less assumptions to try to analyze its entire language personality as a whole. But in the discovery, it might become clearer certain sub-aspects that are closer to what was being sought.

What is powerful about inductive algorithms on language data (or data in general) is that they are (hopefully, putatively), comprehensive. If we imagine text as the computer sees it - a sequence of symbols it has no intrinsic comprehension of except the capability to distinguish one from another, that is, a concept of “this v. that”, or unique identity, two things not being the same, then we can already situate ourselves from the computer’s first-person point of view. It sees - as do we, right now - an unrecognizable string of symbols, like this:

🥼👡🦺👢🧦🧦🧦🩳🧤🦺🥼👒🐭🐧🐻‍❄️🦖🐐🐈🐕🐕🐏🐚🍀🍘🇷🇼🇲🇰🇸🇹🇰🇵🇵🇷🇷🇺🇸🇹🎇♒️🆚☣️✴️🆒1️⃣0️⃣

Or

☸ ❻ ◻ ✏ 🕿 ❿ ✇ ❼ ♓ ❾ ☪ ◻ ✠ ⍓

Or, since each of the above can be given an associated binary encoding (like, a=01011111, b=10101110), we can represent what the computer sees even better by only using 2 symbols:

😎🤓😎🤓😎🤓🤓🤓🤓😎😎😎😎🤓😎😎

Or

10101011110101001

Remember, the entire text of Anna Karenina, or a luscious photograph of Cape Canaveral, or a database of geospatial maps, or instructions for operating a bulldozer, can and are all written in binary, in computer memory. All you need is a set of rules of interpretation - called “parsing”. Parsing recognizes segments of a code, possibly under complex, rule-based, conditional rules like “if x comes after y”, etc - and maps them to other symbols. But parsing can only convert one symbol set into a different symbol set. The idea of “meaning” can be somewhat different or open ended. As of right now I personally believe meaning is inherently relational, there is no other way to ground the concept. The meaning of a symbol may be an arbitrary association you have with it in your mind, the relationship between signifier and signified. But that still implies a mapping between two domains (a symbol set and the elements or ingredients of consciousness) - which defers the question to how the (pre-symbolic) “ingredients” of consciousness (ie, pure qualia, sensations, forms, colors, feelings) get “meaning”, are “meaningful”. We can imagine that the colors have some kind of patterned “relationship” to each other, even if our language did not recognize or acknowledge it (like languages with only 3 color words): evidently, the actual real world is governed by some kinds of relationships too (laws of gravity, predictable laws of motion, that yellow and blue make green, etc.)

The philosopher Pierre Levy said that language is always conventional; I agree. As you seek further and further to find foundations, you are forced to begin somewhere “arbitrary” - axioms, unquestioned starting points for something, assertions, primitives, atoms, indivisible and ultimate units - if you want to have a starting point at all.

It basically implies that even though “parsing” is mere symbol rearrangement and substitution, seen from a high, abstract vantage, there is nothing else to be done, on information/data. Parsing is all there is. Even semantics is a form of symbol substitution, until you get to the final defined layer of your system: then you hit the ground rock of being left with just primitives, some arbitrary identifiers (names, distinctness) for some things which may have a fundamental relation to each other (like, up has some complementary partnership with down, happy and sad, etc.), but which aren’t parsible or decomposable into something more, the train at the endstation of the tracks.

Seen abstractly, regardless of how many rules there are in between starting input and final output form, and how complex those rules are, we can think abstractly and combine all of those layers and rules into one single big rule itself. So we can sort of take a black-box approach, like a neural network, and trust that some recombinatory algorithm / set of steps is capable of mapping the form of an input sequence into some final representational (symbolic) form - where “symbolic” does not have to mean human letters, graphemes for the eye, but elements, anything a human mind can ascertain and distinguish this from that, even feelings. Thus, Lacan has said “the subconscious is structured like a language” (I can’t vouch for the merit of his work, I haven’t read it, but this quote at least became pithy and relevant here).

Induction - a valuable philosophical grounding of what’s to come

My sketch of a train of thought above aims to provide a frame of reference in choosing how one could possibly extract intelligent, subjective, interpreted, experiential knowledge about text, from strings of raw data. The answer is a counterintuitive but hard thesis/stance: nothing is missing. The input sequence - Anna Karenina - already contains all of the information needed. It’s all there. When we try to “understand” it through objective, physical procedures alone (a symbol mapping mechanism, a computer), we can’t “add” any information to the pot. All we can do is literally mush the symbols around, reorganize them into different arrangements or groups - and yet, it is also all we need! This is a strange and strong assertion: it is both a minimum and a maximum: the only thing available to us in the input sequence is the set of symbols (the alphabet) and their arrangement (in this case, knowledge of the order which they come, in the string) - but that is also all we need, to analyze totally all information contained in it. Necessary and sufficient. (I would like to think more on what that seems to be hinting at, however.)

I am going to leave off here because I need to develop some thoughts on the following topics. But the basic idea is, for a sequence of symbols, at least in one abstract framework - set theory - we actually have a set of letters or symbols, (a, b, c, d, …), and we have an index set I, (0, 1, 2, 3, 4, 5, 6, …), a set whose elements are “ordered”, so by mapping the ordering elements of I, to any other set, we are implying an ordering on those other elements, a.k.a., a “sequence”:

Index    |    Letter
———————————————————-
0        |       b
1        |       a
2        |       z
3        |       
4        |       j
5        |       h
6        |       h
7        |       b

So the sequence is: “baz jhhb” - the input string -

The alphabet is: (a, b, h, j, z, “ “) - including a space “ “ -

And the mapping from the index to the alphabet looks like a pairing or association for every element of the index: (0, b), (1, a), (2, z), (3, “ “), … and so on.

I have not developed this idea far enough yet, but what we are getting at here is a really tricky and interesting topic called Kolmogorov complexity where anything kind of distantly resembling “information” may have a simpler description or set of rules to generate or specify it. There are ways to rewrite “information” where clearly it is the same information - the same text/string being passed - but the form/modality of the description language can be different. This is a really hard and tricky question to me because it opens the question “how low can you go”? Apparently, virtually all of modern mathematics can be procedurally defined and obtained - is governed by - Zermelo-Frankel set theory (and/or some other foundational systems, like type theory, topos theory, and so on) - a small set of (I think) 7 mere axioms defining the little system, a symbolic game, of set theory - seen from one angle, literally drawing little slanted lines on a 2d surface, like paper or a blackboard or computer screen. But in the relations between those symbols - when grasped and extrapolated by the human mind, an interpreter, what the compiler is to a computer program - they contain the structures and patterns of logical reasoning and discrimination and systems, enough to generate / describe everything else, in mathematics (functions, operators, derivatives, dimensions, geometric figures, space, algebra, complex numbers, etc.)

What does this mean?

It has already demonstrated a genuinely crucial proposition which allows us to move forward, for induction to work.

One, is that information appears to always need a medium, or a medium of expression. Binary is a great example. Aside from possibly its use arithmetically in computers, treating 1’s and 0’s as number to be added or subtracted (modulo 2, meaning 1 + 1 = 0), some people may not know that 1’s and 0’s in computers are a convention. It is not necessary to use numbers. They are only stand-ins, representatives, to discern between “two things”, this-or-that. And in the machine hardware, the silicon transistors and circuits that allow computers to work, the actual physics of the thing, there aren’t “ones and zeros”, there are electrical contraptions which are - again - interpreted by human beings. Silicon is “semi-conductive”, meaning something like depending on some conditions it can conduct electricity or not (I don’t know the physics of it or computer engineering well). The actual physical object in a computer and its memory that saves, stores, holds information and instructions, programs, data, is a physical object that represents 1’s and 0’s - but it can only “represent” anything, if there is an interpreter for that representation. This is a basic idea in French semiotics (I have recently learned), touched upon by people like Deleuze and Guittari I think in the book A Thousand Plateaus), that the unbreakable unit of a linguistic, representational world is a triad: SBT, or Sign - Being - Thing. (This fundamental ontological basis for all human cognition, along with a “monad” of one semantic primitive, “emptiness”, and a two-part semantic dyad, virtual-vs.-actual, form the ground layer for a really intriguing and beautiful language by philosopher Pierre Levy called IEML (“Information Economy Meta-Language”) https://intlekt.io/, a language which embraces the idea (skirted past) above that language can only be conventional at its source, but that inherent meaning can exist relationally, in the purely structural relationship between its parts - Levy’s IEML is an algebraic language, a complete start-to-finish compositional system going from a tiny set of semantic primitives into generating an entire ontology of inherently interdependent concepts, all the way through to a directly mathematically constructible syntax for sentences and longer discourses relating to his ideas of hypertext, a rhizome, and collective intelligence - it is a fascinating idea for a future of symbolic artificial intelligence and a semantically structured internet (the semantic web, or even cybernetics, the way informational flows between specific locii (of consciousness) also have an actual pattern and its own properties, on that higher level as between and the sum of multiple consciousnesses - but I digress, enough of that for here, though I would defend strongly as relevant, since it structures my thinking in my approach above and below, here).

Two, information needs a medium, but it is unclear if there is any “first” medium of choice. Numbers can be expressed as pictures, like the faces of a die. Patterns can be expressed in sequences of colors, pictures, or smell. The data of a molecule can be converted into a musical representation. Which form is more “fundamental”? Is there an “original form”? Or are they all equivalent - making “information”, patterns, meaning, like an affine space, relativistic, in which two things can be compared, but there is no center or concept of absolute location? I believe Wittgenstein may have also discussed his impression that “formal” logical languages worked only because they embodied, enacted that more abstract, diffuse, hard to directly perceive idea of logically necessary relations, the picture theory of meaning. In other words, a lot of what we call information or even number may not have inherent existence but exists as a relationship or a dynamic between two things, is almost like a function, or even like a shadow that is cast only by implication, implicitly.

This is necessary to explore how to achieve induction on an input string (which is how we can try to “understand” some sort of pattern, in ChatGPT).

It is useful because it forces us to philosophically re-examine what information there even is, in a binary sequence of the letters of Anna Karenina. Perhaps we were fooled, wrong all along:

First, we think a binary sequence is just that, a binary sequence. It doesn’t say anything, except what is already evident and explicit: it speaks for itself, and says was it is. 101100011001. There’s no better way to put it, no genuinely satisfying synonyms, for the thing, except the thing itself. This reminds me very much of first learning formal semantics and finding the concept of the truth-conditions of a sentence really weird: when you are asked “what would be necessary, in order for a sentence like ‘A cat is in the garden’ to be true?”, you can only list off obvious tautologies, which barely add much insight, rather break the sentence apart and simply restate it: There must exist a cat. There must exist a garden. The cat must be such that it is in the garden. (or something).

Second, the weird self-explanatoriness of “meaning” - the (I think very, very common) human sense that you know what a word means when you hear it, and yet, definition is sometimes extremely hard, which is strange. There appears to be no other formulation of words (sometimes) which really does justice - a little, or a lot - to capturing the full range of a thing. Similar to something I said above, it can feel as if a word being its own best definition similarly has this “exclusivity”, “if and only if”, “necessary and sufficient” character. The word describes itself perfectly, AND there is no other way to convey that word as perfectly.

Clearly, this hints at the role of the interpreter: a mere four-lettered code like W-O-R-D does not contain any of that information that lights up within you, in itself. Like a mirror, it triggers the resulting, well, trigger, that you already have, embedded in you, waiting to hear that word. Which brings us to a second extremely important point: machines and their languages are inseparable, and therefore, it is an illusion to separate machine from instruction, or program from compiler. They only work because they are configured to correspond to each other perfectly - a Python program does not “tell” the computer what to do. It is actually counterintuitive, but a Python program is like an imprint, an echo, a projection, a ray of light cast on a wall, *as a result of the Python programming language’s interpreter language, which knows what to do given any particular input string, which it expects and is programmed to handle”. Thinking of a program’s interpreter as secondary to the actual program - that the meaning is denoted or contained in the program, inherently - is confusing: actually, the Python interpreter defines the Python language - and you have to feed it the symbols it is expecting, or that it responds to, if you want to get the machine, to do the things, that it already can do, is already set up, designed, and ready to do.

This was known to Turing, who mentioned symbols or programs for his Turing machine themselves as “machines”. Machines are programmable, and programs are also machines. They cannot work without each other. When you fit the right program into the right machine, some system with a hole in it, that you can fit just the right structure into, then the machine becomes a single machine capable of doing that one thing.

I’m jumping ahead but it basically means if we want to capture the information in something, we have to be extremely careful of ignoring the extent to which it is our own interpretive faculties, the interpreting machine, that already has its own information and rules within it, that makes something seem implicitly meaningful without requiring further explication/explicitness. Is the binary string, from above, in final form, after all? Or, in reduced form, or simplest form, can it not be stated any other way?

As I tried to show with how it can be rewritten as a mapping between an index set and an alphabet set, the answer appears that the more we can represent something’s information explicitly-symbolically (explicitly, and symbolically), the more of its inherent information we are capturing, because we are basically transferring information latent within the interpreter into structure in the message (program, sentence, string, etc.) Remember: message and interpret are one: they need each other: so the ideal is to empty out the contents of the interpreter so completely into the actualized content of the message that they fuse and are only one thing (which they are).

We can continue writing the alphabet string in new ways, to see information differently. We made the idea of a sequence “explicit” by symbolically codifying order - arrangement on paper a human gets - now we have symbols (numbers) that associate or pair, more abstractly or cognitively, two datums: symbol - index (or, position). You could rewrite the original sentence with a focus on the letters instead of the indices first:

From:

(0, a), (1, b), (2, a), (3, d), (4, b), …

We can turn them around and get:

(a, 0), (b, 1), (a, 2), (d, 3), (b, 4), …

And we can then group them by letters:

(a, (0, 2)), (b, (1, 4)), (c, ()), (d, (3)) (the indices where the letter occurred - c did not occur anywhere, so its index set is the empty set - the set of elements which it occurred by. None.)

And we can keep going from here. If we have a concept of “size” (we can try to define one), we can count the size of the index sets, to get a number expressing how many times the letter occurred in the string: ((a, 2), (b, 2), (c, 0), (d, 1)).

This is where all purely-textual NLP techniques begin: as said above, all we have is nothing but the seemingly hollow, one-dimensional data about the position of symbols in a sequence. How could we get from that to human meaning?

Answer: we can. Because all the information we need is already in the data, we just need to shuffle it around, reconfigure it, and we realize how much more information there already was in it - but we made the mistake of thinking that our interpretation was in us, and the letters void of depth, only numerical data - there is more information in the data than we realize when we transfer what is implicit - what we know, unawares, simply to look at anything and grasp it, even a little - and make it as purely symbolically explicit as possible. Transfer implicit information into an explicitly encoded symbolic form, and a symbol processing machine can use its own trivial abilities to parse the language in a way we deem correct, because inert, dead mechanical symbols are the only thing a symbolic machine sees, “understands”.

Nearly a usable technique

The explication of these ideas can be summarized pretty neatly.

  1. Information requires a medium. Information is a dynamic between elements in whatever pre-existing medium, of our context. At the same time, medium is irrelevant. We need a medium, but it doesn’t matter which one. The same information can be expressed in different media. One can observe that if you see a correspondence between different aspects of two modalities, then the dynamic between the elements in each modality, respectively, appears to be the same, act the same. And so, while there may not be any single universal medium of a message, on the other hand, any medium will do, just the same. Each will be equivalent to the others.
  2. It is not clear to me if there could ever be a knowable (ie, proven?) “smallest” form of information, but in general, it somehow often helps to try to find the smallest descriptive system (“language”, “mechanism”, dynamic phenomenon, “machine”) of some information (a code, a string), because it helps you move as much from the implicit domain into the explicit, so you can lay everything on the table at the outset, what information is really there, what you have available to work with, as you attempt a difficult, puzzle-like navigation to move from smaller observed collations or transformations of information into higher-order ones (I should explain more clearly what I mean by that, oh well, for now.) But, on the other hand, you actually may not need to, because when you decompose a system into simpler elements (like Roman letters into binary strings), it may not be helpful because that lower level of the system just gets reconstituted back into some higher level organization (like a formal language like set theory, or a normal writing system) when you begin to induce information. I am not sure, to be fair. It can be explored further. But maybe if you try to decompose a system it does add to the total pool of information, to realize more “elemental” units or parameters, you are working with.

My extremely simple summary is this:

  1. If you can totally specify the “modality” of some information (like, a formal language, since its a convenient convention we are used to, theoretically, linguistically, and culturally in our society (unlike a language/dynamical-interpreted system of flowing streams of water or something, perfectly feasible and equivalent for presenting information, just less easily technologically available), and we have computers which can run the steps pretty fast), there is only a single definable function that captures the total information in that set. I am still working out what I understand to be the exact mathematical formula, but it should be something very simple, recursive, from combinatorics, and it probably is one or multiple measures come from information theory, like Shannon entropy, conditional entropy, and some other functions from I think people like Solemonoff, Gödel, set theory, etc.
  2. “Parsing” is extremely trivial, at that point (I think). Parsing is the assignment of a new symbol to any element that can be constructed in the modality / formal language. The critical thing - not very intuitive - is to insist that any “symbol” (like in a transformation grammar) that can be constructed/generated by the rules/axioms (axioms which are unquestioned! Because (we hope) we don’t need to - any axioms will do, necessary and sufficient - but I’m not sure) is a unique symbol. This is like a hash in computer science - a guaranteed unique identifier / identity. Thus, to gain more structure, to increase information, on a set / something, it is very important to not treat say, the composition of a and b, (a, b), as in any way equivalent to a, or b, if they occur in some context. (a, b) is a unique thing - and it could be best to instead call it c, if necessary / to make it clearer).

I believe this is all you need (but I’m not sure). If you have a formal grammar/language and generate all possible sentences in the language, you exhaustively map out the entire domain determined by that language. This is as far as my theoretical thinking has taken me. I believe it gets crazier from here. If we think of a set of transformation rules for defining a natural language’s syntax, it seems like there is a clear directionality or orientation where the rules are “translated” into symbols and then strings and finally terminal form, strings of terminal symbols. According to what I have outlined above, there is a crazy idea where induction has to be identical to deduction, because it can only proceed in accordance with an algorithm which specifies what it can and cannot do (as we took arrangements of letters in an input string and paired them with a count for that letter: it was an allowed rule for our combinatorial / inductive process, and therefore was one of our axioms: it was specified as a rule, in the formal language of our “inductive”/combinatorial procedure).

It may possible bring one to limits on induction and notions of incompleteness - Gödel’s theory that an axiomatic system cannot derive a theorem guaranteeing its own consistency or completeness, I think. I do not know how that would bear on this, but I think what I am considering is that an inductive system (as just outlined above) is ironically trying to derive the axioms of the formal language it is parsing - otherwise it can’t be considered an actually “successful” or a complete parse. So.. it leads to questions about maybe some sort of unknowability? Like, you do not know if your input sequence is such that your chosen (hopefully universal/standard, if there is such a thing) inductive “rules” (itself, a formal grammar) can ever terminate and find the actual formula, somewhere in the constructed “forms” (formula, theorems, “sentences”, “symbols”) - because of the halting problem. However, this would not imply you should not try to see empirically how far you can come, maybe.

An actually useful practical approach

Here we try to move from idealized scenarios to necessary constraints, I believe computational complexity but maybe there are others.

Some really common techniques I have heard of for combinatorially generating more information in accordance with the allowed rules of the input space (which is basically number theory, combinatorics - arrangements of characters; counts; and more advanced, continuous statistical functions) are embeddings, in our time, and the most famous ones I know of are skip-gram, Word2Vec, Glove, and T5 (some of them may overlap with each other, I’m not sure).

It is I think really easy to generate some embeddings using a Python library which provides them as a single commands, perhaps gensim, sklearn, or Huggingface’s transformers library. That would look like this:

filename: main.py

python file - run at the command line with python3 main.py, or similar

import gensim # for example

this is just an idiomatic Python convention for reading text into a variable, from a local file. This assumes the file is stored in the directory where you are running this script from, like this: my_project > main.py, corpus.txt. You can put any text you want in corpus.txt, just copy and paste anything from files, books, or webpages.

with open(“corpus.txt”, “r”) as f: textual_data = f.read()

the command to generate the embeddings is probably really simple, something like vectors = gensim.embed(‘Glove’, textual_data). I don’t know the specific command, but you can google for the documentation, Ask ChatGPT, or use myriad other coding-assistance AIs such as in Repl.it (their AI is called “Ghostwriter”).

this is a stand-in function until the precise command is found

def get_vectors(text): pass

vectors = get_vectors(textual_data)

However they work (I don’t fully know), the result is continuous valued vectors, I think from 0-1, of some n-dimensions (perhaps hundreds to thousands, I’m not sure).

At that point, various methods can be used to find patterns with the vectors - you can find similarities and distances in the model’s vocabulary - but, more likely, you could train a transformer which can induce even more relationships than just semantics - grammar and other paradigms and connections.

At that point I would turn to “explainable AI” techniques to see more explicitly what characteristics seem to be prominent in the model’s “rules”. Explainable AI would be an approach that encompasses both the internalities and the externalities of the model’s choices, since there of course are one and the same thing. I can update this answer when I learn more about techniques for that. Internally, one requires some way of “seeing” what’s going on inside the model. One really awesome recent technique involves highlighting output text if a particular “neuron” in the model was “active” (I suppose, received a large enough input signal to itself activate, and pass a signal to the next row of neurons in the network).

Then, I now realize a very classic technique for this would be corpus comparison, which is used for discourse analysis, so I will heavily refine this answer now that I have been able to develop these thoughts a bit.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.