“What lies at the heart of every living thing is not a fire, not warm breath, not a ‘spark of life.’ It is information, words, instructions… If you want to understand life, don’t think about vibrant, throbbing gels and oozes, think about information technology.”
Richard Dawkins
Introduction
What is information, and how does it get around?
This post started as an excuse to explore this high-level question and gain a more comprehensive understanding of the expansive concept we call information.
Over the course of writing this post, I went down many rabbit holes, emerging some hours later, dazed and confused but full of wonder. Perspectives on this question ricochet from the most abstract analyses of the philosophy of information to the most mathematical and scientific studies in the bowels of information theory or science. The first part of this question evokes strong responses from the camps of philosophy, metaphysics, mathematics, physics, biology, computer science, and art. “What is information?” simultaneously unites and divides all those who seek answers or to answer it.
It was a delightful topic to write about, what with all those opinions bouncing around at all these intersections. The process was full of opportunities to challenge my assumptions and expand my understanding of the world.
The intent of this post is to communicate the breadth and depth implied by this question. And hopefully, to instill a bit of wonder about the world.
What is Information?
Simple question, right?
At least that’s what I first thought.
To start exploring this question, I’ll cover how the concept of information has been acknowledged by science and information theory. Then I’ll provide a brief overview of information theory to a limited extent, and finally, present some parting thoughts on the question.
Why cover science and then information theory?
Increasingly, the physicists and the information theorists are one and the same. The bit is a fundamental particle of a different sort: not just tiny but abstract—a binary digit, a flip-flop, a yes-or-no. It is insubstantial, yet as scientists finally come to understand information, they wonder whether it may be primary: more fundamental than matter itself. They suggest that the bit is the irreducible kernel and that information forms the very core of existence.
Gleick, James. The Information: A History, a Theory, a Flood (pp. 9-10). Knopf Doubleday Publishing Group. Kindle Edition.
Science!
In 1929, Leo Szilard addressed a paradox that had plagued physicists for half a century. He posited that information could be dispensed in small units (bits) that could counterbalance entropy (simply put, entropy is disorder). In his thought experiment, a new and proverbial solar system of thought was created. (Learn more about the paradox and his thought experiment here.)
In sum – information brings organization and order; it is the counterpart to disorder.
Information, in its connotation in physics, is a measure of order—a universal measure applicable to any structure, any system. It quantifies the instructions that are needed to produce a certain organization. This sense of the word is not too far from the one it once had in old Latin. Informare meant to “form,” to “shape,” to “organize.”
This information flow, not energy per se, is the prime mover of life—that molecular information flowing in circles brings forth the organization we call “organism” and maintains it against the ever-present disorganizing pressures in the physics universe. So viewed, the information circle becomes the unit of life.
Loewenstein, Werner R.. The Touchstone of Life (pp. xv-xvi). Oxford University Press. Kindle Edition.
How much information is needed to counterbalance disorder?
The more disorder, the more information is needed to bring order. Any element/system that can be reproduced in many (equivalent) ways is perceived as disorderly and requires more information to bring about order.
Over time, systems shed their information (order) and maximize their entropy (disorder). This phenomenon, while rooted in physics, is known in software engineering; it’s called software entropy. Unless maintainers and builders actively work against the buildup of disorder, as the system is modified and added to over time, the disorder will increase. How can we prevent this? Some folks suggest fixing metaphorical broken windows as an effective preventative measure.
So that’s how information, as a concept, came into being, at least in physics.
Science, out.
Information Theory
This section will, at an extremely high level, provide a brief history of the concept of information in the realm of information theory, and then outline the basics of information theory.
The Concept of “Information”
This field had a promising start in the 1920s when telegraphs were booming (not literally, of course), and communications infrastructure was rapidly expanding. Due to the ever-increasing importance of reliable transcontinental telegraphs, many aspects of that communication system came under theoretical and applied scrutiny during the telegraph boom. When there was no established field of information theory, these topics were studied at a messy/amazing/unsustainable intersection of engineering, mathematics, and “communication systems.”
Here’s an abridged history:
- 1924: Harry Nyquist provides a distinction between the actual content of the signal and the information carried within the message. He begins to discuss how a system could be optimized for the transmission of intelligence (his words, not mine). As a part of that, he realized that communication channels have a certain transmission maximum. He didn’t talk about information – just intelligence.
- 1928: R.V.L. Hartley builds upon Nyquists’ ideas by removing some of the more interpretive/subjective elements (namely, the concern over meaning). He develops mathematical proofs for measuring the flow of intelligence. In his own words, he hoped ” to accomplish…. a quantitative measure whereby the capacities of various systems to transmit information may be compared.” Source.
- 1948: Claude Shannon, widely regarded as the father of information theory, cites Nyquist’s and Hartley’s papers in his groundbreaking paper, Mathematical Theory of Communication. More below. Note: Shannon made the jump from intelligence to information.
There are two parts to Shannon’s work:
- Modeling the conceptualization of information and information sources and working from these models:
- Developing theories on the sending of information across the channel, the limits of the amount of information, and noise.
I touch on Shannon’s work a bit more in a future section of this post, but if you want to know all about it, I recommend reading Shannon’s actual paper, this useful summary of his life from the Scientific American, or just giving him a good ole’ Google.
Shannon’s work allowed all communication systems – radio, television, telegraph, etc. – to be unified under one model with common characteristics and problems. Although “Shannon’s model does not cover all aspects of a communication system… in order to develop a precise and useful theory of information, the scope of the theory has [sic] to be restricted” (Source).
Information theory was born.
The (Very) Basics of Information Theory
Information theory is a mathematical representation of the conditions and parameters affecting the transmission and processing of information (Encyclopaedia Brittanica).
Information theory deals with three basic concepts:
(a) the measure of source information (the rate at which the source generates the information),
(b) the information capacity of a channel (the maximum rate at which reliable transmission of information is possible over a given channel with an arbitrarily small error), and
(c) coding (a scheme for efficient utilization of the channel capacity for information transfer). These three concepts are tied together through a series of theorems that form the basis of information theory summarized as follows:
If the rate of information from a message-producing source does not exceed the capacity of the communication channel under consideration, then there exists a coding technique such that the information can be sent over the channel with an arbitrarily small frequency of errors, despite the presence of undesirable noise.
Information Theory, Coding and Cryptography by Arijit Saha, Nilotpal Manna, Mandal
That’s dense, so let’s break that down.
A) Measure of source information
- “The rate at which the source generates the information.”
- This is how many envelopes the source produces or how much information is in the message.
B) Information capacity of a channel
- “The maximum rate at which reliable transmission of information is possible over a given channel with an arbitrarily small error.”
- How many envelopes can fit into that channel, the speed of the envelope moving through the channel, and the tolerance for small errors in envelope sealing (just an example of an error type)
C) Coding
- “A scheme for efficiently using the communication channel’s capacity.”
- The envelope itself, the methodology of fitting the message in the envelope, the way information is encoded as a message, etc.
Just a bit more about Shannon’s paper
Shannon also introduced two other concepts about information in the context of a communication system:
- Information is uncertainty. “More specifically, if a piece of information we are interested in is deterministic, then it has no value at all because it is already known with no uncertainty. From this point of view…. the continuous transmission of a still picture on a television broadcast channel is superfluous. Consequently, an information source is naturally modeled as a random variable or a random process, and probability is employed to develop the theory of information” (Source).
- Information to be transmitted is digital. “This means that the information source should first be converted into a stream of 0’s and 1’s called bits,and the remaining task is to deliver these bits to the receiver correctly with no reference to their actual meaning” (Source).
And he proved two theorems, which are highly related to the a), b), and c) above.
1. The source coding theorem introduces entropy as the fundamental measure of information which characterizes the minimum rate of a source code representing an information source essentially free of error. The source coding theorem is the theoretical basis for lossless data compression.
2. The second theorem, called the channel coding theorem, concerns communication through a noisy channel. It was shown that associated with every noisy channel is a parameter, called the capacity, which is strictly positive except for very special channels, such that information can be communicated reliably through the channel as long as the information rate is less than the capacity. These two theorems, which give fundamental limits in point-to-point communication, are the two most important results in information theory.
Information Theory and Network Coding by Raymond W. Yeung
To learn more, I highly recommend picking up the book cited above.
So…
What is information?
We’ve covered how “information” was discovered as a concept in both science and information theory, how information theory was established, and what concepts information theory touches. But my original question still stands.
Well. Shannon was certainly cautious about answering this question. Here’s what he had to say about it in the late 1940s.
The word ‘information’ has been given different meanings by various writers in the general field of information theory. It is likely that at least a number of these will prove sufficiently useful in certain applications to deserve further study and permanent recognition.
It is hardly to be expected that a single concept of information would satisfactorily account for the numerous possible applications of this general field.
The Lattice Theory of Information by C. Shannon
It’s an eloquent way of saying:
But hey – that was in 1940-something. Things have probably changed, right?
Work on the concept of information is still at that lamentable stage when disagreement affects even the way in which the problems themselves are provisionally phrased and framed.
Information: A Very Short Introduction, Luciano Floridi, 2010
And:
What is information? The question has received many answers in different fields. Unsurprisingly, several surveys do not even converge on a single, unified definition of information (see for example Braman [1989], Losee [1997], Machlup and Mansfield [1983], Debons and Cameron [1975], Larson and Debons [1983]).
Source: https://plato.stanford.edu/entries/information-semantic/#2
It’s time for a bit of a leap of faith.
I’m going to make an assumption…
…That whoever is reading this wants a f**king answer to this question, if you’ve made it this far.
We’re going to branch down from “information” to “data”, and explain it from the perspective of the “semantic” philosophical theory. I don’t really know what that means either, but assume that lots of people agree and disagree with the direction I’m taking this, and it’s not a black & white answer.
The General Definition of Information (GDI)
This is a controversial and highly subjective answer to a seemingly simple question.
The General Definition of Information (GDI) in terms of data + meaning. Various fields have adopted the GDI, generally, those that consider data and information to be more concrete entities, e.g., information science.
The General Definition of Information (GDI):
x is an instance of information, understood as semantic content, if and only if:(GDI.1) x consists of one or more data;
(GDI.2) the data in x are well-formed;
(GDI.3) the well-formed data in x are meaningful.
Source: https://plato.stanford.edu/entries/information-semantic/#1
Let’s break down each of the words in italics.
Data
This gets weird and metaphysical, real fast.
Again, sticking to the GDI’s accepted definition a singular data point (a datum): a datum is a fact regarding some difference or lack of uniformity within some context.
For example, the top-level domain (TLD) of my website is .dev. True story. This is a datum (fact) about my website (context). There are many top-level domains, but even not knowing that fact, the existence of .dev suggests there may be non-.dev possibilities.
There’s loads more to the definition, but I’m not going to go into it here. If you want to learn more, this is the place to go.
Well-Formed
This means that the data are clustered together correctly, according to the rules (syntax) that govern the chosen system, code, or language. Syntax is what determines the form, construction, composition, or structuring of something.
For example, in a tree graph, a parent node will always appear above a child node. A child node will always appear below the parent node. This is syntax.
Meaningful
The data adhere to the semantics (meaning) of a system.
In the graph, we understand that the child nodes are sub-elements of the parent node. That relationship is due to one or multiple shared characteristics, functions, or properties. This is the semantic structure of a tree graph.
Becoming Information
“We can see now that information is what our world runs on: the blood and the fuel, the vital principle. It pervades the sciences from top to bottom, transforming every branch of knowledge. Information theory began as a bridge from mathematics to electrical engineering and from there to computing. What English speakers call “computer science” Europeans have known as informatique , informatica , and Informatik.”
The Information: A History, a Theory, a Flood by James Gleick
Put one or more well-formed and meaningful datum together, and what have you got?
I can communicate something like this – a rudimentary diagram of the Domain Name System namespace! Information!
Final Thoughts
The Information Lifecycle (The Circle of Life)
Once information (data + meaning) could be said to exist, people or systems need to gain access to it via communication systems. Communication systems shape and are shaped by the information lifecycle. But the communication system is just part of the lifecycle of information:
The information lifecycle and the contents of this diagram deserves a post of its own. In essence, we’ve only really looked at a portion of this cycle.
The Information Galaxy
As I read and wrote more about this topic, the image I kept coming back to was one of a galaxy. While the metaphor to astronomy is, so to speak, quite vast; this was more focused on the mindset I adopted while exploring this topic.
We know things about the galaxy. We don’t know things about the galaxy. There are observed phenomena and rules created to explain them. And there are observed or unobserved phenomena that we haven’t discovered or been able to explain yet. This holds true for the concept of information. We know some things about information and how it gets around. And then we don’t know some things about information and how it gets around. There are unsolved puzzles and undiscovered frontiers.
When considering information, we encounter the far reaches of human knowledge and understanding. Information binds us together and creates boundaries that separate in equal measure.
One thing is for sure, there’s always more to explore.
Resources
These resources are mostly all linked throughout this post, but some of these are more so additional reading for the curious or some of my favorite materials about the topic.
- Information Theory and Network Coding 2nd Ed. by Raymond W. Yeung
- Stanford Encylopedia of Philosophy, Information
- Information Theory, Coding and Cryptography by Arijit Saha, Nilotpal Manna, Mandal
- Information: A Theory, A History, and Flood James Gleick
- Information: A Very Brief Introduction by Luciano Floridi
- The Touchstone of Life, Werner R. Loewenstein
Chloe Tucker 