Levels Of Science - The Technology Level

Levels Of Science - The Technology Level

If you haven't yet read "Tantalus on the Road to Asymptotia", Ed Leamer's recent essay, and if you're at all interested in statistics, empirical economics, or science in general, then you should go read it. The essay is primarily a reply to an extremely important 2010 discussion paper by Joshua Angrist and Jorn-Steffen Pischke, called "The Credibility Revolution in Empirical Economics". That paper in turn is mainly a response to a 1983 Leamer essay called "Let's Take the Con Out of Econometrics". Such are the time scales over which deep academic debates are conducted. Actually, you should read all three.
 
Tech Levels are a convenient abstract used to categorize technology according to relative complexity and sophistication. They are based on the idea that looking back, history can be divided into distinct technological eras (stone age, bronze age, iron age, middle ages, renaissance, etc.) Each era can be considered rungs on a ladder, or a distinct level, that must be reached before the next one becomes possible.

On this site, we take that basic concept and project it into the future, anticipating distinct technological eras that have yet to come based on the cumulative best guesses of scientists, engineers, and various writers. Technological innovations are put on this numeric scale that represents when they first enter widespread use (ie, out of the experimental and testing phases.) The number represents a historical era, past or future, and its accompanying overall level of development.
The tech level scale used on this site ranges from 1 to 25+, where 1 through 10 represents real historical eras and 11+ represents future ages. Categorizing innovations by tech levels is NOT a precision science and represents only best guesses as to when an innovation will come on-line.
Also, this scale is by no means a smooth curve. Tech Level 2 represents a period of roughly 4500 years, while tech level 9 represents barely 1/100^th that. Tech level 11 represents a mere 10 years, and the higher numbers (20+) represent leaps of many millennia. Each level represents a broad number of various innovations, and the scale only shoots up to the next level when a large number of breakthroughs significantly changes both society and the technological landscape as a whole.
For most innovations, especially near-future ones (Tech Levels 11 through 15), I try to be as conservative as reasonable in guessing when they may emerge, factoring both prevailing cultural and economic factors as well as technological ones. For example, even though it may be possible that someone will discover the means for an FTL drive within the next ten years, it will much more likely be many centuries (if ever) before we beat a beam of light to another star. Thus FTL travel has a tech level of 16+, an indication of many centuries of advancement, as opposed to the ten-plus years in the furture tech level 11 represents.
A brief breakdown of Tech Levels by group and what they mean:
Tech Level 0
This represents no technology.
Tech Level 1
This represents the slow climb of technology in Prehistory, from the time our pre-homo-sapiens ancestors first picked up a stick to dig for grubs until advancements in agriculture allowed for the formation of cities and recorded history.
Tech Levels 2-9
These represent actual historical eras (see chart)
Tech Level 10
This represents modern day Earth, today.
Tech Levels 11-15
These levels represent the near future, from tomorrow until about 100 years from now. Usual estimates for this period include steady refinements in electronic and computer technology, ever-increasing global connectiveness and travel, the advent of "base" cybernetics and genetic engineering, and a slow but steadily increasing human presence in space.
Tech Levels 16-20
This is the "far" future, representing up to 1000 or more years beyond Tech Level 15. These are the tech levels of most mainstream science fiction stories and tropes, such as starships, interstellar colonies, intelligent robots, death rays, alien contact, etc, etc.
Tech Levels 21-25
These levels represent the frontiers of the easily imaginable future, a great many thousands of years beyond today. Humankind opens up the secrets of the universe and creates ultra-sophisticated technology whose feats would seem to border on the magical to us today. Wormholes may connect distant parts of the galaxy. Black holes may be tapped for energy. Involuntary death may be unknown.
Tech Levels 26 and Beyond
Levels beyond 25 represent god-like technologies in power and reach. Cosmos-spanning networks, reality-altering machines, pocket universes, and more mind-boggling concepts inhabit these levels.

---

TECH LEVEL TABLE

0 No Tech
1 Prehistory, aka Stone Age (5000 BC and Before)
2 Early City-States (5000 BC to 500 BC)
3 Iron Age (500 BC to AD 500)
4 Middle Ages (AD 500 to AD 1450)
5 Rennaisance (AD 1450 to AD 1700)
6 Age of Reason (1700 to 1850)
7 Victorian Era (1850 to 1900)
8 The World Wars (1901 to 1945)
9 Cold War Era (1946 to 1991)
10 Modern-Day Earth (1992-present)
11 circa +10 years
12 circa +25 years
13 circa +50 years
14 circa +75 years
15 circa +100 years
16 Low Far Future Technology
20 High Far Future Technology
21 Low "Ultra" Technology
25 High "Ultra" Technology
26+ god-Like Technologies.

The more recent two essays are discussing the idea of "natural experiments", and to what degree these make empirical economic studies (econometrics) more reliable. This is a very deep question about science. Normally, statistics can only see correlation, not causation; for example, you see that every time roosters crow, the sun comes up shortly afterward, but this doesn't tell you which caused which. A "natural experiment" would be if, for example, some disease killed all the roosters in town. Seeing that even without roosters, the sun still came up, you could conclude that rooster crowing (or, at least, rooster crowing in this specific town) was not necessary to summon the sun.

This natural experiment is very similar to a lab experiment. In fact, how is it different? Well, you might say, a lab experiment is controlled, and a natural experiment is not; in a lab, you can make sure outside stuff isn't disturbing your setup, while in a natural experiment you can't. This, in fact, is Ed Leamer's critique of natural experiments.

But I'm not sure that's right. In a lab experiment, we only convince ourselves that we've excluded all the outside causes. But sometimes, stuff that we didn't think about is messing with our experiment - cosmic rays, or the composition of the air, etc. Sure, lab experiments tend to exclude a lot more causes than natural experiments, but this need not be the case. For example, in finance experiments, even if you use an incredibly simple asset-market setup, subjects' behavior may be distorted by their pre-existing beliefs about the real-world stock market.

As I see it, the biggest advantage of lab experiments is that you can do them many times. You just can't do that with natural experiments. First of all, that allows you to control for a lot more things, since any confounding influence would have to be constant over space and time. Second, you can generate as much data as you want, making small-sample problems (another problem noted by Leamer) irrelevant. And third, it allows you to vary the setup intentionally, exploring the scope of an effect or a theory, and gaining a more complete picture of how the thing works. In other words, in even a perfectly designed natural experiment, we don't get to choose the questions we ask the world. In the lab, we do.

Ed Leamer focuses on the effect of confounding influences in natural experiments. He suggests doing sensitivity analysis to find what assumptions and specifications you need for a result to hold. I think that's a good idea. Basically, it's a way of groping around for something that looks like a set of scope conditions - testing the hypothesis to failure, to get a slightly better idea of when your theory works and when it doesn't. (Of course, if the natural experiment were a lab experiment, you could do this infinitely better, but if wishes were horses...)

As I see it, there are basically four "levels" of science. Each level gives you more confidence in your understanding of the world (i.e., in your theories and models). The levels are:

Level 1: History
This is basically just establishing precedents. It helps you define the set of things that can happen. Imagine a world without writing, and you'll see how important history is.

Level 2: Non-causal statistics
This is basically hunting for correlations. It can help you generate some guesses and ideas about what might cause what. It can also throw cold water on existing theories, since if A causes B, then we should probably see some kind of correlation, however variable or out-of-order, between A and B.

Level 3: Natural experiments
This is when you have some sort of randomized variation, but no ability to control the environment. An ideal natural experiment lets you establish that a causal effect occurred, but it's very hard to tell whether the setting was ideal or confounded, and you get only a limited amount of data.

Level 4: Lab experiments
By allowing replication and control of the environment, lab experiments usually produce more convincing  conclusions about causal effects, generate as much data as you want, and allow you to explore the scope of the scope of the effects you find (i.e. when they do and don't happen).

If we could always understand the world through lab experiments, we would. When we can't put things in a lab - like the macroeconomy, or the Milky Way galaxy - then we should look for natural experiments. But if we can't find sources of random variation, then we should at least look for correlations. And if we don't have reliable quantitative data, the best we can do is just write down what we see.


Update: Noah receives a partial smackdown from...Noah's dad! The father is not satisfied with my one-dimensional classification of research methods, and wants to bring external validity into the picture:

Briefly, research methods vary on two important dimensions, one we can call internal validity (how sure are we that we know what caused our results?), and the other ecological validity (do our observations relate to the real world?). Only the experimental method can logically show cause-and-effect, so it is highest in internal validity, but the artificial situations created by controlling so many factors make it low in ecological validity (also, experiments can be flawed in many ways, such as poor methods, restriction in the range of observations, confounding factors we didn't think about, etc., which is why replication and attempts to falsify claims are intrinsically important to experimental science). Naturalistic observation is highest in ecological validity, lowest in internal validity. Other methods, such as correlation, ex post facto "experiments" (aka, "natural" experiments), and case studies are in-between on both dimensions.  

Even experiments can vary on these two dimensions, some tightly controlled and measured, some using more naturalistic real-world manipulations and more complex settings in which many factors can interact. The ideal situation is one in which experiments at both ends of this continuum show the same thing, thereby bolstering internal and ecological validity. I refer to this approach as "alignment" in a research area, which helps tie real-world phenomena to causes. This ties, for example, my highly controlled lab research on creative cognitive processes (like fixation or incubation) to more naturalistic research with design students, and to research with real designers with real jobs.

Well, there you have it. Note that I was originally trained as a physicist, and in physics, the Principle of Superposition assures you that any conclusion with internal validity will have external validity as well (i.e., the real-world motion of objects is just assumed to be caused by a straightforward combination of things that you can observe in labs). This is less so in other sciences, and much less so in social sciences like psychology and econ.