Alex Pang's Future2: The Future of Futures

Nassim Taleb on signal, noise, and the toxicity of data

Nassim Taleb has a short but very worthwhile piece on the Farnam Street Blog about signal and noise, and how thanks to always-on connectivity and real-time data we tend to consume a lot more of the second than the first. The big idea:

In business and economic decision-making, data causes severe side effects —data is now plentiful thanks to connectivity; and the share of spuriousness in the data increases as one gets more immersed into it. A not well discussed property of data: it is toxic in large quantities —even in moderate quantities.

Our real products were the experts, not the papers

A very interesting article by Gregory Treverton (who I know from his distinction between puzzles and mysteries in forecasting) asks "What should we expect of our spies?" It's available to subscribers, so you may or may not be able to read the whole thing, but a couple points jumped out at me.

The main question he takes on is,

"What should people expect of their intelligence agencies?" Not "what would they like?"; for policymakers would like perfect prescience, if not omniscience. They know that they can have neither.

The first is his distinction between different types of tasks that intelligence analysts are called upon to undertake, and the danger of confusing one with the other. There are demands for both actionable intelligence or answers to specific questions-- i.e., the answers to puzzles-- and a broader awareness of large trends or potential disruptions-- in other words, mysteries-- and the pursuit of the two don't sit together well.

When the Soviet Union would collapse was a mystery, not a puzzle. No one could know the answer: it depended. It was contingent. Puzzles are a very different kind of intelligence problem. They have an answer, but we may not know it. Many of the intelligence successes of the cold war were puzzle-solving about a very secretive foe: were there Soviet missiles in Cuba? How many warheads did the Soviet SS-18 missile carry?

The second interesting point is that at the National Intelligence Council,

I came to think that, for all the technology, strategic analysis was best done in person. I came to think that our real products weren't those papers, the NIEs. Rather they were the NIOs, the National Intelligence Officers-the experts, not papers. We all think we can absorb information more efficiently by reading, but my advice to my policy colleagues was to give intelligence officers some face time. If policymakers ask for a paper, what they get will inevitably be 60 degrees off the target. In 20 minutes, though, the intelligence officers can sharpen the question, and the policy official can calibrate the expertise of the analyst.

This is a nice reminder how just how much our thinking about the future is a product of our current concerns-- that intelligence "is about creating and adjusting stories," as Treverton puts it, and thus "stories not imagined by policy are not likely to be answered or developed by intelligence"-- and how much knowledge about the future is irreducibly craft and tacit knowledge, not formal knowledge. Sure, you can produce a report, this line of argument goes, but that product will by its nature be misleading.

Rankings of most and least accurate pundits

More of this, please:

A Hamilton College class and their public policy professor analyzed the predicts of 26 pundits — including Sunday morning TV talkers — and used a scale of 1 to 5 to rate their accuracy. After Paul Krugman, the most accurate pundits were Maureen Dowd, former Pennsylvania Governor Ed Rendell, U.S. Senator Chuck Schumer (D-NY), and former House Speaker Nancy Pelosi. “The Bad” list includes Thomas Friedman, Clarence Page, and Bob Herbert....

Hamilton students sampled the predictions of 26 individuals who wrote columns in major print media and who appeared on the three major Sunday news shows – Face the Nation, Meet the Press, and This Week – and evaluated the accuracy of 472 predictions made during the 16-month period. They used a scale of 1 to 5 (1 being “will not happen, 5 being “will absolutely happen”) to rate the accuracy of each, and then divided them into three categories: The Good, The Bad, and The Ugly.

The students found that only nine of the prognosticators they studied could predict more accurately than a coin flip. Two were significantly less accurate, and the remaining 14 were not statistically any better or worse than a coin flip.

The examined spy

That's the title of an article by Thanassis Cambanis about IARPA's efforts to improve the accuracy of forecasting-- and effort that's funding the Good Judgment Project, which I'm involved in.

In a field still deeply shaped by arcane traditions and turf wars, when it comes to assessing what actually works — and which tidbits of information make it into the president’s daily brief — politics and power struggles among the 17 different American intelligence agencies are just as likely as security concerns to rule the day.

What if the intelligence community started to apply the emerging tools of social science to its work? What if it began testing and refining its predictions to determine which of its techniques yield useful information, and which should be discarded? Director of National Intelligence James R. Clapper, a retired Air Force general, has begun to invite this kind of thinking from the heart of the leviathan. He has asked outside experts to assess the intelligence community’s methods; at the same time, the government has begun directing some of its prodigious intelligence budget to academic research to explore pie-in-the-sky approaches to forecasting. All this effort is intended to transform America’s massive data-collection effort into much more accurate analysis and predictions.

“We still don’t really know what works and what doesn’t work,” said Baruch Fischhoff, a behavioral scientist at Carnegie Mellon University. “We say, put it to the test. The stakes are so high, how can you afford not to structure yourself for learning?”... Fischhoff and a who’s who of social scientists from psychology, business, and policy departments hope to foment a similar revolution in the intelligence world. Their most radical suggestion could have far-reaching effects and is already being slowly implemented: systematically judge the success rates of analyst predictions, and figure out which approaches actually work. Is intuition more useful than computer modeling? Is game theory better for some situations, and on-the-ground social analysis more accurate elsewhere?

Fischhoff envisions intelligence agencies, in real time, assigning teams with very different approaches to separately analyze real world situations, like the current state of play in Syria and the wider Arab world. Over the course of the next couple of years, researchers would track the success of different approaches to see which methods work best.

That remains only a proposal so far, but the Intelligence Advanced Research Projects Activity, or IARPA — a two-year-old agency that funds experimental ideas — is already trying a novel way to generate imaginative new steps to make predictions better. It is funding an unusual contest among academic researchers, a forecasting competition that will pit five teams using different methods of prediction against one another.

Of course, one can argue-- and indeed many of my fellow futurists will argue-- about what constitutes a "working" forecast, and some may go so far as to claim that even a completely wrong forecast can be useful under the right circumstances.

Larry Swedroe on "Why Experts Fail Us"

CBS MoneyWatch blogger Larry Swedroe glosses David Freedman's Wrong and Philip Tetlock's Expert Political Judgment to explain why expert advice goes wrong, and why experts have incentives to oversell their certainty and expertise:

Most of us want certainty, even when we know, logically, that it doesn’t exist. With investing, it’s a desire to believe that there’s someone who can protect us from bear markets and the devastating losses that can result. However, we’ve seen on numerous occasions that experts simply aren’t that expert. Of course, the next question is: “Why?”

Predictive policing

From Slate:

Police departments have long been in the data game, with such efforts as CompStat. But there's a new twist: They're not just using statistics to assess the past. Now they're trying to predict the future. In November 2009, the National Institute of Justice held a symposium on "predictive policing," to figure out the best ways to use statistical data to predict micro-trends in crime. The Los Angeles Police Department then won a $3 million grant from the Justice Department to finance a trial run in predictive methodology. (The grant, like the rest of the 2011 federal budget, is pending congressional approval.) Other police departments are giving predictive policing a shot, too, from Santa Cruz, which recruited a Santa Clara University professor to help rejigger their patrol patterns, to Chicago, which has created a new "criminal forecasting unit" to predict crime before it happens....

Predictive policing is based on the idea that some crime is random—but a lot isn't. For example, home burglaries are relatively predictable. When a house gets robbed, the likelihood of that house or houses near it getting robbed again spikes in the following days. Most people expect the exact opposite, figuring that if lightning strike once, it won't strike again. "This type of lightning does strike more than once," says [UCLA anthropology professor Jeffrey] Brantingham. Other crimes, like murder or rape, are harder to predict. They're more rare, for one thing, and the crime scene isn't always stationary, like a house. But they do tend to follow the same general pattern. If one gang member shoots another, for example, the likelihood of reprisal goes up....

Data-driven law enforcement shows that the criminal mind is not the dark, complex, and ultimately unknowable thing of Hollywood films. Instead, it's depressingly typical—driven by supply, demand, cost, and opportunity. "We have this perception that criminals are a breed apart, psychologically and behaviorally," says Brantingham. "That's not the case."

History of technology, key to the future

One of the truisms about futures is that insights can come from all kinds of unusual places and unexpected corners of the world. This morning I ran across an illustration of this principle in blog form: an article about a set of 1931 predictions about 2011, via Abnormal Use: An Unreasonably Dangerous Products Liability Blog. Because of course when you think "the history of futures," the next thing that comes to mind is "products liability blogs that interview people on the latest developments in torts."

But to the predictions:

1931 was a long time ago, and few who live today can claim to remember it all too well.... It was a far different time culturally, socially, politically. The issue: What did the great minds of 1931 predict the rapidly approaching 2011 would be like?

There is actually an answer to that question.

Way back on September 13, 1931, The New York Times, founded in 1851, decided to celebrate its 80th anniversary by asking a few of the day's visionaries about their predictions of 2011 - 80 years in their future. Those assembled were big names for 1931: physician and Mayo Clinic co-founder W. J. Mayo, famed industrialist Henry Ford, anatomist and anthropologist Arthur Keith, physicist and Nobel laureate Arthur Compton, chemist Willis R. Whitney, physicist and Nobel laureate Robert Millikan, physicist and chemist Michael Pupin, and sociologist William F. Ogburn.

The most interesting piece, to my mind, is Ogburn's. Of course he got some stuff wrong, but the broad outlines of his vision were pretty spot on:

Technological progress, with its exponential law of increase, holds the key to the future. Labor displacement will proceed even to automatic factories. The magic of remote control will be commonplace. Humanity’s most versatile servant will be the electron tube. The communication and transportation inventions will smooth out regional differences and level us in some respects to uniformity. But the heterogeneity of material culture will mean specialists and languages that only specialists can understand. The countryside will be transformed by technology and farmers will be more like city folk. There will be fewer farmers, more wooded land with wild life. Personal property in mechanical conveniences will be greatly extended. Some of these will be needed to prop up the weak who will survive.

Inevitable technological progress and abundant natural resources yield a higher standard of living. Poverty will be eliminated and hunger as a driving force of revolution will not be a danger. Inequality of income and problems of social justice will remain. Crises of life will be met by insurance.

Not only are the big trends recognizable, but the specificities are interesting too: yes, there's no mention of the microchip, but it strikes me that "the electron tube" is the functional equivalent in his vision. It's also heartening because Ogburn (here's a pretty good biography) was noted for his work at Columbia on social trends, and argued for the growing importance of technology as a driver of human affairs and the future (obviously). He was elected first president of the Society for the History of Technology, but died before he could take office.

illustration from William F. Ogburn, You and Machines, via flickr

Some of his work was controversial-- his 1934 pamphlet You and Machines was banned on the grounds that it was too left-wing-- but the rest of his work was more mainstream, and as Rudy Volti argues (in a recent Technology and Culture article available behind the Project MUSE firewall), deals with issues that have been at the center of the history of technology and STS:

Ogburn's seminal work on technology was Social Change with Respect to Cultural and Original Nature... [which] introduces the concept that has been his greatest sociological legacy: cultural lag. As he explains: "The thesis is that the various parts of modern culture are not changing at the same rate, some parts are changing much more rapidly that others; and that since there is a correlation and interdependence of parts, a rapid change in one part of our culture requires readjustments through other changes in the various correlated parts of culture."

A 1950 edition of the book more explicitly lays out his theory of the "role of an advancing material culture in bringing about social change," and breaks it down into four parts:"

"Invention" is still given top billing, complemented by "accumulation" (the store of past inventions, which expands at an exponential rate) and the diffusion of inventions from other cultures. The fourth element of social change is "adjustment," the process through which lagging cultural elements catch up with the changes driven by invention, accumulation, and diffusion.

You can see Ogburn's model in his 1931 New York Times piece.

Social Scanning article now officially published

I know I've personally sent copies to all fourteen people who are interested in the article, but my piece on social scanning (cleverly subtitled, in Shakespearean fashion, "or, Finally a Use for Twitter") is formally, officially published in Futures. It's part of a special issue on "Global Mindset Change."

Odds are unless you're behind a university paywall you can't actually get to the article, but here's a draft that lays out the argument reasonably well.

The full citation is "Social scanning: Improving futures through Web 2.0; or, finally a use for twitter," Futures v. 42 no. 10 (December 2010), pp. 1222-1230.

Robert Merton on self-fulfilling prophesies

From Merton's "The Self-Fulfilling Prophecy," The Antioch Review 8:2 (Summer 1948), 193-210:

The self-fulfilling prophecy is, in the beginning, a false definition of the situation evoking a new behavior which makes the originally false conception come true. The specious validity of the self-fulfilling prophecy perpetuates a reign of error. For the prophet will cite the actual course of events as proof that he was right from the very beginning.... (195)

[P]ublic definitions of a situation (prophecies or predictions) become an integral part of the situation and thus affect subsequent developments. This is peculiar to human affairs. It is not found in the world of nature.... (195)

The self-fulfilling prophecy, whereby fears are translated into reality, operates only in the absence of deliberate institutional controls. And it is only with the rejection of social fatalism implied in the notion of unchangeable human nature that the tragic circle of fear, social disaster, reinforced fear can be broken. (210)

Another interesting element in this article: Merton presents a scenario of a bank failure as self-fulfilling prophecy that reveals his assumptions about who bankers were back in the day. "It is the year 1932. The Last National Bank is a flourishing institution.... Cartwright Millingville has ample reason to be proud of the banking institution over which he preside."

I will never escape the shadow of Robert Merton

I spent years in graduate school reading Robert Merton's sociology of science: I trucked his collection of essays The Sociology of Science (which I bought in an excellent little overstuffed bookstore on the edge of campus) around with me from Penn to Williams to Stanford to Berkeley to Davis to Chicago and back to California, and had his work assigned in any number of classes.

In all those years, I never realized that he had a second intellectual life as, essentially, a futurist; or more specifically, as a scholar of ideas that we use in everyday thinking about the future. But a few weeks ago I found a 1936 article of his on "The Unanticipated Consequences of Purposive Social Action" when researching a piece on unintended consequences. So it should really come as no surprise, when I idly searched for scholarly works on the concept of "self-fulfilling prophecies" that Merton would have written one of the definitive articles, this time in the Antioch Review in 1948 (it's available on JSTOR).

His generation of social scientists-- along with contemporaries like John Kenneth Galbraith, Daniel Bell, Robert Hielbroner, and Donald Michael (now less well-known but still very interesting)-- really knew how to ask big questions that connected contemporary issues with futures.

File under "self-fulfilling prophecies"? The Hindenburg Omen looms

We'll know in September. From the Wall Street Journal:

Forget about Friday the 13th. Many on Wall Street took to whispering about an even scarier phenomenon—the "Hindenburg Omen."

The Omen, named after the famous German airship in 1937 that crashed in Lakehurst, N.J., is a technical indicator that foreshadows not just a bear market but a stock-market crash. Its creator, a blind mathematician named Jim Miekka, said his indicator is now predicting a market meltdown in September.

Wall Street has been abuzz about whether the Hindenburg Omen will come to bear, with some traders cautioning clients about the indicator and blogs pondering all the doom and gloom.

There's a technical but still accessible explanation of it on Zero Hedge. I confess I haven't heard of it, but apparently the fact that it's "easily the most feared technical pattern in all of chartism," combined with the fact that warnings about it came out on Friday the 13th, provides something for everyone to worry about.

On the use of economic forecasts

Tyler Cowan asks, "Why does anyone support private macroeconomic forecasts?"

I've long found the market for such forecasts to be a puzzling practice.... Even if some forecasts are quite useful, what's the value of supporting a marginal or additional forecast? Is the next forecast to come along so much better? Forecasts would seem to be the classic example of a public good.

Given my interest in turning scanning from a private resource into a public good, Tyler's question caught my eye. He suggests that it's not about getting better knowledge at all, but about something more like social status: perhaps

outsiders pay for the forecast to join a more exclusive club of clients with other privileges. It's a bit like how art galleries won't sell their best pictures to complete outsiders but instead ask that you "pay your dues" by being a loyal customer for years. In other words, it's an arbitrary fee to enforce price discrimination, backed by some plausible pretext.

Under a related model, the firm pays for the forecast as a means of generating publicity, signaling its size, seriousness, and audience, and in general marketing itself to outside clients. It is unclear who bears the final incidence of these expenditures, the firm or the clients, but still "forecasting isn't about knowledge," as Robin Hanson would have said to the oracle at Delphi.

In other words, the "utility" of macroeconomic forecasts doesn't come from the actual information you get by buying them, but from the signal buying the forecasts sends to competitors about your seriousness. I've heard similar kinds of arguments about futures and scenario work-- that Hermann Kahn's work on nuclear scenarios in the 1960s was intended not just to improve the Joint Chiefs' or SAC's thinking, but to make the Soviets worry about what we going to do in the future-- and I think there are almost always local reasons for engaging futurists that don't have much to do with the official reasons you hire futurists. (Indeed, the really good consultants are able to figure out these reasons, and to adjust their engagements accordingly.)

Bruce Bartlett proposes another answer:

Why Tyler may not realize is that forecasting companies do far more than generate aggregate data; they also produce a vast amount of industry specific data that is enormously useful for investors, managers and others that need to know how a particular industry is expected to perform given the forecast for GDP, inflation etc.

In some cases, the industry data may even contribute to price collusion.

I should note that there's an excellent study of futures orientation, forecasts and uncertainty in forestry-- a profession that sees itself as managing processes that play out over centuries-- but we could use many more studies of how our work is really used.

Forecasting arrogance as professional service

Robin Hanson looks at three recent studies measuring forecasting overconfidence in medicine, law and finance. None of them do very well, and in all three cases they're consistently overconfident in their forecasts. He comments:

I strongly suspect these patterns are driven mostly by customers, i.e., that more accurate professionals would be less successful in inspiring confidence by others in them. If you are a successful professional, that is probably in part because of your unjustified arrogance.

That sounds about right to me.

The very long shadow of the history of technology

A confession: when it comes to thinking about the future, I hold two views. On one hand, I find the black swans work of Nassim Taleb-- the argument that the speed and complexity of the modern world has left it vulnerable to more, and more unpredictable, crises-- pretty convincing. (Call this the New View.)

On the other, I also believe that much of what we claim is novel about this modern age is not so new. Many facets of globalization-- the importance of migration, global trade, etc.-- are actually as old as civilization. I also believe that other things, like a belief in greater vulnerability to epidemics and financial panics (or just as worrying, the belief that we are now immune from such things), are a product of a relatively short-term view of history. You could better understand our current world, and think more clearly about the future, if you stretch your view of the past from the last 50 years to the last 500 or 5,000. (Call this the Long View.)

Obviously, the New View and the Long View are contradictory. I get around that by not thinking about both of them at the same time. But I'm trying to construct a framework that fits them together.

This morning I ran across another data-point in the Long View: a new piece by Diego Comin, Erick Gong, and William Easterly looking at very long-term trends in technology and economic development. As Easterly explains,

We collected crude but informative data on the state of technology in various parts of the world in 1000 BC, 0 AD, and 1500 AD.

1500 AD technology is a particularly powerful predictor of per capita income today. 78 percent of the difference in income today between sub-Saharan Africa and Western Europe is explained by technology differences that already existed in 1500 AD – even BEFORE the slave trade and colonialism.

From the abstract (pdf):

The emphasis of economic development practitioners and researchers is on modern determinants of per capita income such as quality of institutions to support markets, economic policies chosen by governments, human capital components such as education and health, or political factors such as violence and instability.

Could this discussion be missing an important, much more long-run dimension to economic development?... Is it possible that history as old as 1500 AD or older also matters significantly for today’s national economic development? A small body of previous growth literature also considers very long run factors in economic development.... This paper explores these questions both empirically and theoretically. To this end, we assemble a new dataset on the history of technology over 2,500 years of history prior to the era of colonization and extensive European contacts.... We detect signs of technological differences between the predecessors to today’s modern nations as long ago as 1000 BC, and we find that these differences persisted and/or widened to 0 AD and to 1500 AD (which will be the three data points in our dataset, with 1500 AD estimated from a different collection of sources than 1000 BC and 0 AD). The persistence of technological differences from one of these three “ancient history” data points to the next is high, as well as robust to controlling for continent dummies and other geographic factors.

Our principal finding is that the 1500 AD measure is a statistically significant predictor of the pattern of per capita incomes and technology adoption across nations that we observe today.

Of course, one can get into how this is a different set of forces than most futurists are interested in-- but to the degree that it serves as a corrective to the tacit view held some futurists that history doesn't matter at all-- a kind of social science version of transhumanism, in which thanks to technology (or migration or whatever) we're able to ignore the past and its gravitational pull-- it's worth reading and pondering.

Another example of an unintended consequence

This time from Wondermark, so naturally it's kind of (okay, very) absurd:

Designing an ECAST: How to bring citizens into science policy

Darlene Cavalier has a great piece in Discover about citizen science and reimagining the Office of Technology Assessment. As she explains,

What originally began as Science Cheerleader’s effort to help reopen the Congressional Office of Technology Assessment (an agency, shut down in the 90’s, that helped Congress better understand policy implications of complex, science issues), has evolved into this reincarnation.

Why? It became apparent after two years-worth of numerous discussions with a variety of stakeholders, that reopening the “old” OTA would leave little, if any, opportunities to invoke contemporary applications critical to 21st century governing: decentralized expertise (tapping the knowledge of scientists across the nation) and citizen engagement, to name but two....

Government policymakers, businesses, non-governmental organizations, and citizens rely on analysis to capably navigate the technology-intensive world in which we now live. The new model, described in the report, would provide opportunities to generate input from a diverse public audience, while promoting societal discussions and public education.

This redefines the technology assessment model by recommending the formation of a first-of-its-kind U.S. network to implement the recommendations: Expert and Citizen Assessment of Science and Technology (ECAST).

I'm very interested in systems like this, so I want to take a quick shot at outlining a couple properties that an ECAST would actually have to have to work.

First is a philosophical question. Does this kind of knowledge about the potential impacts of science and technology simply exist somewhere? Or does it need to be created?

Put another way, if you assume the former, your task is to find the person-- the orthogonal thinker in a dorm room, the visionary at the startup-- who can share a cool insight. If it's the second, then your task is to bring together interesting people, and get them to think together about the future of science and technology.

I've had some clients who were firm believers in the first approach. They wanted me to find the undiscovered visionary: one client more or less told me that my mission was to find the 16 year-old who could become another Steve Jobs, and to find him in China. Wrapped up in this mandate are a couple assumptions: that there's someone out there who sees the future really clearly, and we just need to find them; that such people are the ones who make history (and the future); and that we'll know this person when we find them.

I think each of these three assumptions is faulty. History isn't made by visionaries who spend a lifetime pursuing One Single Vision: I'm not sure that Steve Jobs had a vision for the iPhone that I could have extracted from him in 1973, when he was still-- well, before he was Steve. Further, great technologies just aren't made by single people: like all creative endeavours, they're collaborative efforts. Finally, I'm not sure how you'd sort out crackpot from genius ideas about the future in any over-the-transom process.

But this is not say that a simple process that taps the raw "wisdom of the crowds"-- say polling people, or opening up a wiki about the implications of science and technology-- is a substitute. My experience trying to get experts to contribute to an open future of science platform makes me skeptical that you'll get useful results just by throwing open the doors, however nice they are. (One of the Discover commenters makes this point, too.)

Rather, you need a process that has several properties.

First, it needs to be accessible to just about anyone who wants to participate. There should be some kind of barriers to participation, to discourage people who want to just advocate for their products or talk about how putting microchips in our food will make us all super-geniuses.

Second, it should combine open-ended scanning with events that have clear dates. You need the former because innovation and other interesting things happen all the time; you need the latter because you need mechanisms to encourage concentration and innovative thinking (and hard deadlines and urgency are shown to stimulate more out-of-the-box thinking than leisure and freedom-- a fact that many an academic has discovered the hard way).

Third, the system should thoughtfully draw on the wide varieties expertise that can be brought together in a virtual platform. Personally I think talking in terms of "citizens" and "experts" threatens to obscure something important, namely that "expertise" about exceptionally complex phenomena is highly distributed and localized. If you want an opinion about the value of Lie numbers in Garrett Lisi's theoretical physics, there are about a dozen people in the world you want to talk to (mainly this guy); if you want to think about the broad implications of synthetic biology, you want Rob Carlson, but you also want a lot of other people who can contribute expertise in law, engineering, manufacturing, policy, etc. etc. As the history of science shows, sometimes the people least likely to see the long-term implications of ideas or inventions are the scientists and engineers most intimately involved with their creation.

Fourth, you need some real-world events. Virtual meetings can be great-- they generally suck, but they can be designed to be great (I make part of my living designing them)-- but face-to-face interactions still produce things that you don't get through online itneractions. Even better are events that combine virtual and real interactions and spaces: if properly designed, you get the best of rich social interactions that, as primates, we're so good at, and the virtues of digital scribing and recording and sharing.

Finally, the exercise has to have an obvious payoff. This means two things. First, if it can be designed to provide some immediate benefits to participants-- class credit for students, data for grad students, citations for professors, networking opportunities for entrepreneurs, a thousand new Facebook for the rest of us-- so much the better. Second, it should be clear that people from NIH (or Merck or CIA or NSF) are actually paying attention to the results of Cubesat Day or Synthetic Biology Week. That raises the stakes, creates more of a sense of urgency, and makes everyone take the event more seriously.

Now, what kind of technology platform would you use?

My answer for now is, try a little of everything. Unless you get caught in the trap of sourcing the whole project to some soul-sucking systems contractor who'll charge your $37 billion and not really ever deliver what you want, you could do a lot of cheap experiments, in lots of cities; so long as you document well and pay close attention, pretty soon you'll see what works and what doesn't, and you can transplant successful efforts to other places. Don't think in terms of a system, in other words: think in terms of an ecosystem, in which you provide some minimal nutrition (seed funding), encourage rapid evolution, have lots of plasmids and transfer RNA around, and quickly reward success. Maybe that sounds like a cop-out, but it's the best way to get a system that's as flexible and interesting as its subject.

Or am I missing something?

Justin Fox on economists versus historians

Harvard Business Review blogger Justin Fox talks about why economists have become more prominent public intellectuals than historians: He argues that

economists had managed a remarkable balancing act between making the guts of their work totally incomprehensible — and thus forbiddingly impressive — to the outside world while continuing to offer reasonably straightforward conclusions. The basic form of an academic economics paper is a couple of comprehensible paragraphs at the beginning and a couple of comprehensible paragraphs at the end, with a bunch of really-hard-to-follow math or statistical analysis in the middle. An academic history paper, on the other hand, is often an uninterrupted cascade of semi-comprehensible jargon that neither impresses a lay reader nor offers any clear conclusions.

The one economist in the audience had another suggestion. Most economic work was aimed at prediction, and the world is always hungry for predictions. He added that most macroeconomic predictions are worthless (he was a microeconomist), but that doesn't seem to have damped the demand for them.

Once again, it's clear that Philip Tetlock's work explains everything.

But seriously, it seems to me that one of the key features of any form of prediction or forecast-- or even something less formal, like scenarios-- should be transparency: if you're asking people to base their actions in the expectations that certain futures are more important to prepare for than others, it's imperative that you be able to explain to yourself and others how you came to think that those futures were worth taking more seriously. It's never possible to clearly describe every step in your thinking: all knowledge work involves a measure of intuition and tacit knowledge that's acquired over years of practice.

There should be no shame in acknowledging that, and I think we have a lot more to gain than to lose from greater transparency. Obviously it opens your work up to criticism, but also to improvement-- and potentially, rapid improvement. It's essential for making users smarter. The most thoughtful clients I've worked with were the ones who best understood what I do. (Knowing how someone else works doesn't make them obsolete: I love to read about tailoring, but that doesn't make me a cutter.) And it's important for any work that people aren't just going to apply, but are expected to adapt and extend-- which is exactly what happens with scenarios and forecasts. Good tinkering and hacking depends on an ability to get under the hood, play with the parts, understand why things are put together this way rather than that, and thus see new possibilities and uses for products. Perhaps scenarios and forecasts should be designed not just to be read, but torn apart, recombined, and reused: they should be able to stand being assembled in new ways and used in contexts their authors never imagined.

[via Ezra Klein]

Leon Fuerth on "Strategic Myopia: The Case for Forward Engagement"

From Leon Fuerth, "Strategic Myopia: The Case for Forward Engagement":

The habit of heavily discounting the future in favor of the nearterm must be abandoned, for the simple reason that the future—defined here as the rate of incidence of major social change—is accelerating. That acceleration represents, in turn, the dramatically quickened pace of science and technology, translated into ethical, political, economic and social consequences. If we are overtaken and swamped by the accelerating rate of change, then it is likely that our society will fail to grasp major opportunities for advancement and forfeit them to others who are more alert. We will also fail to take action in time to mitigate the societal impact of major, abrupt dislocations....

Leaders are not unmindful of the need to think of the longer-term implications of their actions, but they also know that representing the interests of the future often involves significant political risk to themselves in the present. Faced with such a choice, they frequently take comfort from the bromide that it is impossible to predict the future. That is certainly true in a literal sense, but it obscures a much more important fact: that it is entirely feasible to think about the future in disciplined fashion and to reach conclusions about it that ought to be important factors in the making of contemporary policy.

Forecasting will never reach the point at which it eliminates doubt. However, it can be used as part of an orderly policymaking process to diminish risk and to maximize opportunity. Our era is destined to be marked by accelerating, deep change. In such a period it is increasingly dangerous to make policy only in the short term or to look at the universe of possibilities through the filter of ideology. An important hallmark of successful governance is the timely ability to recognize what may happen, in order to have the best possible chance of influencing what does happen. Democratic governance is at risk of losing this capacity by failing to analyze the alternative paths that lead towards futures that are desirable, or away from those that are not, and especially by failing to begin that process early enough to permit adequate time for the debate and deliberation our system requires.

During the Cold War, the United States practiced “Forward Deployment”: placing its intelligence sensors and its military forces at strategic locations chosen to improve our ability to engage the enemy as early as possible, on terms advantageous to ourselves. We should now be practicing what ought to be thought of as “Forward Engagement”: recognizing and responding to major societal challenges sooner rather than later, when our leverage over the course of events is greatest and the costs for influencing them are lowest.

When being wrong shows you're right

The recent New Yorker piece on Paul Krugman may have the most beautiful example of an "I got the future so wrong, which is why I'm so right" argument I've ever seen:

Last fall, Krugman wrote an article for the Times Magazine, “How Did Economists Get It So Wrong?,” about the profession’s failure to anticipate the financial crisis, and what that revealed about its failings in general. He accused his colleagues of mistaking beauty for truth. They were so enamored of the elegance of their models and the consistency of their logic, he wrote, that they had come to believe that assumptions that were originally adopted merely as tools (perfectly rational individuals, efficient markets) by Milton Friedman’s generation were so sacrosanct that economics wasn’t economics without them. Freshwater types, in particular, had forgotten the Depression, forgotten what Keynes had said about the resemblance of financial markets to casinos. So attached were they to the idea that markets always got things right that some actually suggested that unemployment must be a consequence of workers’ choosing not to work. Saltwater economists were less blinkered in their view of markets and the rationality of investors, Krugman wrote (Larry Summers, a saltwater type, once began a paper on finance by declaring “THERE ARE IDIOTS. Look around”), and had retained a Keynesian view of recessions as crises of insufficient demand. But even saltwater models had no room for such wild imperfections as bubbles and banking-system collapse. “Economists will have to learn to live with messiness,” Krugman concluded.

Reactions to his article were quick and outraged. “Who are these economists who got it so wrong?” a Washington University economist, David Levine, wrote. “Speak for yourself kemo sabe. . . . It makes me feel physically ill that a distinguished economist could be so ignorant of his own profession.” “How sad,” John Cochrane, of the University of Chicago, wrote. “Don’t argue with them, swift-boat them. Find some embarrassing quote from an old interview. Well, good luck, Paul. Let’s just not pretend that this has anything to do with economics.” Levine and Cochrane maintained that the fact that freshwater economists had failed to predict the financial crisis was not an embarrassment to their theories but a confirmation of them: “The central empirical prediction of the efficient markets hypothesis is precisely that nobody can tell where markets are going—neither benevolent government bureaucrats, nor crafty hedge-fund managers, nor ivory-tower academics,” Cochrane wrote. If professional economists failed to predict or understand the crisis, how could it make sense for Krugman to argue that bureaucrats would do a good job of curing it? [emphasis added]

Also, James Galbraith follows up on Krugman with a piece about the ones who did get it right. And this Michael Lewis piece about Wall Streeters who saw it coming is also great (as is everything Lewis writes).

Blind oracles, the challenge of prediction, and understanding the future

David Orrell, whose work I find very stimulating (and who I had the pleasure of meeting on my last trip to Oxford), has an essay in the January/February Literary Review of Canada. In theory it's a review of Florin Diacu's Megadisasters: The Science of Predicting the Next Catastrophe, but like many a good review, it uses the book as a launching-point to talk about the bigger issues the book raises.

In this case, it's an especially useful summary of Orrell's basic-- and among scientists, somewhat controversial-- argument about the limits of prediction.

[E]quations are useful tools for describing and understanding extreme events such as earthquakes or tsunamis (a worthy goal in itself), [but] as far as I can see, none of the scientific models can reliably predict them....

This points to a basic problem with the Newtonian approach to prediction: despite its eminent logic, it just does not seem to work very well when applied to complex systems of the type we really want to know about, such as weather, the economy or our own health...

In economics, the inability to predict the future was explained away in the 1960s by the efficient market hypothesis. It saw the market as a kind of deity whose short-term motions no one can anticipate, but held that the long-term risk could still be modelled by equations. The flaws in this theory became increasingly obvious, as the risk models missed even the chance of the credit crunch, and in fact played a large role in making it happen.

In weather forecasting, lack of prediction was explained by chaos theory and by the “butterfly effect.” According to this theory, the atmosphere is so unstable and chaotic in the short term that a butterfly flapping its wings can later cause a hurricane on the other side of the world; but again, long-term prediction of the climate is assumed to be possible. However, while the atmosphere certainly has some unstable dynamics, experiments show they are hardly its defining feature.

In my opinion, both the efficient market and the butterfly effect are fig leaves that explain away forecast error, while allowing scientists to retain some of their oracular authority for longer-term predictions, which are safe because they are for the distant future. The real reason for our lack of forecast ability in all these areas, I believe, is simply that our traditional modelling approach does not work when applied to complex organic systems. These systems tend to be dominated by emergent properties, which by definition cannot be modelled or predicted from knowledge of the components.

I'm not nearly enough of a mathematician to assess the validity of Orrell's more technical claims, but I find several elements of his critique of scientific prediction especially challenging.

One is that we're not fooled by randomness, but by emergence. Nassim Taleb's arguments about black swans may not be right-- or at least, we need to settle whether the big, unexpected phenomena we want to understand are a result of randomness (and thus are completely unpredictable), or a product of emergent phenomena (which are beyond our capacity to reliably model, but which I understand are, in theory, computationally tractable). It doesn't make a lot of difference in the moment if the crisis is caused by randomness or complexity; but over the long run you want to figure these things out.

The second is that there appears to be an inverse relationship between the power of models to explain the past and future. The more time you spend tweaking a model to fit bumps in historical data, and the better the model becomes at reproducing the past, the weaker it becomes as a predictive tool. In contrast, relatively simple models that do a poor job of reproducing past data can sometimes do a better job of prediction. This has the potential to undermine a whole structure of argument among futurists-- and among historians, come to think of it-- that contends that historical understand is useful for making sense of the future. It may be that our models for understanding the past aren't complex enough to fall prey to Orrell's Paradox (I coined the Nunberg Error, so I might as well try again), but it makes me wonder whether there are ways to refine the way we use historical thinking and models to make sense of the future that avoid this problem.

The third question this raises is, how then do you talk about big future problems like climate change without leading people to believe that since the future is unknowable, we don't need to think about it or act in ways to improve it? I can see the argument for creating an alternative to the IPCC work, for example-- given the sensitivity of the models to initial conditions, etc.-- but with environmental issues we no longer seem to have a way to think about things that scientists don't fully understand, but which still seem to exist / are happening and need to be dealt with. It's almost like arguing that because the efficient markets hypothesis doesn't work, we're not in a global recession.

None of this makes me think we should give up forecasting-- the ability to think about the future is one of the things that makes us human, as Daniel Gilbert put it (though that may not be true)-- but we need to think about how to improve it; how we can do so in ways that do not lead us to imitate (and reproduce) the errors in scientific prediction; and how to do so in ways that don't ultimately work to the detriment of our audiences by encouraging passivity in the face of uncertainty.