The Long and Short of Wikiprediction

The Problem with Wikipedia. (Click to  enlarge)In what may be a self-organized example of Occam's Razor, consider the case of reliability of Wikipedia articles.

Recently, Joshua E. Blumenstock of UC Berkeley performed a statistical analysis of 1000's of wikipedia pages, looking for predictors of quality articles. (Where "quality articles" was taken to be featured articles. These articles are given this rating by Wikipedia editors, using specific criteria.   As of this posting, there are approximately 2000 featured articles out of over 2.4 million wikipedia articles.)

In his paper Automatically Assessing the Quality of Wikipedia Articles Blumenstock describes the search for correlation between "featuredness" and a a wikiload of possible variables. The variables included surface features (e.g. # of characters, words, one-syllable words), structural features (e.g. links , images, tables), a variety of readability metrics (e.g. Gunning Fog, Coleman-Liau Index), and part of speech tags (e.g. nouns, past participles, perterites).

He needn't have looked so deeply. It turns out that word count alone is an incredibly potent predictor. Amazingly, Blumenstock found that whether an article had greater or less than 1830 words was all that was needed to predict whether an article was featured with 97% accuracy!

Now why is this?

Click to read more ...

Where the Hell Are They? Now pass the pasta...

Klaatu barada niktoOne of the great lines in all of 20th century science was uttered over lunch by Enrico Fermi. In a discussion of the possible likelihood of many advanced civilizations in our galaxy, Fermi said something to the effect of "well, where the hell are they."

I may have taken some liberty in the way Fermi expressed himself, but there's no doubt that Fermi's questions is one of the more provocative off-the-cuff statements ever made because of the response that it generated, including immortalization as a named dilemma: The Fermi Paradox

In 1961, approximate 10 years after Fermi's lunch-time query, Frank Drake developed an equation that he used to predict the number of advanced civilizations in our galaxy. Upon reading about the Drake Equation many years ago, I was struck by its simplicity and its audacity. Here it is in its full glory.  (Read all about the parameters here ) 

N = R* f_p n_e f_l f_i f_c L

The equation consists of a chain of probabilities, all multiplied together in the fashion of the probability of a string of independent events. Depending on the values of the individual probabilities, estimates of the average number of advanced civilizations/galaxy range from several thousand to less than one.

Click to read more ...

Preparing for Chaos: Patently Lame

The Boston Molasses DisasterIt seems tautological that it's impossible to get ready for Chaos. Yet a recent patent application by researchers at IBM claims to do just that. In their System and method for optimizing the selection, verification, and deployment of expert resources in a time of chaos Robert Friedlander, Richard Hennessy, Anwer Mujahid Khan and James Kraemer describe:

A computer implemented method, apparatus, and computer usable program code for finding skills and resources for a chaotic event. Skills data for the chaotic event are organized. A determination is made whether the skills and the resources are available in response to a receiving an identification of the skills and the resources that are required to manage the chaotic event. The skills and the resources are optimized based on requirements and constraints, potential skills, and enabling resources to determine optimized skills and optimized resources. The availability of the optimized skills and the optimized resources are verified. The optimized skills and the optimized resources are reoptimized in response to a determination that the optimized skills and the optimized resources are unavailable.

Pardon my skepticism here - while what is described here certainly seems laudable, it doesn't sound like much more than a nice robust system that links resources with those who need them. The word "chaos" is being (mis)appropriated to add some juice to the patent claim. There ain't no chaos here, at least not in the sense of CHAOS Theory. The word chaos then doesn't convey anything more than a hook for readers and , presumably, patent examiners.

Maybe the authors do better in actually describing a chaotic event. See what you think:

Click to read more ...

Prescience and Spice and Everything Nice

There is prediction, and there is prognostication.  And, once in a great while, there is true prescience.

Consider the case of J.C.R. Licklider - a seminal figure of whom, until recently, I had never heard. On reading about him, I am staggered by his dead-on prescience in predicting the ultimate power of social connections via the internet - back in 1960. His Man-Computer Symbiosis has to be read to be believed. In it he touches on the obvious: the need for speed, memory, robust programming languages, and effective I/O.  But he goes much deeper, brilliantly identifying memory and its organization (which leads to necessary "searchability") as necessary developments before the symbiosis can happen:

Man-computer symbiosis is an expected development in cooperative interaction between men and electronic computers. It will involve very close coupling between the human and the electronic members of the partnership. The main aims are 1) to let computers facilitate formulative thinking as they now facilitate the solution of formulated problems, and 2) to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs...Prerequisites for the achievement of the effective, cooperative association include developments in computer time sharing, in memory components, in memory organization, in programming languages, and in input and output equipment.

And note the reference to emergent behavior: to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs.

Man-Computer Symbiosis reminds me of one of my favorite sci-fi books, although pigeonholing it into that genre doesn't do justice to Last and First Men: A Story of the Near and Far Future, by Olaf Stapledon. Here Stapledon predicts the fate of mankind from Stapledon's present (1930) through the next two billion years, during which mankind evolves through

Click to read more ...

Do E-fields Cause or Hinder Cancer?

Field near a TNT molecule. Click to enlargeWith the ongoing debate about potential health hazards of living near radio/TV towers, or the dangers of extensive cell-phone use, it is reassuring to see the latest news on the efficacy of electric fields in slowing down some cancers.  (Of course a skeptic might claim that the news is a spin job by cell operators to get activists from fighting the placement of towers and cell sites near population centers.)

Nevertheless, the study is intriguing because it describes a set of potential reasons why the E-fields might be the causal agent for the stunted cancer growth.

In vitro, the electric fields were seen to have two effects on the tumor cells.

First, they slowed down cell division. Cells that ordinarily took less than an hour to divide were still not completely divided after three hours of exposure to an electrical field of 200 kHz...

Click to read more ...

Intelligent String Theory Design - Patently Falsifiable

String Theory by Marlene Healey Prediction plays a crucial role in the continuing debate of whether string theory and intelligent design are linked by a common lack of falsifiability. Here's how this argument goes:

1) ID does not make predictions that are testable, and therefore not falsifiable, failing Popper's main criteria for categorization of a theory as scientific.

2) String Theory has not produced a prediction that is testable because it requires that there exist objects that are simply not observable - e.g extra dimensions. Therefore it is also not a scientific theory, making it analagous to ID (in a falsifiability sense).

It follow then that if you dismiss ID, you have to dismiss string theory. (See, e.g. W. Dembski's Uncommon Dissent blog)

But do you? An interesting argument against this conclusion is provided by Amanda Gefter's editorial in the Philly Inquirer titled A Scientific Leap Without the Faith. Gefter points out the distinct difference between string theory and ID as one of explanatory power combined with the internal elegance of the mathematics. In fact, it is the presence of mathematics that provides the oomph that catapults string theory over ID:

Click to read more ...

Hurricane Prediction, Pt. 2 - Path and Intensity

Projections for Hurricane Dean (NOAA)This post continues the discussion of hurricane prediction begun in my recent post.  There I described efforts at predicting hurricanes before they actually start, and the use of correlates such as el niño and sand in the Sahara.

What about predictions once the hurricanes are on the scene? The recent news surrounding the potential path of Hurricane Dean, and the predicted intensity, are different types of models.

There's an informative article today by Matt McGrath for BBC News that discusses the issues surrounding these predictions. (See The Science of Hurricane Prediction ) Of interest is the fact that it easier to predict the ultimate path of faster-moving hurricanes. In effect, when hurricanes "hang around" too long, their motion can be affected by many other factors - throwing off the model predictions. 

The BBC News site has a number of clever animations, including an animated guide to how hurricanes form. This is a nice applet - one of a set of applets for many natural disasters, including earthquakes, tsunamis, tornadoes, and volcanoes.

Saharan Sand & Hurricane Prediction

I have written before about modeling earthquakes and hurricanes - two phenomena are often not covered in homeowners insurance because they are Acts of God. There is still a long way to go before these are understood to the satisfaction of a predominance of scientists.

But what about prediction? With hurricane season about to start in earnest in the Atlantic, it has been pretty quiet. How are hurricanes predicted, and how good are these predictions?

El Niño events and hurricanes are highly correlated, and hence the Niño is part of every hurricane forecasters toolkit/ There has been a lot of publicity of late indicating a strong correlation between Saharan sand storms and hurricane activity, suggesting that sand activity would complement El Nino as a predictor.

While no one would dispute the value of another correlate for hurricane activity, I did notice the typical media confusion between understanding what was going on, and what was purely a prediction based on a correlation. I was just about to write one of my typical rants when I came across a blog that already took the media to task back in 2006. Written by Daniel Collins from UW-Madison on his Down To Earth blog, the post title says it all:

Click to read more ...

Modeling Acts of God, Part 2 - Earthquakes

This is #2 in a series. (Click here for #1 on modeling hurricanes)

Earthquake modeling seems to generate a wide range of emotions, vitriol, successes, and failures - so wide as to need a logarithmic scale, a la Richter.

The Parkfield Earthquake Experiment, now running for over 22 years, has been the development test bed and experimental "lab" for US Geological Survey/State of California efforts to develop  physical models of earthquakes that will lead to viable predictions. The USGS site contains a wealth of information on the experiment, and good background on the history of earthquake prediction, which is still highly hit or miss. An interesting excerpt from the site neatly illuminates the need for prediction based on an understanding of physical causes as opposed to one based on statistical correlation only:

Early scientific efforts toward earthquake prediction in the U.S. were directed primarily toward the measurement of physical parameters in areas where earthquakes occur, including seismicity, crustal structure, heat flow, geomagnetism, electrical potential and conductivity, gas chemistry. Central to these efforts was the concept that a precursor might be observed in one or more of these measurements. However, the connection between a commonly accepted precursor and the earthquake was often speculative and uncertain. A coherent physical model was lacking.

Click to read more ...

Constructal Theory of Everything?

A very interesting text that describes what appears to be a bold new approach to modeling many possible systems - even human ones - is about to be released. Titled Constructal Theory of Social Dynamics and  edited by Adrian Bejan and Gilbert W. Merkx, the text "brings together for the first time social scientists and engineers to develop a predictive theory of social organization, as a conglomerate of mating flows that morph in time to flow more easily (people, goods, money, energy, information). These flows have objectives (e.g., minimization of effort, travel time, cost), and the objectives clash with global constraints (space, time, resources). The result is organization (flow architecture) derived from one principle of configuration evolution in time (the constructal law): "for a flow system to persist in time, its configuration must morph such that it provides easier access to its streams."

Begin and Merkx are from Duke. The Duke press release  is very enticing - this is certainly a text that will serve as a reference for the chaos and fractals course, or perhaps a primary source.

Some tantalizing tidbits:

Why does a railway network look like a river? Why do the streets of old Rome look like a leaf? Because whether their shape is determined by the interactions of molecules or the choices made by individual humans, all of these systems of flow are governed by a relatively simple new principle of thermodynamics.

Click to read more ...