Notes

Why statistical thinking reveals hidden truths

Statistical reasoning often contradicts intuition but produces superior results. The German tank problem illustrates this perfectly: Allied forces needed to estimate German tank production during World War II using only serial numbers from destroyed tanks. While human intelligence sources estimated 1500 tanks per month, statisticians calculated approximately 200 per month using a simple but elegant method.

The statistical approach worked by taking the highest observed serial number and adding back the average distance between captured tanks. For example, if tanks numbered 26, 74, and 90 were captured, the estimate would add the average gap (about 32) to the maximum (90) to account for uncaptured tanks. When post-war German records became available, the statistical estimates proved remarkably accurate while human intelligence was wrong by a factor of seven.

Ajay explains the mathematical insight:

"One simple approach is to say 90 is the highest number we have verified as a tank that has been killed. Therefore 90 tanks exist. The sweet mathematical intuition is, no, there could well be more tanks and you just happened to kill number 90. Maybe there's 100, maybe there's 110."

This principle extends beyond wartime applications. Abraham Wald's analysis of returning aircraft demonstrated another crucial statistical insight: survivorship bias. When analyzing bullet damage patterns, the obvious approach suggested reinforcing areas with the most holes. Wald realized the critical question was "where are the missing bullets?" - planes hit in certain areas never returned home, making those locations the most vulnerable despite showing fewer holes in the surviving sample.

The remarkable achievement of C.R. Rao

C.R. Rao stands among the greatest mathematical minds of the 20th century, independently developing two fundamental theorems that bear his name. Born in 1920 in South India, he migrated to Calcutta and entered the world created by P.C. Mahalanobis at the Indian Statistical Institute. Working in relative isolation from European mathematical centers, Rao produced results that became cornerstones of statistical theory.

The Cramér-Rao lower bound, developed in 1943, establishes theoretical limits on how accurate any statistical estimator can be. This fundamental result answers the question of whether statisticians can continuously improve estimators indefinitely - the answer is no, there are mathematical boundaries that cannot be crossed. The Rao-Blackwell theorem provides a systematic method for improving existing estimators by one step.

Ajay recalls learning about these achievements:

"Just imagine how fundamental that result is - given a model, given a data set, you get an analytical derivation of what the best estimator can possibly exist. The Cramér-Rao lower bound shows us the limit of what estimators can do. You can run and run and build better and better estimators, but you can't cross the Cramér-Rao lower bound."

The fact that these theorems carry Rao's name alongside European mathematicians like Cramér and Blackwell reflects both the quality of his work and the intellectual honesty of figures like Ronald Fisher, who recognized excellence regardless of its geographic origin. This stands as a remarkable achievement for someone working in pre-independence India without institutional support from major universities.

The genius and limitations of P.C. Mahalanobis

P.C. Mahalanobis created the institutional foundation that enabled C.R. Rao's achievements. A physicist by training who discovered statistics while reading Biometrica during a sea voyage, Mahalanobis built the Indian Statistical Institute using personal funds and sheer determination. His approach demonstrated that institutional excellence didn't require government bureaucracy.

Mahalanobis possessed the characteristics of a driven academic: methodical, demanding, and confident in his abilities. Students had to maintain daily activity diaries and sign attendance registers. He expected precision in all matters and rarely engaged in small talk, reserving expressions of affection primarily for pets. Yet this rigor created an environment where world-class mathematics flourished.

His vindication came in 1944 when random sample surveys proved vastly superior to traditional enumeration methods. The Bengal government commissioned both approaches to measure jute production, with traditional counting proving 16.6% inaccurate while Mahalanobis's sampling achieved 0.3% accuracy at one-tenth the cost.

However, Mahalanobis exhibited the characteristic flaws of high modernism. Milton Friedman observed that mathematicians, accustomed to absolutes, become dangerous when applying themselves to economic planning. Hannah Arendt noted his authoritarian tendencies, describing him as someone who had completely bypassed democratic liberal notions. As one colleague noted:

"He thought he knew best in practically every matter."

This combination of genuine brilliance with authoritarian instincts reflects a broader pattern where technical expertise breeds overconfidence in complex social domains.

The transition from mathematical to computational statistics

The field of statistics underwent a fundamental transformation in the 1970s and 1980s that India's academic community largely missed. Traditional mathematical statistics focused on developing closed-form estimators that could be written as explicit formulas. This approach dominated the early decades when computers were unavailable or extremely limited.

The revolution began with innovations like Bradley Efron's bootstrap method in 1979, which represented a shift toward computational approaches. Instead of seeking mathematical formulas, statisticians began designing algorithms that could produce estimates through iterative computational processes. This opened vast new possibilities for handling complex data and previously unsolvable estimation problems.

Ajay describes this crucial shift:

"The unit of analysis stops being a mathematical estimator that can be written down in closed form; the estimator becomes an algorithm. And it opens up a whole set of possibilities for how we can design estimators."

India's statistics community remained wedded to theorem proving and mathematical analysis while the global frontier moved toward computational methods requiring integration of statistics with computer science. This institutional inertia meant that despite the country's early leadership in mathematical statistics, it failed to participate in the computational revolution that defines modern data science.

The failure illustrates broader challenges in Indian academic institutions. The same organizational structures that couldn't maintain the excellence Mahalanobis created also couldn't adapt to fundamental changes in their fields. Today's data science boom represents the flowering of computational statistics techniques that Indian institutions could have helped develop but instead watched from the sidelines.

The broader lessons about institutional excellence

The story of the Indian Statistical Institute offers insights about creating and maintaining intellectual excellence. Mahalanobis succeeded by operating outside bureaucratic structures, funding his vision personally, and maintaining uncompromising standards. The result was an institution that produced globally significant research and trained multiple generations of talented statisticians.

Yet this excellence proved fragile. Despite the remarkable initial achievements, ISI couldn't sustain its position at the global frontier. This pattern repeats across Indian public sector research and educational institutions, suggesting systematic management problems rather than isolated failures.

The contrast between individual genius and institutional sustainability appears throughout the episode. C.R. Rao continued his productive career well into his nineties, but the ecosystem that produced him couldn't replicate its success. Similarly, modern efforts like the C.R. Rao.jl statistical computing package represent attempts to honor that legacy while working with contemporary tools.

Ajay reflects on the fundamental challenge:

"Whether it's IITs or IISC, there's something about the modern Indian public sector, state-funded organization that is just incompatible with quality and excellence. We are not able to recreate what Mahalanobis and C.R. Rao were doing at that age."

The solution may lie not in building larger bureaucracies but in supporting more individuals like Mahalanobis who can create isolated centers of excellence. The lesson suggests that institutional innovation requires visionary individuals operating with substantial autonomy rather than systematic bureaucratic approaches to fostering excellence.

Supplementary Resources

• The Lady Tasting Tea by David Salsburg (Book)
• Statistics and Truth by C. Radhakrishna Rao (Book)
• The Theory of Moral Sentiments by Adam Smith (Book)
• The Road to Serfdom by Friedrich A. Hayek (Book)

The complete transcript file is available to download below.