Large-Scale Inference

Bradley Efron explains how to perform thousands of simultaneous estimates and tests, as required by new scientific technology.

Modern scientific technology (such as microarrays and fMRI machines) produces data in vast quantities. Bradley Efron explains the empirical Bayes methods that help make sense of a new statistical world. This is essential reading for professional statisticians and graduate students wishing to use and understand important new techniques like false discovery rates.We live in a new age for statistical inference, where modern scientific technology such as microarrays and fMRI machines routinely produce thousands and sometimes millions of parallel data sets, each with its own estimation or testing problem. Doing thousands of problems at once is more than repeated application of classical methods. Taking an empirical Bayes approach, Bradley Efron, inventor of the bootstrap, shows how information accrues across problems in a way that combines Bayesian and frequentist ideas. Estimation, testing and prediction blend in this framework, producing opportunities for new methodologies of increased power. New difficulties also arise, easily leading to flawed inferences. This book takes a careful look at both the promise and pitfalls of large-scale statistical inference, with particular attention to false discovery rates, the most successful of the new statistical techniques. Emphasis is on the inferential ideas underlying technical developments, illustrated using a large number of real examples.