Advertisement
Ingenuity
Ingenuity

Q&A: Why small is beautiful

Large sample sizes are not the be-all and end-all of clinical research.

By | June 15, 2011

WIKIMEDIA COMMONS

Early stage research often gets dinged for not including enough trial subjects to be statistically valid. But adhering to the large sample-size dogma is counterproductive, says Peter Bacchetti, a biostatistician at the University of California, San Francisco. Large sample sizes waste time on unsuccessful ideas as most early stage trials fail, and can even prevent innovative treatments from moving forward if trials that don't recruit enough patients are never performed, he argues in a perspectives piece published online today (June 15) in Science Translational Medicine.

This week, Bacchetti took time to speak with The Scientist about why sample size isn’t everything, and what scientists can use instead to measure a study’s worth.

The Scientist: Why does starting small make sense?

Peter Bacchetti: Because you don’t know what’s going to happen, and sometimes that means you can learn enough from something small, or have a better idea where to go next. Having a large study means spending a lot of money, it means taking a lot of time, and it can also mean taking a lot of risk. That’s not always justified right off the bat with a new idea.

Peter Bacchetti
Peter Bacchetti

TS: Why are sample size calculations often counterproductive?

PB: There is this idea that you must prove that you have 80 percent power in order for a study to be worth doing. That means given some particular assumption about what’s really true, you’ll have an 80 percent chance of getting a P-value of less than .05.

The goal of 80 percent power isn’t well justified. It’s not like it’s worthless until you reach 80 percent power, and then it’s good. Studies that have less than 80 percent power may still be very worthwhile and worth doing.

What I talk about in the article is the reality of diminishing marginal returns. If you add another subject, they produce less information than the one that came before them. So you get less and less for each additional person you add.

TS: How can a study be useful if doesn’t have statistical power?

PB: Sample size can often be a distraction from more important issues. Deciding whether a study is worth doing could depend on how interesting the idea is, whether the design is sound or susceptible to bias, what the potential upsides are if it turns out to be right.

TS: How does sample size play into later-stage trials?

PB: It makes sense to weigh the costs when you’re planning later studies, as well. In reality, investigators do. They don’t do a sample size calculation and then use that sample size regardless of how infeasible it may be. You can’t say “well, a thousand subjects, pretty good but it’s not quite significant under these assumptions, so double the cost of the study, go up to 1100 subjects.” That typically would not make sense.

TS: You mention in your paper that requirements of a certain sample size can often lead reviewers to reject proposals that may actually be worth funding. What should reviewers do instead?

PB: Well, we’ve recommended they assume sample size is a reasonable choice unless there’s a really extreme, obvious case. For example, if you’re studying something that’s so rare that you’re not even likely to see an instance of it in your study, then of course you’re not likely to learn anything.

Sometimes people will fall back to the argument that more is always better. Sample size calculation can’t be as simple as that, otherwise every study would include everyone on the planet.

P. Bacchetti et al., “Breaking free of sample size dogma to perform innovative translational research,” Science Translational Medicine, 3: 1-4, 2011.

Advertisement

Add a Comment

Avatar of: You

You

Processing...
Processing...

Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo

Comments

Avatar of: Rich Patrock

Rich Patrock

Posts: 1457

June 16, 2011

It might be better to have 20 small studies than 1 large study and do a meta-analysis.  Of course, I'm a Bayesian and argue that once you do a small sample you have a better idea of the probability of obtaining a specific result.  Based on that foreknowledge you have a different set of hypotheses to test than those with which you started.  Typically, one has an initial set of hypotheses and a trial results in some data.  Given these initial results, the 'objective' scientists find the ones that look most promising and go after those.  In an emergency situation where life or death is involved, then this approach could be warranted.  In other situations, it represents nothing more than cheating at solitaire.

Avatar of:

Posts: 0

June 16, 2011

It might be better to have 20 small studies than 1 large study and do a meta-analysis.  Of course, I'm a Bayesian and argue that once you do a small sample you have a better idea of the probability of obtaining a specific result.  Based on that foreknowledge you have a different set of hypotheses to test than those with which you started.  Typically, one has an initial set of hypotheses and a trial results in some data.  Given these initial results, the 'objective' scientists find the ones that look most promising and go after those.  In an emergency situation where life or death is involved, then this approach could be warranted.  In other situations, it represents nothing more than cheating at solitaire.

Avatar of:

Posts: 0

June 16, 2011

It might be better to have 20 small studies than 1 large study and do a meta-analysis.  Of course, I'm a Bayesian and argue that once you do a small sample you have a better idea of the probability of obtaining a specific result.  Based on that foreknowledge you have a different set of hypotheses to test than those with which you started.  Typically, one has an initial set of hypotheses and a trial results in some data.  Given these initial results, the 'objective' scientists find the ones that look most promising and go after those.  In an emergency situation where life or death is involved, then this approach could be warranted.  In other situations, it represents nothing more than cheating at solitaire.

Avatar of:

Posts: 0

June 17, 2011

Large sample sizes, particularly in human studies, also iron out the inherent variability in the human population. Drugs may work (or be toxic) in some people but not in others, because of different metabolism and genes , and genes may be associated with disease in some populations, but not in others, because of gene/gene and gene/environment interactions. Small sample sizes, but with more measurements to determine what conditions what, may be more useful.

Avatar of:

Posts: 0

June 17, 2011

Large sample sizes, particularly in human studies, also iron out the inherent variability in the human population. Drugs may work (or be toxic) in some people but not in others, because of different metabolism and genes , and genes may be associated with disease in some populations, but not in others, because of gene/gene and gene/environment interactions. Small sample sizes, but with more measurements to determine what conditions what, may be more useful.

Avatar of: Chris Carter

Chris Carter

Posts: 5

June 17, 2011

Large sample sizes, particularly in human studies, also iron out the inherent variability in the human population. Drugs may work (or be toxic) in some people but not in others, because of different metabolism and genes , and genes may be associated with disease in some populations, but not in others, because of gene/gene and gene/environment interactions. Small sample sizes, but with more measurements to determine what conditions what, may be more useful.

Avatar of: Will Dracup

Anonymous

June 20, 2011

"Large sample sizes waste time on unsuccessful ideas as most early stage
trials fail"
Oh dear. Most early stage trials fail _because_ the sample sizes were too small!  Particularly in experiments using the new life science technologies which generate vast amounts of data per sample, small samples sizes are the road to random results. This is why there are over 1000 published biomarkers for every FDA approved biomarker - and it's killing the reputation of biological science.

Avatar of:

Posts: 0

June 20, 2011

"Large sample sizes waste time on unsuccessful ideas as most early stage
trials fail"
Oh dear. Most early stage trials fail _because_ the sample sizes were too small!  Particularly in experiments using the new life science technologies which generate vast amounts of data per sample, small samples sizes are the road to random results. This is why there are over 1000 published biomarkers for every FDA approved biomarker - and it's killing the reputation of biological science.

Avatar of:

Posts: 0

June 20, 2011

"Large sample sizes waste time on unsuccessful ideas as most early stage
trials fail"
Oh dear. Most early stage trials fail _because_ the sample sizes were too small!  Particularly in experiments using the new life science technologies which generate vast amounts of data per sample, small samples sizes are the road to random results. This is why there are over 1000 published biomarkers for every FDA approved biomarker - and it's killing the reputation of biological science.

Follow The Scientist

icon-facebook icon-linkedin icon-twitter icon-vimeo icon-youtube
Advertisement

Stay Connected with The Scientist

  • icon-facebook The Scientist Magazine
  • icon-facebook The Scientist Careers
  • icon-facebook Neuroscience Research Techniques
  • icon-facebook Genetic Research Techniques
  • icon-facebook Cell Culture Techniques
  • icon-facebook Microbiology and Immunology
  • icon-facebook Cancer Research and Technology
  • icon-facebook Stem Cell and Regenerative Science
Advertisement
LI-COR
LI-COR
Advertisement
The Scientist
The Scientist
Life Technologies