Opinion: Confounded Cancer Markers

Prognostic signatures have become popular tools in cancer research, but it turns out signatures made of random genes are prognostic as well.

By | December 7, 2011

ISTOCK, ALENGO

Ethic guidelines drastically limit experiments on human subjects. Hence, the fundamental mechanisms of human diseases are mostly studied in vitro or in animal models. These are only substitutes for understanding human physiology and disease. Proving that a mechanism responsible for disease progression in a model system is also relevant to human diseases—not to mention then translating it into a new therapeutic—is a major bottleneck in biomedicine. In the end, only clinical interventions on human will bridge models and human disease.

One approach is to look for correlations. If you can show that patients with tumors expressing, for example, stem cell markers have a much worse prognosis than those without them, that would suggest that stem cells are involved in human disease progression. This line of thinking has long been popular in oncology because you need only access surgical specimens, some mRNA or protein marker, and a follow up of patients. And with the recent advent of efficient microarray screens, this approach has become all the rage, reducing the discovery of signatures, i.e. multi genes markers, to a nearly automatic procedure.

The signatures’ prognostic potential can then be tested instantly in genome-wide compendia of expression profiles for hundreds of human tumors, all available for free in the public domain. Besides stem cells markers, signatures linked to all sorts of biological mechanisms or states have been shown to be associated with human cancer outcome. Indeed, several new signatures are published every month in prominent journals.

But such correlations are not all that they seem. The accumulation of signatures with all sorts of biological meaning, but nearly identical prognostic values, already looked suspicious to us and others back in 2007. It seemed that every newly discovered signature was prognostic. We collected from the literature some signatures with as little connection to cancer as possible. We found, for example, a signature of the blood cells of Japanese patients who were told jokes after lunch, and a signature derived from the microarray analysis of the brains from mice that suffered social defeat. Both of these signatures were associated with breast cancer outcome by any statistical standards.

We then went back to published cancer signatures and found that 60 percent were no more prognostic than signatures made by picking up genes at random among the 21,000 human genes. The problem occurred with single gene markers, but became dramatic with multigenes signatures. A gene chosen at random already has roughly one in five chance of being prognostic; for signatures made of more than 100 genes, 90 percent are prognostic. How is this possible? We showed that in breast cancer the expression of a large fraction of the genome correlates with the proliferation rate, which is prognostic in this disease.

It took us four years and six rejections to get this work finally published in a computational biology journal (PLoS Comput Biol, 2011)—not the most efficient venue to reach the oncology community. Meanwhile, a steady stream of studies confounded by proliferation rates has appeared. This has to be said, one can no longer stay silent about the rather limited self-correction capability of the top tier publishing system (Cell, Nature Genetics, PNAS, etc.), which promoted these studies in the first place.

The oncogenomic-based literature has forgotten the pitfalls of non-specific effects and the value of negative controls. It is not enough to show that a signature is prognostic; biological conclusions may be drawn only if its prognostic value is specifically driven by the mechanism/state under investigation. Importantly, we question prognostic signatures as specific research tools, not as clinical guides: smoke does not drive fire, yet it is powerful indicator of when and where a fire is burning.

Vincent Detours is a researcher at the Université Libre de Bruxelles in Belgium.

Add a Comment

Avatar of: You

You

Processing...
Processing...

Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo

Comments

Avatar of: rusty94114

rusty94114

Posts: 9

December 7, 2011

This is really shocking -- not only that medical research has been misled by sloppy statistical interpretation, but that leading scientific journals have attempted to suppress the discovery of this error.

Avatar of: David Kessel

David Kessel

Posts: 1457

December 7, 2011

Perhaps another example of rushing into 'translational' studies when there is really nothing yet to translate

Avatar of: lifebiomedguru

lifebiomedguru

Posts: 4

December 7, 2011

There may be a few good reasons why the author's article was rejected so many times.

First, this article does a disservice by portraying those of us who have been
involved in biomarker development studies as semi-morons who are not
capable or willing to use the scientific method to validate our
findings.

Second, how many random genes should be expected to be associated with cancer outcome is surely not sufficiently addressed by these studies.  But it is also not sufficiently addressed by this article.  Everyone knows that when you do multiple hypothesis testing with microarrays that SOMETHING will appear to be significant, so how many should be expected?  That does not mean such studies are useless, as is clearly implied by the author of this article.  

Third, proper phased biomarker development study designs exist to rule out those findings that do not validate.  The routine is Discovery, Evaluation, and Validation.  No signature found to not generalize to samples from patients collected and run at different times in the same medical center will likely be pushed forward.  This is built into phased biomarker development study designs.  No signature found to not validate across different centers is likely to proceed to clinical use.   This is also built into phased study designs.  Most people doing discovery phase studies know they are doing discovery phase research.

Granted, the discovery phase is messy, in part due to a lack of deep understanding that FDR control methods such as B-H do not accurately control FDR, and that reporting an estimate of the FDR is preferred to direct attempts to control.  It's also messy due to a profound lack of efforts to standardize methods for data representation (normalization and transformation) and the continue use of silly measures like fold-change to measure differences.  So why not study methods iteratively within each study to see if they provide internally consistent results (e.g., Jordan et al., 2008)?  Still, if a clinical research team has a panel of putatively prognostic biomarkers, and an associated algorithm, they should be given the chance to further evaluate their generalizability with additional data from their own centers, and if that goes well, validate them in a multi-center trial.  Then they can do a VDS (Voluntary Data Submission) to the FDA (probably should include data from all phases).  The re-analysis of individual data sets at the discovery phase alone from studies like this sheds little light on the value of the entire, mature process of biomarker development.

[Jordan R,
Patel S, Hu H, Lyons-Weiler J. 2008. Efficiency analysis of competing tests for finding
differentially expressed genes in lung adenocarcinoma.Cancer Inform. 2008;6:389-421.]

BTW, and fourth, the article uses the term 'confounded' wrong.  The problem the author is studying is Type I Error Rate Inflation due to Multiple Hypothesis Testing, which occurs with, or without, true confounding.  The jokes told to Japanese during lunch cannot possibly be a true confounding variable. unless it somehow influences the gene expression profiles of breast cancer patients.  Age (old cancer px vs. young normal controls) would be a possible confounding variable because it introduces bias.  Some genes appearing different because they are randomly selected could in fact reflect massive genomic shifts associated with cancer.

While we're busy promoting our own literature, I might hasten to add that in
my opinion, people doing -omic based prognostic modeling should avoid
using the log-rank test to test the 'significance' of their Kaplan-Meier curves.  The log-rank test not a sufficiently critical test in this application, and does
not track with model accuracy as well as another test called the F* test
(Berty et al., 2010).  Professional critiques have had fun from the sidelines and have sidetracked a lot of good effort.  Rather than write a scathing critique against straw men, perhaps some effort should be made into identifying weaknesses in a paradigm and making a contribution with a more viable, robust alternative.

[Berty HP, Shi H, Lyons-Weiler J. 2010. Determining the statistical significance of survivorship
prediction models. J Eval Clin Pract. 16(1):155-65.]

Finally, and most importantly, there is methodology to assess how much better a cancer signature is from randomly selected genes, e.g.

[Lyons-Weiler, J. R. Pelikan, H.J. Zeh III, D.C.
Whitcomb, D.E. Malehorn, W.L. Bigbee and M. Hauskrecht. 2005. Assessing the statistical significance of the achieved classification error of classifiers constructed using serum peptide profiles and a prescription
for random resampling repeated studies for massive high-throughput genomic and proteomic studies, Cancer Informatics,1:1 53-77.]

http://www.la-press.com/assess...

Anyone doing biomarker development should choose random sets of gene/protein measurements and feed them into the model optimization via cross-validation to generate a distribution of performance evaluation measures to see if, for their study, a problem exists with random gene selection.

For additional information on how far we've come in understanding the complexities of biomarker research, I still recommend David Ransohoff's occasional articles on the topic.
 

Avatar of: djadams

djadams

Posts: 3

December 7, 2011

Thank you for creating awareness of this issue.  Gene signatures for anticancer drug response are very popular now as well.  As a pharmacologist, I have been skeptical of how such signatures could predict drug ADMET, especially in solid tumors with impaired vasculature.  Yet the work appears in top tier journals and platform presentations at major meetings.  Meanwhile, standard but informative drug PK/PD studies are relegated to the poster sessions.

Avatar of: Yassine Rihane

Yassine Rihane

Posts: 1457

December 7, 2011

Thank you for creating awareness of this issue !<h1>sobakawa
pillow
</h1>

Avatar of:

Posts: 0

December 7, 2011

This is really shocking -- not only that medical research has been misled by sloppy statistical interpretation, but that leading scientific journals have attempted to suppress the discovery of this error.

Avatar of:

Posts: 0

December 7, 2011

Perhaps another example of rushing into 'translational' studies when there is really nothing yet to translate

Avatar of:

Posts: 0

December 7, 2011

There may be a few good reasons why the author's article was rejected so many times.

First, this article does a disservice by portraying those of us who have been
involved in biomarker development studies as semi-morons who are not
capable or willing to use the scientific method to validate our
findings.

Second, how many random genes should be expected to be associated with cancer outcome is surely not sufficiently addressed by these studies.  But it is also not sufficiently addressed by this article.  Everyone knows that when you do multiple hypothesis testing with microarrays that SOMETHING will appear to be significant, so how many should be expected?  That does not mean such studies are useless, as is clearly implied by the author of this article.  

Third, proper phased biomarker development study designs exist to rule out those findings that do not validate.  The routine is Discovery, Evaluation, and Validation.  No signature found to not generalize to samples from patients collected and run at different times in the same medical center will likely be pushed forward.  This is built into phased biomarker development study designs.  No signature found to not validate across different centers is likely to proceed to clinical use.   This is also built into phased study designs.  Most people doing discovery phase studies know they are doing discovery phase research.

Granted, the discovery phase is messy, in part due to a lack of deep understanding that FDR control methods such as B-H do not accurately control FDR, and that reporting an estimate of the FDR is preferred to direct attempts to control.  It's also messy due to a profound lack of efforts to standardize methods for data representation (normalization and transformation) and the continue use of silly measures like fold-change to measure differences.  So why not study methods iteratively within each study to see if they provide internally consistent results (e.g., Jordan et al., 2008)?  Still, if a clinical research team has a panel of putatively prognostic biomarkers, and an associated algorithm, they should be given the chance to further evaluate their generalizability with additional data from their own centers, and if that goes well, validate them in a multi-center trial.  Then they can do a VDS (Voluntary Data Submission) to the FDA (probably should include data from all phases).  The re-analysis of individual data sets at the discovery phase alone from studies like this sheds little light on the value of the entire, mature process of biomarker development.

[Jordan R,
Patel S, Hu H, Lyons-Weiler J. 2008. Efficiency analysis of competing tests for finding
differentially expressed genes in lung adenocarcinoma.Cancer Inform. 2008;6:389-421.]

BTW, and fourth, the article uses the term 'confounded' wrong.  The problem the author is studying is Type I Error Rate Inflation due to Multiple Hypothesis Testing, which occurs with, or without, true confounding.  The jokes told to Japanese during lunch cannot possibly be a true confounding variable. unless it somehow influences the gene expression profiles of breast cancer patients.  Age (old cancer px vs. young normal controls) would be a possible confounding variable because it introduces bias.  Some genes appearing different because they are randomly selected could in fact reflect massive genomic shifts associated with cancer.

While we're busy promoting our own literature, I might hasten to add that in
my opinion, people doing -omic based prognostic modeling should avoid
using the log-rank test to test the 'significance' of their Kaplan-Meier curves.  The log-rank test not a sufficiently critical test in this application, and does
not track with model accuracy as well as another test called the F* test
(Berty et al., 2010).  Professional critiques have had fun from the sidelines and have sidetracked a lot of good effort.  Rather than write a scathing critique against straw men, perhaps some effort should be made into identifying weaknesses in a paradigm and making a contribution with a more viable, robust alternative.

[Berty HP, Shi H, Lyons-Weiler J. 2010. Determining the statistical significance of survivorship
prediction models. J Eval Clin Pract. 16(1):155-65.]

Finally, and most importantly, there is methodology to assess how much better a cancer signature is from randomly selected genes, e.g.

[Lyons-Weiler, J. R. Pelikan, H.J. Zeh III, D.C.
Whitcomb, D.E. Malehorn, W.L. Bigbee and M. Hauskrecht. 2005. Assessing the statistical significance of the achieved classification error of classifiers constructed using serum peptide profiles and a prescription
for random resampling repeated studies for massive high-throughput genomic and proteomic studies, Cancer Informatics,1:1 53-77.]

http://www.la-press.com/assess...

Anyone doing biomarker development should choose random sets of gene/protein measurements and feed them into the model optimization via cross-validation to generate a distribution of performance evaluation measures to see if, for their study, a problem exists with random gene selection.

For additional information on how far we've come in understanding the complexities of biomarker research, I still recommend David Ransohoff's occasional articles on the topic.
 

Avatar of:

Posts: 0

December 7, 2011

Thank you for creating awareness of this issue.  Gene signatures for anticancer drug response are very popular now as well.  As a pharmacologist, I have been skeptical of how such signatures could predict drug ADMET, especially in solid tumors with impaired vasculature.  Yet the work appears in top tier journals and platform presentations at major meetings.  Meanwhile, standard but informative drug PK/PD studies are relegated to the poster sessions.

Avatar of:

Posts: 0

December 7, 2011

Thank you for creating awareness of this issue !<h1>sobakawa
pillow
</h1>

Avatar of:

Posts: 0

December 7, 2011

This is really shocking -- not only that medical research has been misled by sloppy statistical interpretation, but that leading scientific journals have attempted to suppress the discovery of this error.

Avatar of:

Posts: 0

December 7, 2011

Perhaps another example of rushing into 'translational' studies when there is really nothing yet to translate

Avatar of:

Posts: 0

December 7, 2011

There may be a few good reasons why the author's article was rejected so many times.

First, this article does a disservice by portraying those of us who have been
involved in biomarker development studies as semi-morons who are not
capable or willing to use the scientific method to validate our
findings.

Second, how many random genes should be expected to be associated with cancer outcome is surely not sufficiently addressed by these studies.  But it is also not sufficiently addressed by this article.  Everyone knows that when you do multiple hypothesis testing with microarrays that SOMETHING will appear to be significant, so how many should be expected?  That does not mean such studies are useless, as is clearly implied by the author of this article.  

Third, proper phased biomarker development study designs exist to rule out those findings that do not validate.  The routine is Discovery, Evaluation, and Validation.  No signature found to not generalize to samples from patients collected and run at different times in the same medical center will likely be pushed forward.  This is built into phased biomarker development study designs.  No signature found to not validate across different centers is likely to proceed to clinical use.   This is also built into phased study designs.  Most people doing discovery phase studies know they are doing discovery phase research.

Granted, the discovery phase is messy, in part due to a lack of deep understanding that FDR control methods such as B-H do not accurately control FDR, and that reporting an estimate of the FDR is preferred to direct attempts to control.  It's also messy due to a profound lack of efforts to standardize methods for data representation (normalization and transformation) and the continue use of silly measures like fold-change to measure differences.  So why not study methods iteratively within each study to see if they provide internally consistent results (e.g., Jordan et al., 2008)?  Still, if a clinical research team has a panel of putatively prognostic biomarkers, and an associated algorithm, they should be given the chance to further evaluate their generalizability with additional data from their own centers, and if that goes well, validate them in a multi-center trial.  Then they can do a VDS (Voluntary Data Submission) to the FDA (probably should include data from all phases).  The re-analysis of individual data sets at the discovery phase alone from studies like this sheds little light on the value of the entire, mature process of biomarker development.

[Jordan R,
Patel S, Hu H, Lyons-Weiler J. 2008. Efficiency analysis of competing tests for finding
differentially expressed genes in lung adenocarcinoma.Cancer Inform. 2008;6:389-421.]

BTW, and fourth, the article uses the term 'confounded' wrong.  The problem the author is studying is Type I Error Rate Inflation due to Multiple Hypothesis Testing, which occurs with, or without, true confounding.  The jokes told to Japanese during lunch cannot possibly be a true confounding variable. unless it somehow influences the gene expression profiles of breast cancer patients.  Age (old cancer px vs. young normal controls) would be a possible confounding variable because it introduces bias.  Some genes appearing different because they are randomly selected could in fact reflect massive genomic shifts associated with cancer.

While we're busy promoting our own literature, I might hasten to add that in
my opinion, people doing -omic based prognostic modeling should avoid
using the log-rank test to test the 'significance' of their Kaplan-Meier curves.  The log-rank test not a sufficiently critical test in this application, and does
not track with model accuracy as well as another test called the F* test
(Berty et al., 2010).  Professional critiques have had fun from the sidelines and have sidetracked a lot of good effort.  Rather than write a scathing critique against straw men, perhaps some effort should be made into identifying weaknesses in a paradigm and making a contribution with a more viable, robust alternative.

[Berty HP, Shi H, Lyons-Weiler J. 2010. Determining the statistical significance of survivorship
prediction models. J Eval Clin Pract. 16(1):155-65.]

Finally, and most importantly, there is methodology to assess how much better a cancer signature is from randomly selected genes, e.g.

[Lyons-Weiler, J. R. Pelikan, H.J. Zeh III, D.C.
Whitcomb, D.E. Malehorn, W.L. Bigbee and M. Hauskrecht. 2005. Assessing the statistical significance of the achieved classification error of classifiers constructed using serum peptide profiles and a prescription
for random resampling repeated studies for massive high-throughput genomic and proteomic studies, Cancer Informatics,1:1 53-77.]

http://www.la-press.com/assess...

Anyone doing biomarker development should choose random sets of gene/protein measurements and feed them into the model optimization via cross-validation to generate a distribution of performance evaluation measures to see if, for their study, a problem exists with random gene selection.

For additional information on how far we've come in understanding the complexities of biomarker research, I still recommend David Ransohoff's occasional articles on the topic.
 

Avatar of:

Posts: 0

December 7, 2011

Thank you for creating awareness of this issue.  Gene signatures for anticancer drug response are very popular now as well.  As a pharmacologist, I have been skeptical of how such signatures could predict drug ADMET, especially in solid tumors with impaired vasculature.  Yet the work appears in top tier journals and platform presentations at major meetings.  Meanwhile, standard but informative drug PK/PD studies are relegated to the poster sessions.

Avatar of:

Posts: 0

December 7, 2011

Thank you for creating awareness of this issue !<h1>sobakawa
pillow
</h1>

Avatar of: mlerman

mlerman

Posts: 4

December 7, 2011

I never liked this "signature" business and their association with disease states, especially as prognostic markers of cancer progression/metastasis and therefore prognosis. As correctly mentioned in this short assay, the sequential silencing of TSG will not be in the"signature". Once again, the "omics" driven "new" concepts and applications are just tools in the war for monies. And no sign of a "scientific revolution"
Michael Lerman, M.D., Ph.D.

Avatar of: Robert Hurst

Robert Hurst

Posts: 1

December 7, 2011

The problem is not "omics". The problem is their simplistic use by investigators who forget that (a) it's a network, stupid, not pathways, and therefore those 40,000 measurements are not independent and (b) there are many biological variables other than those directly associated with survival encapsulated in those measurements that will make large gene signatures not robust. These failures were noted at least as far back as 2005 in a Bioinformatics paper by an Israeli group (L. Ein-Dor, et al. PMID:15308542).   

Avatar of:

Posts: 0

December 7, 2011

I never liked this "signature" business and their association with disease states, especially as prognostic markers of cancer progression/metastasis and therefore prognosis. As correctly mentioned in this short assay, the sequential silencing of TSG will not be in the"signature". Once again, the "omics" driven "new" concepts and applications are just tools in the war for monies. And no sign of a "scientific revolution"
Michael Lerman, M.D., Ph.D.

Avatar of:

Posts: 0

December 7, 2011

The problem is not "omics". The problem is their simplistic use by investigators who forget that (a) it's a network, stupid, not pathways, and therefore those 40,000 measurements are not independent and (b) there are many biological variables other than those directly associated with survival encapsulated in those measurements that will make large gene signatures not robust. These failures were noted at least as far back as 2005 in a Bioinformatics paper by an Israeli group (L. Ein-Dor, et al. PMID:15308542).   

Avatar of:

Posts: 0

December 7, 2011

I never liked this "signature" business and their association with disease states, especially as prognostic markers of cancer progression/metastasis and therefore prognosis. As correctly mentioned in this short assay, the sequential silencing of TSG will not be in the"signature". Once again, the "omics" driven "new" concepts and applications are just tools in the war for monies. And no sign of a "scientific revolution"
Michael Lerman, M.D., Ph.D.

Avatar of:

Posts: 0

December 7, 2011

The problem is not "omics". The problem is their simplistic use by investigators who forget that (a) it's a network, stupid, not pathways, and therefore those 40,000 measurements are not independent and (b) there are many biological variables other than those directly associated with survival encapsulated in those measurements that will make large gene signatures not robust. These failures were noted at least as far back as 2005 in a Bioinformatics paper by an Israeli group (L. Ein-Dor, et al. PMID:15308542).   

Avatar of:

Posts: 0

December 8, 2011

Thank you for this eye opening information. I see what a waste reseach is. I have seen some engineers make simalar mistakes.

Avatar of:

Posts: 0

December 8, 2011

I'm not sure that this is as big a deal breaker as it appears on the surface.  I suspect that what is going on is the following.  Suppose that there was a part of biology that affected a very large number of genes in breast cancer patients, and which was also associated with survival, say for example proliferation. If you took a large random selection of genes that were not coordinately expressed, and formed a principal component it is likely that this would focus on the little bit that the genes have in common and result in a version of the proliferation signature, although most of the genes in the signature would have low correlation with this
signature summary.  If you then tested this you would find that it is associated with survival.   So the signature’s association with survival is real it’s just not what you thought it was.  The key is to not take the signature as given but to make sure that the genes in the signature are correlated with each other and so represent a specific biology rather than a random collection of genes, and
then to further check external validation to make sure that the biology fits the
signature identification

Avatar of:

Posts: 0

December 8, 2011

Thank you for this eye opening information. I see what a waste reseach is. I have seen some engineers make simalar mistakes.

Avatar of:

Posts: 0

December 8, 2011

I'm not sure that this is as big a deal breaker as it appears on the surface.  I suspect that what is going on is the following.  Suppose that there was a part of biology that affected a very large number of genes in breast cancer patients, and which was also associated with survival, say for example proliferation. If you took a large random selection of genes that were not coordinately expressed, and formed a principal component it is likely that this would focus on the little bit that the genes have in common and result in a version of the proliferation signature, although most of the genes in the signature would have low correlation with this
signature summary.  If you then tested this you would find that it is associated with survival.   So the signature’s association with survival is real it’s just not what you thought it was.  The key is to not take the signature as given but to make sure that the genes in the signature are correlated with each other and so represent a specific biology rather than a random collection of genes, and
then to further check external validation to make sure that the biology fits the
signature identification

Avatar of: primativewriter

primativewriter

Posts: 22

December 8, 2011

Thank you for this eye opening information. I see what a waste reseach is. I have seen some engineers make simalar mistakes.

Avatar of: George Wight

George Wight

Posts: 1

December 8, 2011

I'm not sure that this is as big a deal breaker as it appears on the surface.  I suspect that what is going on is the following.  Suppose that there was a part of biology that affected a very large number of genes in breast cancer patients, and which was also associated with survival, say for example proliferation. If you took a large random selection of genes that were not coordinately expressed, and formed a principal component it is likely that this would focus on the little bit that the genes have in common and result in a version of the proliferation signature, although most of the genes in the signature would have low correlation with this
signature summary.  If you then tested this you would find that it is associated with survival.   So the signature’s association with survival is real it’s just not what you thought it was.  The key is to not take the signature as given but to make sure that the genes in the signature are correlated with each other and so represent a specific biology rather than a random collection of genes, and
then to further check external validation to make sure that the biology fits the
signature identification

Avatar of:

Posts: 0

December 9, 2011

The simple and necessary solution is to forfeit "signatures" when deciding what to do with a cancer patient who is desperately asking to help him/her.

Avatar of:

Posts: 0

December 9, 2011

The simple and necessary solution is to forfeit "signatures" when deciding what to do with a cancer patient who is desperately asking to help him/her.

Avatar of: mlerman

mlerman

Posts: 4

December 9, 2011

The simple and necessary solution is to forfeit "signatures" when deciding what to do with a cancer patient who is desperately asking to help him/her.

Avatar of:

Posts: 0

December 12, 2011

It is perhaps useful to insist we are not
questioning the
prognostic value of published signatures. On the contrary, Fig. 6
or our article confirms that most of them are associated with
outcome and that these associations are reproducible across
cohorts. Our paper investigate *why* signatures work.  This
matters little to patients, but it as consequences for basic
cancer research.  It's common place to suggest that because a
marker for a given biological process is prognostic in cancer,
then this process must be involved in breast cancer progression.
Our study reduces this argument ad absurdum by showing that
random signatures are prognostic too (Fig. 1 in the paper).

Contrary to what lifebiomedguru says in a previous thread, the
term 'confounded signatures' is appropriate. There are thousands
genes correlated with outcome, as shown by Ein-Dor et al.,
PMID:15308542.  We showed in our study that the overwhelming
majority of these genes are correlated with a proliferation
metagene. In addition, the vast majority of the published
signatures loose their prognostic value if you adjust the data
for proliferation.  Thus, 1- genes picked up at random are likely
to be prognostic. 2- if, for example, an hypoxia signature is
prognostic, but loose its prognostic value after adjustment for
proliferation, can you claim that its prognostic value supports
the role of hypoxia in cancer progression? No, because you just
can't determine from such evidence wether hypoxia or any of the
many processes statistically associated with proliferation in a
complex tissue cause the correlation with disease progression.

Note that our study addresses breast cancer in which
proliferation is prognostic, it does not necessarily apply to all
cancers.

lifebiomedguru raises the issue of multiple testing.  Our study
is not about multiple testing situations. We are estimating the
probability that a paper presenting a **single** signature finds
a significant association with outcome. Consider for example this
typical situation: Dr. Smith derives a stem cell signature and
finds it to be associated with outcome. The question we asked is:
what is the probability that Dr. Smith would find the same result
if he completely mess up his stem cell experiment and his
signature is in fact a random set of genes? I.e. we are
interested in the proportion of random genes sets that are
significantly associated with outcome.  It is estimated in
Figs. 1 and 2 of our paper: if his signature is >100 genes, Dr
Smith as a 90% chance to find an significant association even if
his signature is made of random genes. Therefore, Dr Smith cannot
draw any biological conclusion from the fact that his stem cell
signature is associated with outcome.  Lifebiomedguru's comment
is relevant to studies evaluating several markers. The q-value
calculation for single gene markers in our paper addresses it:
26% of the genes are associated with outcome at p<0.05 and 17% at
multiple testing-adjusted q<0.05. The percentages will only be
higher for multi-genes markers (relevant to studies investigating
a collection of multi-genes signatures).

We agree with the thread of George Wight, except that having the
genes correlated with one another is not enough. Supplementary
informations of the paper present PCA for individual
signatures. Several have PC1 explaining most of the variance, but
it's highly correlated with proliferation.  Thus one needs also
to rule out the confounding effect of proliferation.  This confounder
has been recognized by several authors, but the method they used
to rule it out does not work (Fig. 5 of our paper)

Finally, lifebiomedguru wrote "this article does a disservice by
portraying those of us who have been involved in biomarker
development studies as semi-morons who are not capable or willing
to use the scientific method to validate our findings." I think
being wrong is a risk inherent to innovative science.  But the
community has to recognize errors, it's no disservice to point
them out.  As noted by Robert Hurst in this discussion, the major
conclusions of our study are really logical consequences of the
findings of Ein-Dor et al. 2005, PMID:15308542, Wirapati et
al. 2008, PMID:18662380, and others. A number of us have been
aware of the issue. But, judging from the biological conclusions
that continue to be drawn from prognostic breast cancer markers
on a monthly basis in top journals, many researchers have not
measured the implications of these earlier papers.  Someone had
to pin them down more explicitly. I commented about our
difficulty to publish this study because I realized others had
similar experiences.  Read, for example, this story about the
statisticians who scrutinized the Potti/Nevins Duke data,
http://www.sciencemag.org/cont...

Vincent Detours

Avatar of:

Posts: 0

December 12, 2011

dsda

Avatar of: Someone

Someone

Posts: 1457

December 12, 2011

It is perhaps useful to insist we are not
questioning the
prognostic value of published signatures. On the contrary, Fig. 6
or our article confirms that most of them are associated with
outcome and that these associations are reproducible across
cohorts. Our paper investigate *why* signatures work.  This
matters little to patients, but it as consequences for basic
cancer research.  It's common place to suggest that because a
marker for a given biological process is prognostic in cancer,
then this process must be involved in breast cancer progression.
Our study reduces this argument ad absurdum by showing that
random signatures are prognostic too (Fig. 1 in the paper).

Contrary to what lifebiomedguru says in a previous thread, the
term 'confounded signatures' is appropriate. There are thousands
genes correlated with outcome, as shown by Ein-Dor et al.,
PMID:15308542.  We showed in our study that the overwhelming
majority of these genes are correlated with a proliferation
metagene. In addition, the vast majority of the published
signatures loose their prognostic value if you adjust the data
for proliferation.  Thus, 1- genes picked up at random are likely
to be prognostic. 2- if, for example, an hypoxia signature is
prognostic, but loose its prognostic value after adjustment for
proliferation, can you claim that its prognostic value supports
the role of hypoxia in cancer progression? No, because you just
can't determine from such evidence wether hypoxia or any of the
many processes statistically associated with proliferation in a
complex tissue cause the correlation with disease progression.

Note that our study addresses breast cancer in which
proliferation is prognostic, it does not necessarily apply to all
cancers.

lifebiomedguru raises the issue of multiple testing.  Our study
is not about multiple testing situations. We are estimating the
probability that a paper presenting a **single** signature finds
a significant association with outcome. Consider for example this
typical situation: Dr. Smith derives a stem cell signature and
finds it to be associated with outcome. The question we asked is:
what is the probability that Dr. Smith would find the same result
if he completely mess up his stem cell experiment and his
signature is in fact a random set of genes? I.e. we are
interested in the proportion of random genes sets that are
significantly associated with outcome.  It is estimated in
Figs. 1 and 2 of our paper: if his signature is >100 genes, Dr
Smith as a 90% chance to find an significant association even if
his signature is made of random genes. Therefore, Dr Smith cannot
draw any biological conclusion from the fact that his stem cell
signature is associated with outcome.  Lifebiomedguru's comment
is relevant to studies evaluating several markers. The q-value
calculation for single gene markers in our paper addresses it:
26% of the genes are associated with outcome at p<0.05 and 17% at
multiple testing-adjusted q<0.05. The percentages will only be
higher for multi-genes markers (relevant to studies investigating
a collection of multi-genes signatures).

We agree with the thread of George Wight, except that having the
genes correlated with one another is not enough. Supplementary
informations of the paper present PCA for individual
signatures. Several have PC1 explaining most of the variance, but
it's highly correlated with proliferation.  Thus one needs also
to rule out the confounding effect of proliferation.  This confounder
has been recognized by several authors, but the method they used
to rule it out does not work (Fig. 5 of our paper)

Finally, lifebiomedguru wrote "this article does a disservice by
portraying those of us who have been involved in biomarker
development studies as semi-morons who are not capable or willing
to use the scientific method to validate our findings." I think
being wrong is a risk inherent to innovative science.  But the
community has to recognize errors, it's no disservice to point
them out.  As noted by Robert Hurst in this discussion, the major
conclusions of our study are really logical consequences of the
findings of Ein-Dor et al. 2005, PMID:15308542, Wirapati et
al. 2008, PMID:18662380, and others. A number of us have been
aware of the issue. But, judging from the biological conclusions
that continue to be drawn from prognostic breast cancer markers
on a monthly basis in top journals, many researchers have not
measured the implications of these earlier papers.  Someone had
to pin them down more explicitly. I commented about our
difficulty to publish this study because I realized others had
similar experiences.  Read, for example, this story about the
statisticians who scrutinized the Potti/Nevins Duke data,
http://www.sciencemag.org/cont...

Vincent Detours

Avatar of: Someone

Someone

Posts: 1457

December 12, 2011

dsda

Avatar of:

Posts: 0

December 12, 2011

It is perhaps useful to insist we are not
questioning the
prognostic value of published signatures. On the contrary, Fig. 6
or our article confirms that most of them are associated with
outcome and that these associations are reproducible across
cohorts. Our paper investigate *why* signatures work.  This
matters little to patients, but it as consequences for basic
cancer research.  It's common place to suggest that because a
marker for a given biological process is prognostic in cancer,
then this process must be involved in breast cancer progression.
Our study reduces this argument ad absurdum by showing that
random signatures are prognostic too (Fig. 1 in the paper).

Contrary to what lifebiomedguru says in a previous thread, the
term 'confounded signatures' is appropriate. There are thousands
genes correlated with outcome, as shown by Ein-Dor et al.,
PMID:15308542.  We showed in our study that the overwhelming
majority of these genes are correlated with a proliferation
metagene. In addition, the vast majority of the published
signatures loose their prognostic value if you adjust the data
for proliferation.  Thus, 1- genes picked up at random are likely
to be prognostic. 2- if, for example, an hypoxia signature is
prognostic, but loose its prognostic value after adjustment for
proliferation, can you claim that its prognostic value supports
the role of hypoxia in cancer progression? No, because you just
can't determine from such evidence wether hypoxia or any of the
many processes statistically associated with proliferation in a
complex tissue cause the correlation with disease progression.

Note that our study addresses breast cancer in which
proliferation is prognostic, it does not necessarily apply to all
cancers.

lifebiomedguru raises the issue of multiple testing.  Our study
is not about multiple testing situations. We are estimating the
probability that a paper presenting a **single** signature finds
a significant association with outcome. Consider for example this
typical situation: Dr. Smith derives a stem cell signature and
finds it to be associated with outcome. The question we asked is:
what is the probability that Dr. Smith would find the same result
if he completely mess up his stem cell experiment and his
signature is in fact a random set of genes? I.e. we are
interested in the proportion of random genes sets that are
significantly associated with outcome.  It is estimated in
Figs. 1 and 2 of our paper: if his signature is >100 genes, Dr
Smith as a 90% chance to find an significant association even if
his signature is made of random genes. Therefore, Dr Smith cannot
draw any biological conclusion from the fact that his stem cell
signature is associated with outcome.  Lifebiomedguru's comment
is relevant to studies evaluating several markers. The q-value
calculation for single gene markers in our paper addresses it:
26% of the genes are associated with outcome at p<0.05 and 17% at
multiple testing-adjusted q<0.05. The percentages will only be
higher for multi-genes markers (relevant to studies investigating
a collection of multi-genes signatures).

We agree with the thread of George Wight, except that having the
genes correlated with one another is not enough. Supplementary
informations of the paper present PCA for individual
signatures. Several have PC1 explaining most of the variance, but
it's highly correlated with proliferation.  Thus one needs also
to rule out the confounding effect of proliferation.  This confounder
has been recognized by several authors, but the method they used
to rule it out does not work (Fig. 5 of our paper)

Finally, lifebiomedguru wrote "this article does a disservice by
portraying those of us who have been involved in biomarker
development studies as semi-morons who are not capable or willing
to use the scientific method to validate our findings." I think
being wrong is a risk inherent to innovative science.  But the
community has to recognize errors, it's no disservice to point
them out.  As noted by Robert Hurst in this discussion, the major
conclusions of our study are really logical consequences of the
findings of Ein-Dor et al. 2005, PMID:15308542, Wirapati et
al. 2008, PMID:18662380, and others. A number of us have been
aware of the issue. But, judging from the biological conclusions
that continue to be drawn from prognostic breast cancer markers
on a monthly basis in top journals, many researchers have not
measured the implications of these earlier papers.  Someone had
to pin them down more explicitly. I commented about our
difficulty to publish this study because I realized others had
similar experiences.  Read, for example, this story about the
statisticians who scrutinized the Potti/Nevins Duke data,
http://www.sciencemag.org/cont...

Vincent Detours

Avatar of:

Posts: 0

December 12, 2011

dsda

Avatar of:

Posts: 0

December 13, 2011

It is perhaps useful to insist we are not questioning the
prognostic value of published signatures. On the contrary, Fig. 6
or our article confirms that most of them are associated with
outcome and that these associations are reproducible across
cohorts. Our paper investigate *why* signatures work.  This
matters little to patients, but it as consequences for basic
cancer research.  It's common place to suggest that because a
marker for a given biological process is prognostic in cancer,
then this process must be involved in breast cancer progression.
Our study reduces this argument ad absurdum by showing that
random signatures are prognostic too (Fig. 1 in the paper).

Contrary to what lifebiomedguru says in a previous thread, the
term 'confounded signatures' is appropriate. There are thousands
genes correlated with outcome, as shown by Ein-Dor et al.,
PMID:15308542.  We showed in our study that the overwhelming
majority of these genes are correlated with a proliferation
metagene. In addition, the vast majority of the published
signatures loose their prognostic value if you adjust the data
for proliferation.  Thus, 1- genes picked up at random are likely
to be prognostic. 2- if, for example, an hypoxia signature is
prognostic, but loose its prognostic value after adjustment for
proliferation, can you claim that its prognostic value supports
the role of hypoxia in cancer progression? No, because you just
can't determine from such evidence wether hypoxia or any of the
many processes statistically associated with proliferation in a
complex tissue cause the correlation with disease progression.

Note that our study addresses breast cancer in which
proliferation is prognostic, it does not necessarily apply to all
cancers.

lifebiomedguru raises the issue of multiple testing.  Our study
is not about multiple testing situations. We are estimating the
probability that a paper presenting a **single** signature finds
a significant association with outcome. Consider for example this
typical situation: Dr. Smith derives a stem cell signature and
finds it to be associated with outcome. The question we asked is:
what is the probability that Dr. Smith would find the same result
if he completely mess up his stem cell experiment and his
signature is in fact a random set of genes? I.e. we are
interested in the proportion of random genes sets that are
significantly associated with outcome.  It is estimated in
Figs. 1 and 2 of our paper: if his signature is >100 genes, Dr
Smith as a 90% chance to find an significant association even if
his signature is made of random genes. Therefore, Dr Smith cannot
draw any biological conclusion from the fact that his stem cell
signature is associated with outcome.  Lifebiomedguru's comment
is relevant to studies evaluating several markers. The q-value
calculation for single gene markers in our paper addresses it:
26% of the genes are associated with outcome at p<0.05 and 17% at
multiple testing-adjusted q<0.05. The percentages will only be
higher for multi-genes markers (relevant to studies investigating
a collection of multi-genes signatures).

We agree with the thread of George Wight, except that having the
genes correlated with one another is not enough. Supplementary
informations of the paper present PCA for individual
signatures. Several have PC1 explaining most of the variance, but
it's highly correlated with proliferation.  Thus one needs also
to rule out the confounding effect of proliferation.  This confounder
has been recognized by several authors, but the method they used
to rule it out does not work (Fig. 5 of our paper)

Finally, lifebiomedguru wrote "There may be a few good reasons
why the author's article was rejected so many times. First, this
article does a disservice by portraying those of us who have been
involved in biomarker development studies as semi-morons who are
not capable or willing to use the scientific method to validate
our findings." I think being wrong is a risk inherent to
innovative science.  But the community has to recognize errors,
it's no disservice to point them out.  As noted by Robert Hurst
in this discussion, the major conclusions of our study are really
logical consequences of the findings of Ein-Dor et al. 2005,
PMID:15308542, Wirapati et al. 2008, PMID:18662380, and others. A
number of us have been aware of the issue. But, judging from the
biological conclusions that continue to be drawn from prognostic
breast cancer markers on a monthly basis in top journals, many
researchers have not measured the implications of these earlier
papers.  Someone had to pin them down more explicitly. I
commented about our difficulty to publish this study because I
realized others had similar experiences.  Read, for example, this
story about the statisticians who scrutinized the Potti/Nevins
Duke data,
http://www.sciencemag.org/cont...

Vincent Detours

Avatar of: Someone

Someone

Posts: 1457

December 13, 2011

It is perhaps useful to insist we are not questioning the
prognostic value of published signatures. On the contrary, Fig. 6
or our article confirms that most of them are associated with
outcome and that these associations are reproducible across
cohorts. Our paper investigate *why* signatures work.  This
matters little to patients, but it as consequences for basic
cancer research.  It's common place to suggest that because a
marker for a given biological process is prognostic in cancer,
then this process must be involved in breast cancer progression.
Our study reduces this argument ad absurdum by showing that
random signatures are prognostic too (Fig. 1 in the paper).

Contrary to what lifebiomedguru says in a previous thread, the
term 'confounded signatures' is appropriate. There are thousands
genes correlated with outcome, as shown by Ein-Dor et al.,
PMID:15308542.  We showed in our study that the overwhelming
majority of these genes are correlated with a proliferation
metagene. In addition, the vast majority of the published
signatures loose their prognostic value if you adjust the data
for proliferation.  Thus, 1- genes picked up at random are likely
to be prognostic. 2- if, for example, an hypoxia signature is
prognostic, but loose its prognostic value after adjustment for
proliferation, can you claim that its prognostic value supports
the role of hypoxia in cancer progression? No, because you just
can't determine from such evidence wether hypoxia or any of the
many processes statistically associated with proliferation in a
complex tissue cause the correlation with disease progression.

Note that our study addresses breast cancer in which
proliferation is prognostic, it does not necessarily apply to all
cancers.

lifebiomedguru raises the issue of multiple testing.  Our study
is not about multiple testing situations. We are estimating the
probability that a paper presenting a **single** signature finds
a significant association with outcome. Consider for example this
typical situation: Dr. Smith derives a stem cell signature and
finds it to be associated with outcome. The question we asked is:
what is the probability that Dr. Smith would find the same result
if he completely mess up his stem cell experiment and his
signature is in fact a random set of genes? I.e. we are
interested in the proportion of random genes sets that are
significantly associated with outcome.  It is estimated in
Figs. 1 and 2 of our paper: if his signature is >100 genes, Dr
Smith as a 90% chance to find an significant association even if
his signature is made of random genes. Therefore, Dr Smith cannot
draw any biological conclusion from the fact that his stem cell
signature is associated with outcome.  Lifebiomedguru's comment
is relevant to studies evaluating several markers. The q-value
calculation for single gene markers in our paper addresses it:
26% of the genes are associated with outcome at p<0.05 and 17% at
multiple testing-adjusted q<0.05. The percentages will only be
higher for multi-genes markers (relevant to studies investigating
a collection of multi-genes signatures).

We agree with the thread of George Wight, except that having the
genes correlated with one another is not enough. Supplementary
informations of the paper present PCA for individual
signatures. Several have PC1 explaining most of the variance, but
it's highly correlated with proliferation.  Thus one needs also
to rule out the confounding effect of proliferation.  This confounder
has been recognized by several authors, but the method they used
to rule it out does not work (Fig. 5 of our paper)

Finally, lifebiomedguru wrote "There may be a few good reasons
why the author's article was rejected so many times. First, this
article does a disservice by portraying those of us who have been
involved in biomarker development studies as semi-morons who are
not capable or willing to use the scientific method to validate
our findings." I think being wrong is a risk inherent to
innovative science.  But the community has to recognize errors,
it's no disservice to point them out.  As noted by Robert Hurst
in this discussion, the major conclusions of our study are really
logical consequences of the findings of Ein-Dor et al. 2005,
PMID:15308542, Wirapati et al. 2008, PMID:18662380, and others. A
number of us have been aware of the issue. But, judging from the
biological conclusions that continue to be drawn from prognostic
breast cancer markers on a monthly basis in top journals, many
researchers have not measured the implications of these earlier
papers.  Someone had to pin them down more explicitly. I
commented about our difficulty to publish this study because I
realized others had similar experiences.  Read, for example, this
story about the statisticians who scrutinized the Potti/Nevins
Duke data,
http://www.sciencemag.org/cont...

Vincent Detours

Avatar of:

Posts: 0

December 13, 2011

It is perhaps useful to insist we are not questioning the
prognostic value of published signatures. On the contrary, Fig. 6
or our article confirms that most of them are associated with
outcome and that these associations are reproducible across
cohorts. Our paper investigate *why* signatures work.  This
matters little to patients, but it as consequences for basic
cancer research.  It's common place to suggest that because a
marker for a given biological process is prognostic in cancer,
then this process must be involved in breast cancer progression.
Our study reduces this argument ad absurdum by showing that
random signatures are prognostic too (Fig. 1 in the paper).

Contrary to what lifebiomedguru says in a previous thread, the
term 'confounded signatures' is appropriate. There are thousands
genes correlated with outcome, as shown by Ein-Dor et al.,
PMID:15308542.  We showed in our study that the overwhelming
majority of these genes are correlated with a proliferation
metagene. In addition, the vast majority of the published
signatures loose their prognostic value if you adjust the data
for proliferation.  Thus, 1- genes picked up at random are likely
to be prognostic. 2- if, for example, an hypoxia signature is
prognostic, but loose its prognostic value after adjustment for
proliferation, can you claim that its prognostic value supports
the role of hypoxia in cancer progression? No, because you just
can't determine from such evidence wether hypoxia or any of the
many processes statistically associated with proliferation in a
complex tissue cause the correlation with disease progression.

Note that our study addresses breast cancer in which
proliferation is prognostic, it does not necessarily apply to all
cancers.

lifebiomedguru raises the issue of multiple testing.  Our study
is not about multiple testing situations. We are estimating the
probability that a paper presenting a **single** signature finds
a significant association with outcome. Consider for example this
typical situation: Dr. Smith derives a stem cell signature and
finds it to be associated with outcome. The question we asked is:
what is the probability that Dr. Smith would find the same result
if he completely mess up his stem cell experiment and his
signature is in fact a random set of genes? I.e. we are
interested in the proportion of random genes sets that are
significantly associated with outcome.  It is estimated in
Figs. 1 and 2 of our paper: if his signature is >100 genes, Dr
Smith as a 90% chance to find an significant association even if
his signature is made of random genes. Therefore, Dr Smith cannot
draw any biological conclusion from the fact that his stem cell
signature is associated with outcome.  Lifebiomedguru's comment
is relevant to studies evaluating several markers. The q-value
calculation for single gene markers in our paper addresses it:
26% of the genes are associated with outcome at p<0.05 and 17% at
multiple testing-adjusted q<0.05. The percentages will only be
higher for multi-genes markers (relevant to studies investigating
a collection of multi-genes signatures).

We agree with the thread of George Wight, except that having the
genes correlated with one another is not enough. Supplementary
informations of the paper present PCA for individual
signatures. Several have PC1 explaining most of the variance, but
it's highly correlated with proliferation.  Thus one needs also
to rule out the confounding effect of proliferation.  This confounder
has been recognized by several authors, but the method they used
to rule it out does not work (Fig. 5 of our paper)

Finally, lifebiomedguru wrote "There may be a few good reasons
why the author's article was rejected so many times. First, this
article does a disservice by portraying those of us who have been
involved in biomarker development studies as semi-morons who are
not capable or willing to use the scientific method to validate
our findings." I think being wrong is a risk inherent to
innovative science.  But the community has to recognize errors,
it's no disservice to point them out.  As noted by Robert Hurst
in this discussion, the major conclusions of our study are really
logical consequences of the findings of Ein-Dor et al. 2005,
PMID:15308542, Wirapati et al. 2008, PMID:18662380, and others. A
number of us have been aware of the issue. But, judging from the
biological conclusions that continue to be drawn from prognostic
breast cancer markers on a monthly basis in top journals, many
researchers have not measured the implications of these earlier
papers.  Someone had to pin them down more explicitly. I
commented about our difficulty to publish this study because I
realized others had similar experiences.  Read, for example, this
story about the statisticians who scrutinized the Potti/Nevins
Duke data,
http://www.sciencemag.org/cont...

Vincent Detours

Follow The Scientist

icon-facebook icon-linkedin icon-twitter icon-vimeo icon-youtube
Advertisement

Stay Connected with The Scientist

  • icon-facebook The Scientist Magazine
  • icon-facebook The Scientist Careers
  • icon-facebook Neuroscience Research Techniques
  • icon-facebook Genetic Research Techniques
  • icon-facebook Cell Culture Techniques
  • icon-facebook Microbiology and Immunology
  • icon-facebook Cancer Research and Technology
  • icon-facebook Stem Cell and Regenerative Science
Advertisement
LI-COR Biosciences
LI-COR Biosciences
Advertisement
PITTCON
PITTCON
Life Technologies