I find the word "flaw" - used at the end of this article - a bit too strong - "limitation" would be a fairer expression. It is certainly a _limitation_ of this study that there was no objective third party judging the quality of each article - if I would have such an objective quality rating, I could have controlled for it as well. However, there is no such easily obtainable "third-party, objective" measure for quality - which is exactly why people use citations as a proxy for quality (I used and adjusted for data like the authors past citation history as proxy for article quality). The only way to address this issue would have been to give all 1492 papers to 2-3 experts and let them rate the quality, and then adjust for these quality ratings (due to high inter-rater variability it may even be that more raters are needed to come to a "reliable" conclusion on one articles' importance). If somebody gives me half a million dollar to pay each of these experts $100, I am happy to add this analysis. In the absence of funding I did what was feasible, namely adjusting for quality proxies (funding source, authors' past citation history etc). I also asked authors to self-rate their papers to show that there was no difference between the groups. While authors are naturally biased and may overrate the importance of their study, I think nobody is more qualified to judge the potential impact of their paper than the author, and the important part is that there was no difference between the groups in self-rated "quality".