Supplemental or detrimental?

Journals debate the value of supplemental materials

By | February 24, 2011

Last November, Journal of Neuroscience abolished supplemental materials -- the extra figures and tables that appear online, but not in the printed version of a paper. Editor-in-chief linkurl:John Maunsell;http://f1000.com/thefaculty/member/305132154309970 argued linkurl:in an editorial;http://www.jneurosci.org/cgi/content/full/30/32/10599 that the escalating amount of supplemental materials had begun to devalue the peer review process.
Image: Wikimedia commons, Autopilot
The decision highlights a tension between the need for rigorous peer review of scientific research and the desire to provide as much data as possible to the scientific community. "More data, in and of itself, is always a good thing -- if there aren't adverse effects," said Maunsell, who is also a neuroscientist at Harvard University. But peer review was becoming less effective because many reviewers failed to evaluate the supplemental materials, which the journal wasn't even required to provide, he explained. "We were taking a hit on peer review for something that wasn't formally our responsibility." Usually embraced as a benefit of publishing online, supplemental materials allow researchers to show additional data in the form of figures, tables, video or audio clips. Short format journals like Nature or Science rely heavily on supplemental materials to include data that don't fit into the severe space limitations of their articles. But Journal of Neuroscience doesn't limit the number of figures in a paper, and still supplemental materials have grown over the last ten years to rival the amount of material in the main body of a paper. The ease of getting this material online led to what Maunsell refers to as a "supplemental arms race" in which reviewers started asking for more data, and authors responded by including more data to immunize themselves against these demands. But the reviewers weren't keeping up, Maunsell said. While some said they had evaluated the supplemental materials, others said they hadn't, and still others didn't say. "We can't afford to let peer review get devalued quietly without attention to it," Maunsell said. Seeking to clarify what had been reviewed, and concerned by the amount of time reviewers spent on supplemental materials compared to the main paper, they decided to do away with supplemental materials altogether. Instead, authors can publish one URL that refers to supplemental materials on a Web site maintained by the authors. Some have decried this decision as a step backwards for data transparency, which advocates making more data available -- not less. "The benefits outweigh the risks," said Iain Hrynaszkiewicz, a journal publisher at open-access publisher BioMed Central, who wrote a defense of supplemental materials when Journal of Neuroscience announced its decision. But more and more journals are beginning to disagree. In March 2010, Neuroscience instituted a similar policy to do away with supplemental material, except movies and audio clips. And in October 2009, Cell Press introduced a policy that allowed only one supplemental figure per figure in the main body of a paper, which must directly support a point made in that figure. "It had become a limitless bag of stuff," said linkurl:Emilie Marcus,;http://www.elsevier.com/wps/find/newsroom.newsroom/bio_emiliemarcus editor-in-chief of Cell Press journals. The publisher did not consider abolishing supplementary materials altogether because they have a diverse readership, with different levels of interest in a study's details, Marcus explained, but it was necessary to rein it in. "I do think there are different solutions for different journals," Marcus said. "Scientific communities and journals have probably not given enough thought to what to do with this capacity for supplemental materials. That needs to evolve." Hrynaszkiewicz agreed that the hodgepodge of content currently found in supplemental materials is a far cry from a standardized format for sharing data, but argued that it can act as a stopgap until something better becomes available. "When there aren't any other options in certain fields, then publishing data online with the journal seems to be a good interim measure," he said. But both Marcus and Maunsell say that their supplemental material policies have been well received. And Maunsell hasn't yet detected a change in number or length of the manuscripts submitted. "There was a large contingent of authors and readers who felt that this was stuff they didn't need," he said.
**__Related stories:__***linkurl:Publish or post?;https://www.the-scientist.com/blog/display/57613/
[9th August 2010]*linkurl:I Hate Your Paper;https://www.the-scientist.com/2010/8/1/36/1/
[August 2010]*linkurl:Peer review trickery?;https://www.the-scientist.com/blog/display/57130/
[2nd February 2010]*linkurl:New site pits 'published' vs. 'posted';https://www.the-scientist.com/news/display/53294/
[19th June 2007]

Comments

Avatar of: miguel gama

miguel gama

Posts: 25

February 24, 2011

Information as supplemental data just makes more difficult reading the manuscript. It is worst when the text refers directly to supplemental data which sometimes is not printed in the pdf format directly.
Avatar of: Christopher Lee

Christopher Lee

Posts: 50

February 24, 2011

Knowing how to be precise and concise is part of the art. Research institutions should know how to archive data for a reasonable period, and be able to respond to questions.\n\nClinical or epidemiological studies may be another matter. Possibly, a suitable criterion for allowing additional data to be linked to a paper would depend on the possiblilty that someone else might wish to re-analyse a data set without consulting the authors.
Avatar of: Jarrod Cusens

Jarrod Cusens

Posts: 2

February 24, 2011

I think the question would be better framed as "What type of supplemental material should be held?" \nI am currently undertaking a relatively large meta-analysis in ecology requiring me to build an extensive database. It obviously is completely impractical to publish this in an article. The best option is supplementary database.
Avatar of: anonymous poster

anonymous poster

Posts: 9

February 24, 2011

I agree with the other posts. With large scale microarray studies you would not want the exhaustive list of all 56,000 probes in the actual text of the MS, these however need to be provided for not only peer review but to increase both the knowledge presented and the repeatability of the results.
Avatar of: Ellen Hunt

Ellen Hunt

Posts: 74

February 24, 2011

I can see their point. I have gotten used to the practice of putting certain explanation details into supplements, along with formulas and spreadsheets that make the method easier to use and understand. \n\nI think it's better than it used to be. I remember being in a class where the professor came in and said it took him almost an hour to figure out what just one table meant in an old publication. \n\nPapers are supposed to be a permanent, understandable record. Many older ones are not very understandable. So I come down on the side of more.
Avatar of: Michael Morris

Michael Morris

Posts: 19

February 24, 2011

Authors need not be too concerned about the issue of supplemental material. Each journal will decide its own policy based on a variety of considerations, including the need for editors to look after the interests of reviewers, who are almost always short of time. Authors will look to see if a journal's policy suits them - if it doesn't there will almost always be a plethora of alternative journals of similar ranking whose policies do.\n\nThe alternative to supplemental material is the parenthetic "data not shown". The decisions here come down to appropriate 'tensions' between authors, editors and reviewers. It is possible for authors to submit manuscripts with additional material only required to assist the editors and reviewers in their decision making. Similarly, it is possible for editors and reviewers to request additional information from the authors, which does not necessarily then need to be published even as supplemental material. \n\nIn the quest for transparency, the use of supplemental material is a train beginning to run out of control. For the most part, much of it is not of sufficiently general interest or value. There are, of course, exceptions including for example videos and array data which can be usefully mined by other researchers. Aside from such exceptions, it's either worth presenting the information or it's not - "data not shown" is just fine.\n\n
Avatar of: Ting Wang

Ting Wang

Posts: 15

February 24, 2011

It is a good thing that Journal of Neuroscience abolished supplemental materials. Some other journals should consider if they should abolish supplemental materials or not.
Avatar of: Bjoern Brembs

Bjoern Brembs

Posts: 14

February 24, 2011

This whole debate is absurd, if one disregards the history of how we got here. Hardly anybody reads the printed version of these journals anymore anyway. \n\nTherefore, I say: abolish the printed versions and put everything in the paper that needs to be in the paper (all raw data in the appropriate database).\n\nFor those journals which nevertheless are bent on continuing to print some select portion of their content on paper, they can ask the authors to submit a one page printed supplement which just states what the authors have done and what came out.
Avatar of: Jan Velterop

Jan Velterop

Posts: 1

February 25, 2011

I agree with Bjoern Brembs. Ironic, though, that he still refers to an article as a 'paper'. Subconsciously he may still be in the print era.The subliminal effects of terminology matters. I would also like to get rid of 'journal' (how many are indeed daily?), but the association between 'journal' and 'daily' is ? at least in English ? weaker than the association between paper and, well, paper.
Avatar of: Bjoern Brembs

Bjoern Brembs

Posts: 14

February 25, 2011

I like 'paper' because it's shorter than the alternatives :-)\n\nBut indeed, we should put 'journal' to rest when the entire scientific literature (soon) can fit on a single physical storage device. 'Journal' is a bout as much of an anachronism today as 'stone slab'...
Avatar of: Hans Johansson

Hans Johansson

Posts: 1

March 16, 2011

The opinions expressed herein are my own... \n\nScientists can be said to "own" the results of their experiments, until they are published. Then they are in the public domain, but with the rights held by the publishers. So it comes down what the publishers are going to claim as theirs, and what patent lawyers will define as prior art. If supplemental data is held off the publishers' sites and conveniently disappears, is the entire publication then retracted? \n\n...now they are in the public domain.\n

Popular Now

  1. Thousands of Mutations Accumulate in the Human Brain Over a Lifetime
  2. Can Young Stem Cells Make Older People Stronger?
  3. Two Dozen House Republicans Do an About-Face on Tuition Tax
  4. Insects’ Neural Learning and Memory Center Discovered in Crustaceans