What's the Value of Conferences?
A Keystone Symposia survey says: $20-30 million in research fund savings
The long reach of Bill Gates finally touched the Keystone Symposia, and all of conference planning, really. I was writing a grant proposal for the Bill & Melinda Gates Foundation and came to an interesting stipulation: We had to show measured value for our program. Another routine survey where respondents write in what they liked and didn't like about our meetings clearly wouldn't do. I had to propose a method for quantitative assessment of the impact of our conferences. Our stated goals are to connect the scientific community for the benefit of society as well as catalyze scientific progress by having the highest quality programs - all well and good, but how to measure this?
© ERIK DREYER/GETTY IMAGES
To answer the question, we put together a survey that was conducted over two seasons of Keystone Symposia's meetings: winter/spring 2004 and winter/spring 2005. It was distributed over 10 different meetings covering a wide range of topics.
TIME AND MONEY
What did we learn? Two results fall under the all important rubric of time and money. Specifically, time and money saved back at the lab based on information garnered at the conference. More than half (52%) of independent scientists agreed that since the meeting they had used something they learned at the conference that had saved time and/or money, or accelerated reaching a research goal. The actual amount of time saved by 250 responders, grouped in quintiles of 50, is shown in Table 1, along with the average estimate of money saved (or diverted to more effective use) from 140 responders; 28 in each quintile. The range of responses was wide, and skewed by some high values. (One extreme one was excluded.) Hence the top quintiles were substantially higher than the medians, which were six weeks and $6,000 saved.
Conservative extrapolation to the entire Keystone Symposia season of meetings would suggest that at least $20 to $30 million in research funds is diverted to more effective use each year because of information learned at our conferences. Since most of the responders are academic and government scientists (more than 80%), the more effective use of research funds is likely to be money that has come from major funding agencies (NIH, MRC, etc.). As for time, saving it might be as simple as meeting someone at the conference who can give you a needed reagent that may have taken you two weeks to prepare yourself. Some claim to have saved two years; it would be interesting to know the details here. The value of time saved is hard to assess. At the easy end of the spectrum, one can simply translate the cost of running a project for a week and multiply it by the weeks saved, and assume that the funds were diverted to more effective use or to advancing the project towards its goal ahead of plans. At the more complex end, how does one assess the value of reaching a goal faster than anticipated?
Another finding, perhaps linked to saving time or money, was that 64% of 1,005 responders at nine months agreed that, since the conference, a new idea or concept that they became aware of during the conference had altered the direction of their research. At 18 months 68% agreed.
At the time of the meeting, attendees were asked to respond to this statement: I established contact with someone at this conference that I anticipate will lead to either a collaboration or future sharing of information, data, or techniques. At the time, 70% agreed or strongly agreed that they anticipated a contact made would lead to either collaboration or future sharing of data or techniques; 26% neutral, and four percent disagreed.
In the nine-month survey, the statement was reworded to determine what actually had occurred since the meeting: Since the conference, I have fostered contact with someone I met at this conference that has lead to either collaboration or sharing of information, data, or techniques.
At nine months, 60% of responders agreed that their contact with someone had actually led to collaboration (see chart, below). It was reassuring to find that at eighteen months 63% agreed. The average attendance of the events in this survey was about 330, and our conferences are held at venues where attendees have lots of time to mingle and do unstructured activities together. It would be interesting to compare these data to results from a much larger meeting held at an urban venue, or with meetings making use of other formats.
The results provide a benchmark from which we can attempt to raise standards. We have shared most of this information with other nonprofit meeting organizations, and at least one that I am aware of, the Wellcome Trust at Hinxton, UK, plans to use it internally as a benchmark. What are the implications of regular surveys trying to quantify value of a broad spectrum of conferences, from the relatively small basic science meetings to the conferences attended by thousands of people at BIO, ASCO, and many others? These meetings have diverse goals, and measurement of success might be done in many creative ways. One lesson I learned was to focus measurements on one's objectives. We asked if collaborations were established, if research direction was altered, and if time was saved reaching research goals. If your conference objective is to inform about new technology, measure whether or not your attendees have adopted the technology. Not all measurements require a survey. If your conference objective is to make a profit, measure your bank account.
Attendees should expect value from a conference, and although most are not accustomed to being asked to quantify it in terms of time or money saved, that is one measure of value. Value, however, is derived in many other ways: a young scientist learning from an experienced leader; scientists making contacts that lead to efficient sharing of resources, or acquiring knowledge that ultimately accelerates a new therapy becoming available. Ultimately, we would like to document what many have intuitively felt for years: Face-to-face meeting of a diverse group of conferees, in an atmosphere conducive to open sharing of information, leads to progress on many fronts, including education, novel concepts, more efficient use of resources, and accelerated discoveries to benefit society.
James W. Aiken is CEO of the Keystone Symposia.