Newly Released NRC Report Rating Ph.D. Programs Attracts Fans, Critics

Fans, Critics Author: Karen Young Kreeger SIDEBAR: Bioscience Make-Over Top Five Institutions in Biological Sciences Doctorate Program In the last couple of weeks, university public relations departments have been issuing press releases and campus newspapers have been publishing front-page articles extolling how well their graduate programs fared in a long-awaited National Research Council (NRC) report rating doctoral research programs. Scientists, higher-education observers, and university officials who have made it at least through the executive summary of the 740-page, five-pound document have lauded it for its breadth, scope, and, in some instances, candor in acknowledging its own limitations. The massive volume was released last month. READ BETWEEN THE LINES: CPST director Catherine Gaddy suggests that students look "beyond the ratings." Critics of the report, however, contend that concerns expressed in the text and by its authors over those limitations are well placed. Chief among their complaints is that some of the statistics and ratings contained in the study have the potential to be misread, misused, and abused. The report is titled Research-Doctorate Programs in the United States: Continuity and Change (M.L. Goldberger, B.A. Maher, P.E. Flattau, eds., Washington, D.C., National Academy Press). The study-whose authors completed collecting data in 1993 -- contains close to 20 indicators that describe the quality and effectiveness of more than 3,600 doctoral programs in 41 fields at 274 universities in the United States. These programs have produced 90 percent of the total number of Ph.D.'s graduated in the disciplines covered by the report between 1986 and 1992. "The report will be useful, and it will cause some controversy," predicts Cornelius Pings, president of the Association of American Universities in Washington, D.C. "I think some faculty and some department chairs feel that they were not ranked as high as they should have been. I haven't heard from anyone saying they were ranked too high." Pings refers to the most publicized and potentially most controversial portion of the report -- a survey of almost 8,000 faculty members who assessed the effectiveness of other schools' programs in training students and the scholarly quality of those programs' faculty. This is the so-called reputational ranking that has lately received so much attention in the academic and public press. Faculty participating in the survey answered a questionnaire designed to elicit ratings of the scholarly quality of faculty at about 50 other programs in their field. The reputational rankings were based on a scale of 0 to 5, with 0 meaning "not sufficient for doctoral education" and 5 signifying "distinguished." Each program received about 100 reviews from survey respondents. It was not too surprising that programs at such leading institutions as the Massachusetts Institute of Technology and the University of California, Berkeley, ranked in the top 10 in many fields. But university administrators note that the report is also a big boost for the entire financially stressed University of California system, many of whose component schools fared well, especially the University of California, San Diego. The school placed in the top 10 in six out of eight of the biological sciences categories. While institutions are expending a great deal of energy touting these reputational ratings, report committee members and higher-education researchers stress that there is much more to the report. A variety of sources -- including the surveyed universities and databases from the Philadelphia -- based Institution for Scientific Information -- provided such information as faculty research publications and citations, honors and prizes, and graduate demographics like the percentage of Ph.D.'s awarded to women in a certain program. "I think there's a lot more statistics in this report in addition to the ratings," remarks Pamela Mellon, a professor of reproductive medicine and neurosciences at UC-San Diego and an NRC committee member. "I hope that people use those and don't just look at the ratings." At UC-San Diego, for instance, the neurosciences program scored an average of 4.82 on the reputational ranking. It also ranked highly in other informational categories: 96 percent of the program's faculty published in the academic literature between 1988 and 1992; the program produced 35 Ph.D.'s between the academic years of 1987-88 and 1991-92; and the ratio of the total number of citations to the number of faculty in the program was 242.4 between 1988 and 1992. By comparison, the mean for neuroscience programs in the top 25 percent was: 88 percent of faculty publishing; 19 Ph.D.'s; and 136.5 citations per faculty member. One other aspect of the 1995 document that life scientists and committee members agree will probably prove to be contentious is the reorganization of the categories in the biological sciences section. The 1995 study updates and expands on NRC's 1982 report, An Assessment of Research-Doctorate Programs in the United States (L. Jones, G. Lindzey, P.E. Coggeshall, eds., National Academy Press, Washington, D.C.). The 1995 report-whose intended audience consists of prospective graduate students, educators, administrators, and policymakers -- is already being compared with other rankings of graduate schools, most notably U.S. News & World Report's 1995 ratings of graduate schools (America's Best Graduate Schools: 1995, U.S. News & World Report). In fact, a summary handed out at the September 12 news conference unveiling the NRC report outlined differences in scope, coverage, and breadth of information between the NRC and U.S. News rankings. NRC pointed out that its report covers 25 more fields; has a broader audience; involves more research faculty; and covers more research-doctorate programs within a given field, than U.S. News' rankings. Marvin L. Goldberger, dean of the natural sciences division at UC-San Diego and cochairman of the NRC report committee with Brendan Maher, Edward C. Henderson Professor of the Psychology of Personality at Harvard University, acknowledges his concern about how the reputational rankings data will be used by various institutions. 'WAKE-UP CALL': NRC committee cochairman Marvin L. Goldberger hopes that administrators in poorly ranked programs will use the report to determine "a course of action." "There's a considerable potential for misuse," says Goldberger. "One is that a number of institutions around the country have been trying to [prepare their own] cross-institutional ranking, using various criteria carefully chosen to improve their own stature. One used is the total number of programs that were rated among the top 10 in the NRC report." He says this could communicate a skewed picture of an institution's worth, discounting specialized schools such as the California Institute of Technology, whose number of programs is limited, but highly regarded. "Using that criterion, for example, excluded Caltech from among the top-10 institutions. Now that's silly," says Goldberger. "I think there's an overreverence of the ranking that goes on, particularly at the level of discrimination between No. 5 and No. 6, which for most of these disciplines is trivial," says AAU's Pings. "One person's voting slightly differently could move someone from, say, 8 to 4. but if you're in the top quartile, it's different than if you're in the bottom quartile." UPS AND DOWNS: "What [the rankings] won't have picked up are the programs that are moving rapidly" in either direction, maintains AAU president Cornelius Pings. Adds Terrence Russell, executive director of the Tallahassee, Fla.-based Association for Institutional Researchers, "Like most of these rankings, the distance of the scores between ranks tends not to be very impressive. And, once you move past the top-10 institutions, the ratings become kind of problematic because they essentially become regional, ranking institutes that are training faculty for community colleges and four-year schools in a particular area of the country." Some critics of the report go even further, charging that the reputational rankings are unfair and subjective. They say that faculty who participated in the survey may unduly give lower scores to smaller programs with which they are unfamiliar. Nonetheless, when the committee selected an indicator by which to rank the programs, they chose the admittedly subjective, reputational measure. "Part of the reason for doing that is because it was done in the 1982 study, and one of our charges was to try to [develop] change measures," explains Goldberger. "Inevitably the reputational measures do carry with them a threat of being unfair. Although we made a very strenuous effort to try to augment those reputational responses with many, many objective indicators, they will tend to be overshadowed by the desirability of people to take whatever fair or unfair advantage they want to from the reputational ratings. "We made a whole-hearted effort to have extremely qualified people do the ratings and chose them in sufficiently large numbers so there was a reasonable probability that any program that was rated was judged by a large number of people." Defenders of the study also point to the fact that the report attempted to head off misreading of the rankings. The report contains a two-page section explaining "What Reputational Measures Do and Don't Tell Us." Because of the similarities in design between the 1982 and 1993 studies, 1,916 doctoral programs in 27 fields appeared in both studies, allowing change in ratings to be calculated. According to the report, a "remarkable degree of stability" was found: 85 percent of programs that were in the top quarter in 1982 were also there in 1993. "This is a snapshot of how things were perceived in 1993," cautions Pings. "Snapshots don't measure motion very well. What this won't have picked up are those programs that are moving rapidly, those that may in 1995 be a lot better than they were in 1993; or, by 1995, might not be nearly as good as they were when the survey was conducted." The time lag between when the survey was conducted and the present may also explain the finding that on average the number of program faculty has increased since the 1982 study in every field common to both assessments, despite recent funding cutbacks and hiring freezes. "At the time of the survey, we were probably only into the [downsizing] of state universities a year or two. Whatever the size of the program, that finding probably reflects the buildup of such programs through the 1980s." NRC officials expect that information in the report will likely be used by prospective graduate students in selecting programs and by administrators and policymakers to make funding decisions. Officials agree that the reputational rankings are the data that will be scrutinized the most by department chairpersons, university trustees, state legislators, development officers, and faculty members, as well as students. "On a broader scale, I think the reputational ratings are significant," suggests Pings. "They could be used by department chairs to tell their deans that if they had more resources, they could move up." Conversely, he says, university trustees might question the wisdom of preserving programs whose ratings did not improve or decreased since the last study. "Those schools who fared poorly are extremely unhappy," acknowledges Goldberger. "Some of them ask: 'Why did you do this? Don't you realize how damaging this is going to be for us?'" He says that "by and large" these are schools that also did poorly in 1982. "They're very concerned, particularly in these days of funding shortages, that in some cases their administration is going to ask: 'What the hell do we need that weak program for?' "I don't know how to respond to this, except that it was not our purpose to adversely influence funding decisions. At the same time there is kind of an obligation to tell it like it is. I hope that people will use the report in some cases as a wake-up call to try and figure out a course of action." "The report is very useful to potential graduate students to know where the strong departments are in the field they're considering," says Pings. However, cautions Catherine Gaddy, executive director of the Washington, D.C.-based Commission on Professionals in Science and Technology, "I recommend that any student looking at it go beyond the ratings and look at the components, especially if they're interested in a particular program. For example, a school might have an extremely impressive record of scholarly publications, but the student body might be very large. The student might like a smaller program." Overall, Peter Syverson, vice president for research at the Washington D.C.-based Council of Graduate Schools, praises the study for "informing the opinion of academics and providing lots of data for researchers," but he says it missed the mark on providing a guide to students. In addition to not having any data on how students fare in the marketplace after graduation, Syverson points out, "it doesn't have anything on process. What happens in the grad program? Are the faculty supportive? Are there star faculty there to work with students? But what I did find refreshing is that the report also lamented this [lack of information in the report]." Report committee members also regret that some data could not be included in this go-around. "One of the challenges of doing such a report as this was the recognition that there were questions we would like to have answered and had neither the resources nor, in many cases, a clear methodology for acquiring [the answers]," says Goldberger. "For example, in order to appropriately evaluate the effectiveness of programs, we need to know what happened to the graduate students from, say, the period 1982 to 1986 among those departments that were judged to be effective in their programs." NRC committee member Mellon concurs: "That was what we really wanted to know, where are these people? What are they doing? Are they using their degrees?" SIDEBAR: Bioscience Make-Over For information on obtaining the report, contact the National Academy Press at 2101 Constitution Ave., N.W., Washington, D.C. 20418; (202) 334-3313. Fax: (202) 334-2451. E-mail: amerchan@nas.edu. Selected tables are available at http://www.nas.edu. A CD-ROM with more detailed information is being developed and will be distributed for public use in the near future. (The Scientist, Vol:9, #20, pg.3, 6-7 , October 16, 1995) (Copyright © The Scientist, Inc.)

Newly Released NRC Report Rating Ph.D. Programs Attracts Fans, Critics

Interested in reading more?

Become a Member of