This slim volume (51 pages of generously-spaced type) is a report by the National Academy’s Committee on Science, Engineering, and Public Policy (COSEPUP). The study was initiated by the Academy, probably in response to a growing concern in the science community about the likely impact of the Government Performance and Results Act (GPRA) on research. GPRA requires all federal agencies to establish goals and then use performance measures to identify progress toward those goals. Results of this comparison are to be used for management and budgeting. Quoting from the report, "the law requires each agency to produce three documents: a strategic plan, which sets general goals and objectives over a minimal 5-year period; a performance plan, which translates the goals of the strategic plan into annual targets; and a performance report, which demonstrates what targets were met."

The first performance reports are due next March. Under OMB’s guidance, federal agencies have been struggling to implement GPRA. Agencies with readily identified performance measures, such as number of Superfund sites cleaned up or number of people trained or moved into or off a program, have relatively straightforward tasks in implementing GPRA. Those agencies with large research programs have been struggling to find metrics that can meet GPRA’s requirements. The Academies offered to do a study to help identify what can be done.

Many researchers who are familiar with GPRA–which, unfortunately, is quite few–take the view that basic research cannot be measured and, therefore, GPRA cannot apply. COSEPUP concluded the opposite. Their report takes two strong positions. First, the useful outcomes of basic research cannot be measured directly on an annual basis, because the usefulness of new basic knowledge is inherently too unpredictable; so the usefulness of basic research must be measured by historical reviews based on a much longer timeframe. Second, that does not mean that there are no meaningful measures of performance of basic research while the research is in progress, because basic research does have annual results that can be evaluated.

The report makes the usual points that the ultimate practical outcomes of basic research cannot be predicted and, consequently, there must be some way to measure the usefulness of this research without evaluating the outcome. Several mentions are made of research that took many decades before practical result was reached, atomic structure work in the late 1930s and late 1940s, the results of which can be seen in, for example, the global positioning system. COSEPUP concludes that applied research reporting is much easier and the suggestion is that an applied research program has milestones giving accomplishments and dates and progress toward meeting those milestones can be used for the GPRA process.

The main purpose of this document, however, is to explain and defend how measures can be used for basic research. Not surprisingly, COSEPUP concludes that expert review is the approach to be used. However, COSEPUP goes beyond the usual treatment, which relies on peer review, and expands it using the concept that "the appropriate measure...is the quality, relevance, and leadership of the research." COSEPUP recommends, consequently, three expert reviews for research programs. The first would be peer review as is well known, using "reviewers who are sufficiently expert in the fields being assessed to perform a quality review. This approach is traditionally called peer review....The talent, objective judgment, and experience of these experts, or peers, are paramount and should be the criteria for their selection."

The second recommended review is a relevance review, in which the question addressed is not the quality of the research program, but rather "is the research program focused on the subjects most relevant to the agency mission?" Here, potential users, along with experts in related fields, are asked to "evaluate the relevance of research to agency goals–is the research on subjects in which new understanding could be important in fulfilling the agency’s mission?"

The final recommended measure is leadership: "Is the research being performed at the forefront of scientific and technological knowledge?" Here, they recommend international benchmarking "by a panel of experts who have sufficient stature and perspective to assess the international standing of research."

COSEPUP recommends that the scientific community become more familiar with and involved in the GPRA process. The COSEPUP members remarked that they "have been struck by the small number of researchers who are aware of the intent of GPRA and its relevance and importance both to their work and to the procedures of federal agencies that support research." The COSEPUP report also stresses the need for agencies to focus on developing the next group of researchers, i.e., the need for education to be part of their goals. COSEPUP notes that research agencies had not included that as a goal.

The report has six conclusions and six recommendations, summarized as follows:

•Both applied research and basic research programs can be evaluated meaningfully on a regular basis.

•Agencies must evaluate their research programs by using measurements that match the character of the research.

•The most effective means of evaluating federally funded research programs is expert review which includes quality review, relevance review, and benchmarking.

•Agencies must pay increased attention to training and educating young scientists and engineers.

•Mechanisms for coordinating research programs in multiple agencies whose fields or subject matters overlap are insufficient.

•The development of effective methods for evaluating and reporting performance requires the participation of the scientific and engineering community.

•Research programs should be described in strategic and performance plans and evaluated in performance reports.

•For applied research programs, agencies should measure progress toward practical outcomes. For basic research programs, agencies should measure quality, relevance, and leadership.

•Federal agencies should use expert review to assess the quality of research they support, the relevance of that research to their mission, and the leadership of the research.

•Both research and mission agencies should describe in their strategic and performance plans the goal of developing and maintaining adequate human resources in fields critical to their missions both at the national level and in their agencies.

•A formal policy should be established to identify and coordinate areas of research that are supported by multiple agencies. A lead agency should be identified for each field of research.

•The science and engineering community can and should play an important role in GPRA implementation.

To provide advice to Congress and the OMB, COSEPUP intends to produce a second report that would "develop mechanisms to evaluate the effects of implementing GPRA on agency program decisions and on the practices of research," which will require identifying lessons learned and best practices.

As are most COSEPUP reports, this is well written (they employ a science writer), succinct, and absent of much detail. The importance of COSEPUP reports comes from the stature of its members. The small membership includes five members of the National Academy of Sciences, four members of the National Academy of Engineering, two members of the Institute of Medicine, two who are members both of the NAS and the NAE, and one who is a member of the IOM and the NAS.

Though only two names appear on the cover of this important, beautifully written (though occasionally repetitious) book, there was a third author who died before its completion. The authors of record, Morrison, an MIT Institute Professor Emeritus, and Tsipis, the retired Director of MIT’s Program in Science and Technology for International Security, are "physicists old enough to recall the daily terrors of World War II, and citizens all too familiar with 50 perilous years of nuclear policy since." (p.6) The book was initiated by a question from the third, active but unlisted author, electrical engineer Jerome Wiesner, former presidential science advisor and president of MIT: "Why don’t you write a book about ...what is possible within the broad objective constraints that delimit our options: the laws of physics, energy, and food availability, population, global wealth, geography, weather...possible global security arrangements that would tend toward peace. Write about what is possible and hopeful, about blue sky."(xi) Both Morrison and Wiesner "actively participated in developing weapons" during W.W.II, Wiesner at the MIT Radiation Laboratory, Morrison in the Manhattan Project as "groupleader, from Chicago to Los Alamos, to Tinian, to Japan"; "Tsipis ... a schoolboy ...in wartime Athens...witnessed close at hand death by bullet and hunger and the arbitrary violence of armed occupiers."(xiii) In the post-W.W.II years, "Wiesner’s was manifestly the way of the serious insider, Morrison’s was of the political outsider, both more academic and more dissident."(xi) All three utilized the general method of the physical sciences: "A quantitative but approximate analysis that emphasizes the rough quantities involved, searching out persistent and large-scale trends and less concerned with complex and shifting political mechanisms that appear to us more contingent than causal."(xiii)

The book plausibly develops, and grounds its conclusions on some fundamental, optimistic, assumptions: "there is a broad, slowly shifting minimum level of well-being and security that leads most people to choose peaceful steps to further change...the human race has now and can maintain the ability to provide that basic level to most people, even in the face of pressures for food, shelter, employment, and security emanating for the first time in history from the great postwar demographic transition that mark’s this century’s end."(156) Drawing from these, the book sketches an efficient, effective, US military posture for the next generation, one which will cost half of the present $300B/year. The savings will cover the costs of the other two suggested components of an overall US security program: a Common Security paradigm consisting of international conventional and nuclear forces - a small body of full time UN "quick-reaction" troops, the rest made up of national forces dedicated to the UN; a Common Development program to facilitate the growth of a peaceful world which would, eventually, make the National and Common Security components less necessary. This Common Development "will have to persist for a generation to show steady results, but its hopeful success along the way will make Common Security requirements manageable and the reduction of national military budgets politically easier."(144) It would further our ethical, economic, and security requirements, costing us perhaps $25B/year: "Rather than have the workers of the Second and Third World come to the industries of the North to improve their standard of living, technology and industry can be transplanted into the less prosperous nations."(149) "The broadly shared humanitarian impulses of the American public coincide not only with our national security but also with our economic interests..."(154)

The conceptual development is compelling - based upon reasonable extrapolations from the world as we know it today - agricultural, cultural, demographic, economic, energetics, industrial, military. I find just two shortcomings. (a)The authors slight the role of CBW (chemical, biological weapons) as a potential leveler between major and minor states or even between states and non-state groups. It may be that "Neither biological nor chemical weapons seem decisive against prepared adversaries."(46) but it is not clear to me what "decisive" implies, nor how easy it will be to become "prepared adversaries". (b) They assume that Common Security will only respond to "crossborder aggression" or to invitations from all concerned belligerents to intercede in an intrastate dispute: "We believe that the United Nations should avoid military entry into such unresolved conflicts."(135) "The community of states will have to decide the nature and scale of intervention into civil war as it has done in the past...We therefore entrust the international intervention into civil conflict to common and not statute law..."(136-7) My problem is with the authors’ hopeful assumption that "transborder aggression" is well defined; this has certainly not been the case in some major recent hostilities. When does a border become a border? Were the attacks of the Yugoslavian Army on Croatia and Slovenia (both "formerly" part of Yugoslavia) civil wars - to which "Common Security" would not apply, or transborder aggressions? Just how many states must recognize an "emerging" statehood before a domestic quarrel becomes an international war?

I also have difficulty with their putting off, until after the successful conclusion of Common Development, the addressing of environmental concerns. "First things first; keep hope alive as we move toward a better and fairer regime of frugality and efficiency. Then we can confront under global consensus the environmental problems whose advent and whose remedy must be found on still grander a scale." (190) There is the problem of "sustainability" (188): where is all the clean air and water, required for the large - even if plateoued - population to come from? And how available is all of the energy required to sustain Common Development? These concerns are put off, till after the Security and Development problems have been settled. It’s not clear to me that Security and Development can be so neatly separated from sustainability!

Still, this is a stimulating, useful, interesting and optimistic read for scholars, and students, citizens and politicians, scientists and laypeople, who wish to contribute to the well being of the future inhabitants of this planet. It deserves a very wide circulation.