AAAS Project 2061: Developing Standards-Based Science Assessment Resources

Mary Koppal and Jo Ellen Roseman

Calling for the development of assessments that are "valid, support and inform instruction, provide accurate information about what students know and can do, and measure student achievement against standards," the U.S. Department of Education's Race to the Top program clearly recognizes the important role that high-quality assessment plays in education reform [1]. Project 2061, the long-term science literacy initiative of the American Association for the Advancement of Science (AAAS), shares this view and has been engaged in assessment-related research and development for more than a decade. This effort takes advantage of Project 2061's foundational work in standards-based reform in science education that aims to help every student graduate from high school with the knowledge and skills needed to make well-informed personal and civic decisions and to pursue science learning over a lifetime. Project 2061's seminal publications include Science for All Americans [2], a description of what it is that everyone should know in science, mathematics, and technology; Benchmarks for Science Literacy [3], a set of K-12 learning goals for all students; and Atlas of Science Literacy [4], a collection of knowledge maps that display connections among science ideas - conceptual, cognitive, and thematic - that contribute to a coherent understanding of the natural and designed worlds.

With funding from the National Science Foundation, the Project 2061 assessment research team is nearing completion of a bank of items for middle and early high school science [5]. Covering 16 topics that are essential to literacy in life, earth, and physical science - including force and motion and energy transformations—the items expect students to make use of a variety of cognitive skills such as recognizing the truth of scientific facts and principles and using targeted ideas to explain, predict, and analyze phenomena. The item bank will be supported by a set of related assessment resources, including detailed clarification of the knowledge students are expected to have for each targeted idea, descriptions of the relevant misconceptions that have been identified and documented, and student response data gathered during national field tests of the items.

Content-aligned assessments

Project 2061's approach to science assessment emphasizes the precise alignment of items to the science ideas being tested and the identification of students' misconceptions about those ideas. The item development process involves both qualitative alignments and the use of quantitative psychometric methods to ensure the overall effectiveness of the items as accurate measures of what students do and do not know and as indicators of the specific difficulties students may have in forming scientifically accurate understanding. After first "unpacking" and defining the boundaries of the ideas to be tested, the research team then reviews the research literature to identify potential misconceptions or alternative ideas that students may have. If the literature is inadequate, the researchers conduct their own interviews with students to supplement the existing data.

During the pilot testing phase of the process, students provide feedback on the items and their reasons for selecting or rejecting each of the answer choices. This feedback helps Project 2061 researchers incorporate the identified misconceptions as distractors (wrong answer choices) and resolve any other problems with the vocabulary, task context, or graphics used in the item that may be confusing to students. Each item is then reviewed by scientists and science education experts using a set of review criteria to ensure content alignment and construct validity. After revisions are made based on the reviews and student feedback, the items are field tested on a large national sample to determine the psychometric properties of the items and clusters of items. Field tests have involved more than 1000 schools and 5900 classrooms, and each item is taken by approximately 1500 students.

Undergraduate implications

Although Project 2061's assessment items are designed for use in middle and early high school, the work also has implications for science teaching and learning at the college level. The research team has begun to use student response data gathered during national field tests to investigate the progression of students' understanding between middle and high school and to examine the differences and similarities in the misconceptions that students at each level hold.

To get a sense of the full range of student responses and how they might change over time, Project 2061 researchers have also administered selected items to college students. As an example, students at two universities were tested for their understanding of a set of ideas about chemistry [6]. The undergraduates included students who had taken high school chemistry and were enrolled in a college-level introductory chemistry course but had not yet had any instruction at the college level and students who had had at least one semester of college-level chemistry instruction. By comparing the performance of the college students with students in middle and high school, the Project 2061 researchers found that students had the most difficulty with ideas about atomic motion, changes of state, conservation of mass, and thermal expansion and were most successful with items testing the ideas that all matter is made up of atoms and that atoms are extremely small. Although the data showed a steady increase in understanding of chemistry from sixth grade to college and suggested a hierarchy of misconceptions that appear with less frequency as students become more familiar with the topic over time, the findings also pointed to the surprising persistence of certain misconceptions even at higher grade levels. For a full discussion of these and other findings, go to http://www.project2061.org/publications/2061Connections/2009/media/DRK-12%20Paper.pdf.

Online assessment resources

Scheduled to launch later this year, Project 2061's science assessment web site will provide free access to the items that have been developed, along with information on the specific ideas and misconceptions that are targeted by each item, the correct answer choice, and the difficulty of the item. The site will also include national field test data - reported by gender, grade level, and primary language of the students - to provide a snapshot of what middle and high schoolers know about this set of important science ideas.

To find out more about Project 2061's assessment research and development and for updates on the status of the assessment web site, visit the Project 2061 home page at www.project2061.org and navigate to http://www.project2061.org/cgi-bin/signup.asp to sign up for a free electronic newsletter.

References

[1] Race to the Top Assessment Program Executive Summary (U.S. Department of Education, 2010).
[2] American Association for the Advancement of Science, Science for all Americans, (New York, Oxford University Press, 1989).
[3] American Association for the Advancement of Science, Benchmarks for Science Literacy, (New York, Oxford University Press, 1993).
[4] American Association for the Advancement of Science, Atlas of Science Literacy, Vol. 1 (2001) and Vol. 2 (2007).
[5] G.E. DeBoer, C.F. Herrmann-Abell, A. Gogos, A. Michiels, T. Regan, and P. Wilson, "Assessment linked to science learning goals: Probing student thinking through assessment," in J. Coffey, R. Douglas, and C. Stearns (eds.), Assessing Student Learning: Perspectives from Research and Practice (Arlington VA, NSTA Press, 2008), pp. 231-252.
[6] G.E. DeBoer, C.F. Herrmann-Abell, J. Wertheim, and J.E. Roseman, Assessment linked to middle school science learning goals: A report on field test results for four middle school science topics, paper presented at the annual conference of the National Association for Research in Science Teaching, (Garden Grove CA, April 2009); C.F. Herrmann-Abell, G.E. DeBoer, and J.E. Roseman, Using Rasch modeling to analyze standards-based assessment items aligned to middle school chemistry ideas, poster presented at the meeting of the National Science Foundation DR–K12 principal investigators (Washington DC, Nov. 2009).

Mary Koppal is the communications director of Project 2061. The assessment research team is led by principal investigator George DeBoer who serves as deputy director of Project 2061, and by co-principal investigator Jo Ellen Roseman who directs Project 2061. The research team also includes Cari Herrmann Abell, Jill Wertheim, Jean Flanagan, David Pollock, and Abby Burrows. Contact us at 202-326-6666 or project2061@aaas.org.

Disclaimer - The articles and opinion pieces found in this issue of the APS Forum on Education Newsletter are not peer refereed and represent solely the views of the authors and not necessarily the views of the APS.