A State-by-State Science and Engineering Readiness Index (SERI): Grading States on Their K-12 Preparation of Future Scientists and Engineers

Susan White and Paul Cottle

In 2007, the authors of Rising Above the Gathering Storm [1, subsequently referred to as “RAGS”] were “deeply concerned that the scientific and technological building blocks critical to our economic leadership are eroding at a time when many other nations are gathering strength.” (p. 3) The highest priority actions recommended by this report focused on science, engineering, and math teaching in K-12 programs.

The Bioscience Industry Organization (BIO) released a report in 2009: “Taking the Pulse of Bioscience Education in America:  A State-by-State Analysis” [2].  This report referred to RAGS explicitly; it provides a framework to assess bioscience education in the US at the state level. Not surprisingly, the report focused mostly on K-12 biology courses and student achievement in this subject.  States were sorted into four categories (“leaders of the pack,” “second tier,” “middling performance,” and “lagging performance”) by their ratings on a list of indicators, including the average scale score of 8th graders on the National Assessment of Educational Progress (NAEP), the average scores on the science section of the American College Test (and specifically on the biology items on the test), the pass rate on the Advanced Placement Biology test, and the percentage of biology teachers who had certification in biology.  Florida was included in the “lagging performance” group, prompting an official of the state’s Department of Education to tell a gathering of school district officials that the state’s students are “pretty much last in the nation in science.” [3]

The BIO report focused on biology. We propose a Science and Engineering Readiness Index (SERI) as a tool for policy-makers and educators to use to examine progress in K-12 physical science and engineering education. The foundation provided in K-12 schools to better prepare students for careers in science and engineering is crucial. 

Economics, Physical Sciences, Math, and Engineering

The authors of RAGS were concerned with the erosion of the scientific and technical building blocks for our economic leadership. Not only is this impact felt at the macroeconomic level, but it also rings true at the microeconomic level, too. The American Institute of Physics recently demonstrated [4] that the very best economic opportunities for new bachelor’s degree graduates are in the mathematical and physical sciences and engineering. 

Research shows that students’ success in the demanding undergraduate programs leading to these degrees depends not only on the work of college faculty in these programs, but also on how well these students are prepared during their K-12 years.  The 2007 study by Tyson et al. [5] that followed Florida high school graduates through their postsecondary years demonstrated strong correlations between high school preparation and bachelor’s degree attainment, particularly in STEM (science, technology, engineering and mathematics) fields.  In particular, Tyson et al. demonstrated the importance of high school physics and advanced mathematics courses (including calculus) in the preparation of successful STEM students.  Sadler and Tai [6] demonstrated that taking a high school physics course is correlated with success in physics at the college level (with the corresponding statements about biology and chemistry being correct as well), while taking calculus in high school is correlated with success in all college science courses. 

Given the importance of high school physics and calculus in preparing students for the engineering and physical science fields that offer the best economic prospects for bachelor’s degree graduates, it makes sense to offer policy makers and the public a succinctly-stated measurement of how well their states are doing in preparing K-12 students for these fields.  Here we propose a Science and Engineering Readiness Index (SERI) that incorporates results from the National Assessment of Educational Progress [7,8] (“NAEP”, conducted periodically by the US Department of Education), Advanced Placement Examination results in calculus and physics [9], the physics course-taking results from the American Institute of Physics National Survey of High School Physics Teachers [10] and information on teacher certification requirements in science compiled by the National Council on Teacher Quality (NCTQ) [11].  The information from these sources is gathered into three scores on mathematics performance, science performance and teacher qualifications.  The scores are then used to assign each state a single composite score.  The formulation of this index provides an opportunity for examining the strengths and weaknesses of each state’s K-12 mathematics and science programs.

All of the information used in formulating SERI is available to the public on the Internet.  While other indicators would have been desirable – the percentage of graduating high school seniors who have taken a quantitative physics class, for example – we focused on indicators that are readily available. 

Math score

Two indicators were used to obtain the SERI math score.  The first was the percentage of students who earned achievement levels of proficient or higher on the 2009 8th grade NAEP Math Assessment [6].  NAEP sorts students into four “achievement levels” – advanced, proficient, basic and “below basic”.  A student below the “proficient” level is not on track for a career in the physical sciences or engineering, so we focus on this achievement level.  Nationally, the percentage of students rated “proficient” on the 8th grade NAEP math assessment was 34%.  The state with the highest percentage is Massachusetts (52%) and the state with the lowest percentage is Mississippi (15%).  The 12th grade NAEP math assessment, which might have been more directly applicable to a readiness index, is not reported on a state-by-state basis.

For each indicator used in SERI (like the percentage proficient on the 8th grade NAEP Math Assessment), we formulate a scaled NAEP sub-score between 1 and 5 so that the lowest state (in the case of the 8th grade NAEP Math Assessment, it is Mississippi) receives a scaled NAEP sub-score of 1 and the highest state (here Massachusetts) receives a scaled NAEP sub-score of 5.  We explain the details of the calculation in the appendix. The sub-scores are scaled so that they reflect the actual variation in a given indicator.

The second indicator used in the formulation of the math score reflects the numbers of students passing Advanced Placement Calculus Examinations (Calculus AB and Calculus BC) in 2010.  The AP Calculus sub-score is calculated using the combined number of students passing (score of 3 or better) both exams, relative to the number of high school seniors in the state.  The numbers of students passing and the numbers of high school seniors are taken from the College Board’s “7th Annual AP Report to the Nation” and the subject and state supplements [8].  Nationally, the number of students passing both the Calculus AB and BC exams divided by the number of high school seniors is 6.3%.  The highest state is Massachusetts with 11% which yields an AP Calculus sub-score of 5. The lowest is Mississippi with 0.8% which yields an AP Calculus sub-score of 1.

Finally, the score for math is calculated to be the average of the sub-scores for the 8th grade NAEP Math Assessment and AP Calculus.  Massachusetts has the highest math score, 5.  The lowest score, 1, was earned by Mississippi.   

Science score
Three indicators were used to calculate the science scores.  Two of them, corresponding to performances on the 2009 8th grade NAEP Science Assessment [7] and the AP Physics examinations, were formulated in ways identical to the corresponding indicators used in the math scores.  The third indicator was the course-taking rate for high school physics as determined by the AIP’s National Survey of High School Physics Teachers [9]. 

Nationally, the percentage of 8th graders who earned “proficient” or above on the NAEP Science Assessment was 30%.  The highest and lowest states by this measure were Montana (42%) and Mississippi (15%), respectively.  Numerical state sub-scores for this indicator were calculated using the same procedure used for the NAEP Math Assessment.

The AP Physics indicator (and the corresponding numerical sub-score) were calculated using the same prescription used for AP Calculus.  However, the two exams used in formulating this indicator were the Physics B exam and the Physics C Mechanics exam (Physics C Electricity and Magnetism was not used because the students who passed this exam were almost certainly just a subset of the students who passed the mechanics exam).  Nationally, the ratio of passing scores on the two AP Physics exams to the number of high school seniors was 2%, and the states with the highest and lowest ratios were New York (4.3%) and North Dakota (0.3%), respectively.  The resulting AP Physics sub-scores were topped by the 5 awarded to New York; North Dakota had the lowest sub-score of 1.

In 2009, the AIP National Survey of High School Physics Teachers [9] sorted states into three categories for physics course-taking – states significantly higher than the national rate of 37% (Massachusetts, Michigan, Minnesota, New Hampshire, Texas and Wyoming), states significantly lower than the 37% national rate (Alabama, Alaska, Arkansas, Idaho, Mississippi, Montana, Nebraska, Nevada, North Carolina, North Dakota, Oklahoma and Tennessee), and states not significantly different from 37% (everybody else).  Similarly, in 2005, the national physics-taking rate was 33%. The above-average states were Massachusetts, Maryland, New Jersey, and Pennsylvania, and the below-average states were Alabama, Arkansas, Mississippi, Oklahoma, and Tennessee. Specific details on how we calculated the physics-taking sub-score are described in the appendix.

The numerical score for science is then computed by averaging the numerical sub-scores for the three indicators.  There are four states that did not participate in the NAEP Science Assessment (Alaska, Kansas, Nebraska and Vermont).  For these states, the numerical score for science is calculated by averaging the numerical sub-scores for AP Physics and the physics-taking rate.  The highest score for science was earned by Massachusetts (4.55).  The lowest was awarded to Mississippi (1.27).    

Teacher qualifications score

In 2010, the NCTQ issued a report [11] on science teacher certification in each state and assigned a grade of “red light”, “yellow light” or “green light” depending on whether a state had discipline-specific certifications for each science discipline.  Some states have only general science certifications, while a few have a certification for each discipline (biology, chemistry, Earth science and physics).  States with discipline-specific certifications earned green lights, while states with general science certifications earned red lights.  Yellow lights were awarded to states with certification procedures that were somewhere in the middle of these two limits. 
We assign a teacher qualifications numerical score of 5 to those states that earned a green light from the NCTQ, a score of 3 to yellow light states, and a score of 1 to red light states. 

Composite SERI score

In calculating a composite SERI score, we weight the scores for math and science to each account for 40% of the total.  The teacher qualifications score accounts for the other 20%.  While teacher qualifications are very important, the NCTQ report provides a very coarse evaluation that does not account for the difficulty of earning (for example) a physics certification.  Under these circumstances, the 20% seemed appropriate. 

Only a few good states

Massachusetts easily leads the field with a SERI of 4.82.   

Minnesota, New Jersey, New Hampshire, and New York score between 3.94 and 4.06. These scores are well above the national average of 2.82. We rate Massachusetts as “Best in the US” and the call the next four states “Well above average.”

A third group of states post above average scores between 3.24 and 3.73. These states are Virginia, Maryland, Indiana, Connecticut, and Maine. We call these states “Above average.”

These ten states with SERI scores above the national average accounted for just over 20% of high school graduates in 2009.

“Average” states post average SERIs between 2.53 and 3.13. Nineteen states fall into this group. These states accounted for just over 37% of 2009 high school graduates.

“Below average” states post SERIs that range from 2.14 to 2.47. These twelve states include California. These states accounted for almost one-third of 2009 high school graduates.

“Far below average” states have a SERI between 1.58 and 2.01. These states accounted for about 9% of 2009 high school graduates.

Finally, Mississippi’s SERI is 1.11. This lags the other states by almost half a point and results in Mississippi being labeled “Worst in the US.”

Table 1 shows the sub-scores, component scores, SERI, and rating for each state. Figure 1 depicts the percentage of seniors graduating from schools by SERI rating, and Figure 2 shows the SERI data by state.

The need for science and engineering readiness

In 2010, the sequel to RAGS, Rising Above the Gathering Storm, Revisited: Rapidly Approaching Category 5 (2010) examined the status of the recommended actions. [12] The findings with respect to K-12 math and science education are not good: “[I]n spite of sometimes heroic efforts and occasional very bright spots, our overall public school system – or more accurately 14,000 systems – has shown little sign of improvement, particularly in mathematics and science.” (p. 4)

Science and engineering readiness are building blocks for scientific and technical achievements which contribute to a better society. Not only do the tools, techniques, and tangible goods developed by scientists and engineers provide for more comfortable living conditions, but the economic viability of new discoveries is vital to the sustaining our place in the global marketplace. The call to improve science and engineering education has been sounded repeatedly. Our goal has been to provide a tool to enable policy makers and others to assess progress in that direction.

References

1. Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future, Committee on Prospering in the Global Economy of the 21st Century: An Agenda for American Science and Technology, 2007, http://www.nap.edu/catalog/11463.html
2. “Taking the Pulse of Bioscience Education in America:  A State-by-State Analysis,” Battelle Memorial Institute, Biotechnology Industry Organization and the Biotechnology Institute, 2009 (http://www.bio.org/local/battelle2009/main.asp).
3. “Tougher graduation rules may leave some kids behind,” Orlando Sentinel, April 22, 2010.
4. “What’s a Bachelor’s Degree Worth?”, AIP Physics Trends flyer, Fall 2009 (http://aip.org/statistics/trends/reports/fall09a.pdf).
5. “Science, Technology, Engineering, and Mathematics (STEM) Pathways: High School Science and Math Coursework and Postsecondary Degree Attainment,” Will Tyson, Reginald Lee, Kathryn M. Borman and Mary Ann Hanson, Journal of Education for Students Placed at Risk, Vol. 12, No. 3, pp. 243-270 (2007).
6. “The Two High-School Pillars Supporting College Science,” Phillip M. Sadler and Robert H. Tai, Science, Vol. 317, pp. 457-458 (2007).
7. 2009 National Assessment of Educational Progress in Mathematics, National Center for Education Statistics, US Department of Education (http://nces.ed.gov/nationsreportcard/mathematics/ ).
8. 2009 National Assessment of Educational Progress in Science, National Center for Education Statistics, US Department of Education (http://nces.ed.gov/nationsreportcard/science/).
9. “7th Annual AP Report to the Nation,” College Board, 2011 (http://apreport.collegeboard.org/report-downloads).
10. “High School Physics Courses & Enrollments:  Results from the 2008-09 Nationwide Survey of High School Physics Teachers,” Susan White and Casey Langer Tesfaye, American Institute of Physics, 2010 (http://www.aip.org/statistics/trends/reports/highschool3.pdf). (For state-specific data, contact Susan White at swhite@aip.org)
11. “The All-Purpose Science Teacher:  An Analysis of Loopholes in State Requirements for High School Science Teachers,” National Council on Teacher Quality, 2010 (http://www.nctq.org/p/publications/docs/NCTQ_All_Purpose_Science_Teacher.pdf).
12. Rising Above the Gathering Storm, Revisited: Rapidly Approaching Category 5, Members of the 2005 “Rising Above the Gathering Storm” Committee, 2010, http://www.nap.edu/catalog/12999.html

Table 1: 2011 Science and Engineering Index indicator scores

	Math			Science				Teachers
State	NAEP	AP Calculus	SCORE	NAEP	AP Physics	Physics-taking	TOTAL	SCORE	SERI	Rating*
AK	2.95	3.09	3.02	N/A	1.92	2.03	1.97	1.00	2.20	Below average
AL	1.54	1.92	1.73	1.57	1.63	2.11	1.77	1.00	1.60	Far below average
AR	2.30	1.74	2.02	2.29	1.35	1.87	1.84	3.00	2.14	Below average
AZ	2.51	1.84	2.18	2.00	1.41	2.88	2.10	1.00	1.91	Far below average
CA	1.86	3.79	2.83	1.71	3.30	2.88	2.63	1.00	2.38	Below average
CO	3.70	3.89	3.80	4.00	3.02	2.88	3.30	1.00	3.04	Average
CT	3.70	4.42	4.06	3.86	4.16	2.88	3.63	1.00	3.28	Above average
DE	2.84	3.74	3.29	2.43	2.79	2.88	2.70	1.00	2.60	Average
FL	2.51	2.96	2.74	2.43	2.42	2.88	2.58	5.00	3.13	Average
GA	2.30	3.19	2.74	2.71	3.30	2.88	2.97	3.00	2.88	Average
HI	2.08	3.22	2.65	1.29	3.53	2.88	2.57	1.00	2.29	Below average
IA	3.05	1.70	2.38	3.86	1.49	2.88	2.75	1.00	2.25	Below average
ID	3.49	2.06	2.77	4.14	2.14	2.43	2.90	1.00	2.47	Below average
IL	2.95	3.53	3.24	2.86	3.12	2.88	2.95	3.00	3.08	Average
IN	3.27	2.68	2.97	3.43	1.87	2.88	2.73	5.00	3.28	Above average
KS	3.70	2.01	2.86	N/A	1.39	2.88	2.14	5.00	3.00	Average
KY	2.30	2.32	2.31	3.71	1.51	2.88	2.70	5.00	3.00	Average
LA	1.54	1.61	1.58	1.71	1.07	2.88	1.89	1.00	1.59	Far below average
MA	5.00	5.00	5.00	4.71	4.87	4.06	4.55	5.00	4.82	Best in the US
MD	3.70	4.44	4.07	2.86	3.89	3.32	3.35	3.00	3.57	Above average
ME	3.16	3.80	3.48	3.86	2.62	2.88	3.12	3.00	3.24	Above average
MI	2.73	3.01	2.87	3.86	2.19	3.34	3.13	1.00	2.60	Average
MN	4.46	3.61	4.04	4.57	2.49	3.80	3.62	5.00	4.06	Well above average
MO	3.27	1.92	2.59	4.00	1.77	2.88	2.88	1.00	2.39	Below average
MS	1.00	1.00	1.00	1.00	1.05	1.77	1.27	1.00	1.11	Worst in the US
MT	4.14	1.64	2.89	5.00	1.19	2.66	2.95	1.00	2.53	Average
NC	3.27	3.29	3.28	2.29	1.89	2.03	2.07	1.00	2.34	Below average
ND	4.03	1.68	2.85	4.86	1.00	2.08	2.65	1.00	2.40	Below average
NE	3.16	1.72	2.44	N/A	1.42	2.54	1.98	1.00	1.97	Far below average
NH	4.03	3.88	3.95	4.43	2.56	3.74	3.58	5.00	4.01	Well above average
NJ	4.14	3.84	3.99	3.71	3.56	3.54	3.60	5.00	4.04	Well above average
NM	1.54	1.91	1.73	1.86	1.51	2.88	2.08	1.00	1.72	Far below average
NV	2.08	2.48	2.28	1.71	1.74	2.71	2.05	1.00	1.93	Far below average
NY	3.05	4.21	3.63	3.29	5.00	2.88	3.72	5.00	3.94	Well above average
OH	3.27	2.71	2.99	4.14	2.29	2.88	3.11	1.00	2.64	Average
OK	1.97	1.76	1.87	2.43	1.58	1.00	1.67	3.00	2.01	Far below average
OR	3.38	2.70	3.04	3.86	2.03	2.88	2.92	1.00	2.58	Average
PA	3.70	2.94	3.32	3.86	2.38	3.32	3.19	1.00	2.80	Average
RI	2.41	3.08	2.74	2.57	2.69	2.88	2.72	1.00	2.38	Below average
SC	2.62	2.88	2.75	2.14	1.74	2.88	2.25	1.00	2.20	Below average
SD	3.92	2.39	3.16	4.57	1.69	2.88	3.05	3.00	3.08	Average
TN	2.08	2.19	2.14	2.86	2.01	1.28	2.05	5.00	2.67	Average
TX	3.27	2.49	2.88	3.00	2.27	3.00	2.76	1.00	2.45	Below average
UT	3.16	3.74	3.45	4.43	2.18	2.88	3.17	1.00	2.85	Average
VA	3.27	3.93	3.60	4.00	2.78	2.88	3.22	5.00	3.73	Above average
VT	4.03	3.82	3.92	N/A	2.93	2.88	2.91	1.00	2.93	Average
WA	3.59	3.44	3.52	3.71	2.80	2.88	3.13	1.00	2.86	Average
WI	3.59	4.39	3.99	4.29	2.29	2.88	3.15	1.00	3.06	Average
WV	1.43	1.46	1.44	2.00	1.12	2.88	2.00	1.00	1.58	Far below average
WY	3.16	1.96	2.56	4.00	1.14	5.00	3.38	1.00	2.58	Average
US average	2.95	3.14	3.04	3.00	2.65	2.89	2.86	2.31	2.82

*Best in the US (SERI = 4.82)
Well above average (3.94 < SERI < 4.06)
Above average (3.24 < SERI < 3.73)
Average (2.53 < SERI < 3.13)
Below average (2.14 < SERI < 2.47)
Far below average (1.58 < SERI < 2.01)
Worst in the US (SERI = 1.11)

Figure 1: Percentage of Seniors Graduating from Schools by SERI Rating, 2009

Figure 2: SERI Scores by State

 

Appendix: Calculating the Sub-scores

We calculated the sub-score for each indicator, S_{(indicator, state)}, in two steps.

In the first step, we calculated a ratio for the indicator for each state relative to the highest indicator:

R_{(indicator, state)} = I_{(indicator, state)} / I_{(indicator, max)}

For example,

R_{(NAEP-Math, MS)} = 15 / 52 = 0.288

where 15 is the Math NAEP Indicator for Mississippi and 52 is the maximum Math NAEP Indicator over all states.

The resulting ratios ranged from a minimum of less than 1 to a maximum of 1. We then converted these ratios, R(indicator, state), into sub-scores where were scaled to range from 1 (for the lowest state) to 5 (for the highest state) for each indicator as follows:

S_{(indicator, state}) = 1 + (R_{(indicator, state)} - R_{(indicator, min)}) / R_{(indicator, range)} * 4)

Where
R_{(indicator, min)} is the minimum ratio for a particular indicator over all states
and
R_{(indicator, range}) = R_{(indicator, max)} - R_{(indicator, min)}

For the physics-taking indicator (I(physics-taking, state)), we used the national average physics-taking rate for states in which the rate did not differ significantly from the national average. For states in which the physics-taking rate was significantly higher than the national average, we used the lower bound of the 95% confidence interval for physics-taking in the state. For states in which the physics-taking rate was significantly lower than the national average, we used the upper bound of the 95% confidence interval for physics-taking in the state. We used the average of two ratios (R(physics-taking, state, 2005) and R(physics-taking, state, 2009)) using data from both the 2005 and 2009 surveys. We chose to use the average to account for year-to-year fluctuations in these data.

Susan White (swhite@aip.org) works for the Statistical Research Center at the American Institute of Physics. Paul Cottle (cottle@phy.fsu.edu) is a Professor in the Department of Physics at Florida State University.

---

Disclaimer- The articles and opinion pieces found in this issue of the APS Forum on Education Newsletter are not peer refereed and represent solely the views of the authors and not necessarily the views of the APS.