The answer: when it comes to science, U.S. 4th graders rank #8 of 36 participating countries; 8th graders rank #11 of 48 countries. Top FOUR nations are from East Asia. (And for all the money Saudi Arabia has, their students rank waaaay near the bottom of the list.)

I invited Bart Leahy, a technical writer from Huntsville, Alabama, and blogger at Bartacus to sift through the complicated charts and figures recently released in the Trends in International Mathematics and Science Study and boil down the basics for us.

His analysis closes with a bonus: suggestions on how we can improve the situation.  Should we do away with the failing ”self-esteem building” curriculum? Perhaps follow Singapore’s lead and bring back corporal punishment?

Read Bart’s full report, below,  and let us know if you have other recommendations on ways we can improve science and math education.

Analyzing the Trends in International Mathematics and Science Study (TIMMS), Part I

By Bart D. Leahy

At Darlene’s invitation, I was given the opportunity to review and discuss the 2007 Trends in International Mathematics and Science Study (TIMMS). This is a large study of math and science scores among 4^th and 8^th grade students, with a lot of data to be crunched, which might explain why the 2007 study is being released in late 2008. Given the sheer amount of data, I will concentrate on just some of the highlights and lowlights about American students’ test scores. Darlene also provided a link to a news story from the Philadelphia perspective.

In the first part of my analysis, I will concentrate on the United States’ ranking compared with other nations of the world, and the federal policy implications. In part two, I will address some of the specific results in the study, and what and how we should teach our children in the future.

First, a little background on the study:

How TIMSS was Conducted

In the United States, TIMSS was administered in spring 2007. The U.S. sample is representative of both public and private school students at 4th and 8th grades nationally. In total, 257 schools and 10,350 students participated at grade four, and 239 schools and 9,723 students participated at grade eight. More information about how the assessment was developed and conducted is included in the technical notes of the U.S. report on TIMSS (http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2009001).

How TIMSS Results are Reported

Like other large-scale assessments, TIMSS was not designed to provide individual student scores, but rather national and group estimates of performance. Achievement results from TIMSS are reported on a scale from 0 to 1,000. In order to compare performance over time, each TIMSS administration is placed on the same scale, which has a mean of 500 and standard deviation of 100. The TIMSS scale average (500) is the mean score of the original TIMSS 1995 countries (including the United States). Countries can compare their scores over time to this standardized TIMSS scale average, as well as compare their scores directly with other countries.

All differences described using TIMSS data are statistically significant at the .05 level. Differences that are not statistically significant are either not discussed or referred to as “not measurably different” or “not statistically significant.”

In addition to numerical scale results, TIMSS includes international benchmarks at four points on the mathematics and science scales—advanced international benchmark (625), high international benchmark (550), intermediate international benchmark (475), and low international benchmark (400).

One thing to bear in mind, which the background document (linked above) makes clear: the 4^th and 8^th grade results are not exactly apples-to-apples comparisons, as different standards were used, as well as different categories of knowledge. For example, the test used to grade 8^th graders the Program for International Student Assessment (PISA), breaks out physical sciences and chemistry, something the 4^th grade test—the TIMMS—does not. Another challenge for the statisticians is that some nations participated in one survey but not the other, while some nations participated in both.

What the Numbers Say

Below are the stats for the participating countries’ math scores and where U.S. students fit in at the 4^th and 8^th grade levels.

Average mathematics scores of fourth- and eighth-grade students, by country: 2007

TIMSS scale average 500

Black Text = Nations participating in only one survey

Red Text = Nations participating in both

Blue Text = United States scores

Now you can slice and dice these numbers any way you like. There’s plenty to work with in just these two data sets. Looking at the tables, one can make the following statements without being too controversial:

• The top five nations’ students are all from Asia; the top four are from East Asian nations.
• U.S. fourth grade students are ranked 11^th out of 36 nations participating.
• The U.S. students’ rank goes up to 9^th out of 48 nations participating among eighth graders, but the average score is lower by 21 points.
• U.S. students trail behind several of our economic competitors in math scores, such as Hong Kong, Japan, and Russia, but also behind developing nations like Kazakhstan.
• It is difficult to determine the rankings of U.S. students compared to our largest economic competitors, the People’s Republic of China and India, as they did not participate in the survey. Of the nations ahead of the U.S., all of them have smaller populations.

Average science scores of fourth and eighth-grade students, by country: 2007

TIMSS scale average 500

Black Text = Nations participating in only one survey

Red Text = Nations participating in both

Blue Text = United States scores

Again, from a high level, here are the facts as writ in the science category:

• The top four nations are again from East Asia.
• U.S. fourth graders come out 8^th out of 36 in science, while U.S. eighth graders 11^th out of 48 participating nations.
• U.S. students’ scores and ranking dropped between fourth and eighth grades.
• The top two nations’ students remained the top two in the four-year gap. All of the participating nations ahead of the U.S. in fourth grade remained ahead of the U.S. in eighth grade.
• The average grades for the sciences are all a few points lower than the average math scores.

Now I will step into the mine field.

What to Do About It All

The numbers are what they are. The controversy starts when we have to decide what to do about them. Let’s assume the U.S. has a vested interest in maintaining the largest, most advanced, and most prosperous economy in the world. Based on that assumption, one would also presume that we can and should do something.

As a purely empirical matter, we need to look at what the nations at the top are doing right, and determine what actions we can duplicate or translate to an American educational environment. More painfully, we must face realistically what things we’re doing wrong and either fix or stop doing them.

Singapore, for example, has an educational tradition rooted in Chinese Imperial examination practices, which emphasized long study and high expectations. That seems simple enough, and ideally all parents would have the same expectations for their children. However, are American parents prepared to accept schools that expect long hours of studying or administer corporal punishment (e.g. paddling, spanking)? Recall the outrage over the caning of an 18-year-old American for vandalism. Or, are educators prepared to focus on “readin’, writin’, and ‘rithmetic” and dispense with a “self-esteem building” curriculum? Would the best teachers be prepared to work at the worst schools with an expectation of improving things and being rewarded according accordingly? Is the federal government prepared to give up centralized control of educational practices to allow the states or local communities determine how best to meet the needs of their students?

It depends on how badly we want to improve and what prices we are willing to pay to remain competitive in the world.

But rather than look to Singapore or Japan, we can simply focus on the two best-performing American states, Massachusetts and Minnesota. What lessons might we learn from those states’ practices, and how far can other states go to duplicate their success, if feasible?

First, both are Northern, liberal (“blue”) states, where higher taxes and “big good government” are expected. These practices might go over fine in some areas, but would be quite a shift for anti-tax, limited-government “red states.” But the bottom-line translation might just be “You get what you pay for.”

What else might one investigate? Their curricula? The respective education departments can be found here for MA and here for MN. I leave that exercise to the reader. Their diversity? Unlikely—Massachusetts is 89% white, Minnesota 92%. Class size? Money spent per student? The numbers vary: $11,981 per student in Massachusetts, $9,138 per student in Minnesota (Alabama, where I live, spends $7,646 per student and had 18% of its students performing above average, as opposed to Massachusetts, which had 51% of its students performing above average). Physical infrastructure? More importantly, even if all other external factors were somehow magically made equal, could the results all be made equal nationwide? I’ll try to answer some of that last question in part two.

This seems to me like a great opportunity for Obama to demonstrate the strength of America’s diversity and creativity by decentralizing our educational establishment. After all, if we have one, centralized education system via No Child Left Behind and that system is causing our students to fall behind other nations of the world, then that system needs to be fixed. Or diversified. Or replaced.

Devolution would offer each state the opportunity to perform its own local experiments in educational achievement. Federal funding could continue, but control and “strings” would exist at the state or local levels. The internet would allow local school districts to observe or share teaching ideas and copy them (or not) elsewhere. Local control of curricula would also open the door to vouchers, allowing for more competition and fewer perverse disincentives to achievement, like cutting funding to struggling schools or good teachers not sharing their ideas for fear of losing merit pay to other teachers. If you think I’m kidding about the latter item, here’s what the NEA says about merit pay:

Merit pay systems force teachers to compete, rather than cooperate. They create a disincentive for teachers to share information and teaching techniques. This is especially true because there is always a limited pool of money for merit pay. Thus, the number-one way teachers learn their craft –learning from their colleagues — is effectively shut down. If you think we have turnover problems in teaching now, wait until new teachers have no one to turn to.

(Mind you, there are probably ways to measure “merit” based on training and cooperating with others, and every profession works within a “limited pool of money” for merit pay, but we’ll leave those arguments aside for now.)

Will all of the local educational experiments succeed? Probably not. However, parallel, diversified approaches would have to be better than one system failing everybody at once. Will the U.S. be willing to fix what needs to be fixed? Again, it depends on how badly we want to.

Bart Leahy is a technical writer living in Huntsville, Alabama. He blogs regularly at http://bartacus.blogspot.com.

Keywords: TIMMS, Trends in International Mathematics and Science Study, 2007 Report, U.S. test scores, academic achievement, science, technology, engineering, math, STEM, National Center for Education Statistics, ranking, 4^th grade, 8^th grade, merit pay, local curriculum control, decentralization, No Child Left Behind, NEA, National Education Association, Massachusetts, Minnesota