The OECD released their “Education at a Glance – 2005 Report” Daniel Drezner summarizes his take on the US Performance:

1) In science and math, the U.S. is ahead of only the really poor OECD countries — Turkey, Mexico, etc. So yes, there is reason to worry.
2) The poor performance is not because of a downward trend — in fact, if you look at chart A7.1 (“Differences in mean performance of eighth-grade students from 1995 to 2003”), you discover an interesting fact: the United States showed the greatest improvement in science and math scores of the sample — including Korea.
3) The poor performance isn’t because of a dearth of funds — table B1.1 shows that, Switzerland excepted, the United States spends the most amount of money per student in the OECD. You get a similar result if the metric is education spending as a percentage of GDP. Indeed, the OECD comments:

Lower expenditure cannot automatically be equated with a lower quality of educational services. Australia, Belgium, the Czech Republic, Finland, Japan, Korea, the Netherlands and New-Zealand, which have moderate expenditure on education per student at the primary and lower secondary levels, are among the OECD countries with the highest levels of performance by 15-year-old students in mathematics.