Opponents of standardized testing have two favorite studies: “Defining Promise” and a University of Chicago study. A thorough look at each reveals that they do not overturn the facts found by the many more studies that show that test scores help colleges select students who will have better college outcomes (typically lower rates of academic probation, higher GPAs, and higher graduation rates). Instead, despite their efforts to the contrary, these studies actually support the role of standardized testing in college admissions. The researchers’ conclusions reveal that they are not doing research to discover what is true. They are propagandists trying to push their narrative and agenda to the detriment of students, especially the most disadvantaged students.
“Defining Promise” Study
Here is how the Lumina Foundation describes its own study: “New research from the National Association for College Admissions Counseling contends that high school grades—not standardized tests—are a much better predictor of future college performance.”
So the quick summary seeks to diminish the importance of standardized testing.
If the primary purpose behind decreasing the role of standardized testing in admissions is for the benefit of disadvantaged students, then the data from the Defining Promise gives us a definitive answer on whether or not there should be test-optional admissions: Absolutely not. On page 45 of the report, the researchers’ own data shows that students at minority-serving institutions graduate at a 54% higher rate if they submit test scores. 54% higher graduation rate! If we could increase graduation rates by that much with something that is very inexpensive or free (because students on fee waivers can take the SAT for free two times and the ACT for free four times and many school districts and states administer the tests for free), then wouldn’t we absolutely do so? Especially if it helped the college outcomes of the most disadvantaged students the most?
But the researchers bury these findings by not highlighting them in their summary: on page 3 of their “major findings,” they make no mention at all of this fantastic discovery that can dramatically improve the outcomes of students at minority-serving institutions.
But it gets worse.
The authors do note in their summary that “in a surprise finding, non-submitters display a distinct two-tail or bimodal curve of family financial capacity” (p.3). What does that mean? Both the poor and the rich apply without test scores in the highest proportions. Either the researchers actually do not understand college admissions, or they are faking that they do not. Of course the wealthy apply without test scores in higher proportions. Cash-strapped colleges (which is mostly all colleges besides those that are selective and highly selective) want full-pay students so badly that they would love to admit the kids of the wealthy without test scores. Who is test-optional really helping? The researchers are silent in their concluding remarks that recruited athletes, students from private schools, and students with wealthy parents use test-optional admissions at higher rates than average applicants and silent that students at the minority-serving colleges in their study had dramatically worse rates of success in college if they did not submit test scores.
In an act of stupefying condescension, the “Defining Promise” researchers do note in their endnotes that “For perhaps understandable reasons, examination of standardized testing policy has not been a top priority at many minority-serving institutions. Given that testing generally does not work in favor of underrepresented minority students in competition for admission, we in recruiting institutions for the study were surprised to find that most well-known minority-serving institutions require testing and don’t seem to have considered a test-optional policy.” Perhaps understandable? Are they serious? They need to look at their own data: 54% higher rate of graduation for students who submitted scores to minority-serving institutions. Absolutely understandable then. Maybe they just don’t think the administrators of the minority-serving institutions are competent or reasonable. They assert that “testing generally does not work in favor of underrepresented minority students”? Maybe not to get accepted. But who cares about getting in if you are then more likely not to graduate and to do so with burdensome student debt that is not dischargeable even in bankruptcy? Why do the researchers not seem to actually care about the outcomes of underrepresented minority students?
University of Chicago Study
The second most cited anti-test study, one by the University of Chicago, violated the most first and most important rule in experiment design: to draw conclusions about a population, you have to have a representative sample from that population.
An exaggerated example can help explain why this first rule is crucial: To estimate what percentage of Americans like basketball, would it be a good idea to go to a professional basketball game and ask people there if they like basketball? Obviously not. Because that’s not a representative sample of all Americans, so you are not going to get accurate results.
In the case of this study, the University of Chicago researchers only analyzed data from non-charter public school high school students in Chicago and the colleges they attended. And then they assert that their research should apply to all public and private, charter and non-charter school students not only in different cities but in different states who attend different colleges in very different proportions than those in their very unrepresentative sample — not to mention homeschool students and students around the world from innumerable different high schools.
Despite violating the first rule of experimental design, they still found that test scores generally increased prediction of a student’s success in college. And, yet, still they recommend against test scores because high school grades are better predictors of college success. But literally no one is saying: “Don’t look at a student’s high school GPA; only look at test scores.” That would be ridiculous. It is equally ridiculous to discard the use of test scores because colleges don’t have all the data on all the students from the 20,000+ high schools in the U.S. to understand what a GPA from each school means, especially when GPAs are scored on many dozens of different scales – and some schools literally only give out “narratives, graphics, and emojis” as grades.
That is precisely why there are standardized tests to compare students from different high schools with different grading systems and scales and why test scores, when used along with high school grades, improve predicting college success. Because testing opponents have an agenda and are potentially too proud to even ask themselves if they might be wrong, they instead interpret everything through the assumption that they are right and simply ignore any and all evidence to the contrary.
Does It Matter?
In 2020, much of the country began an experiment with online schooling. It failed. Students in the U.S. had the largest ever recorded drop in Math scores. Who suffered most? The already disadvantaged students, and racial and income-based gaps in education only grew. We are making the same mistakes in college admissions. And the testing opponents pat themselves on the back, content to feel good rather than to make sure they are doing good.