Institutional Research

Download a PDF Version of the Document

Wittenberg’s Performance in the 2008 US News “Best Colleges Survey”

US News collects hundreds of pieces of data each spring (last year’s “main” survey alone, for example, consisted of 614 questions, many of which required multiple pieces of data to answer).  It uses only fifteen as so-called “ranking indicators”, from which it calculates the score it uses to compile its rankings.  I have listed the indicators below.  Most are self-explanatory; I have described the two indicators that might need some explanation, peer assessment and graduation rate performance.

US News publishes data for 11 of the ranking indicators.  It does not publish data for the other four.  Wittenberg showed improvement this year in 7 of the 11 published indicators:

• Peer assessment (US News sends a survey to college presidents and deans, asking them to evaluate the quality of approximately 250 similar institutions by assigning a number from 1 – lowest quality – to 5 – highest quality.  Wittenberg’s average is its peer assessment score.)
• Six-year graduation rate (Note that the published data are NOT the data used to compile the college rankings.  See below for an explanation.)
• Freshman retention rate
• Proportion of classes enrolling 50 or more students
• Average standardized test scores for the freshman cohort
• Percentage of the freshman cohort in the top 10% of their high school class
• Freshman cohort acceptance rate

Wittenberg declined in two:

• Proportion of classes enrolling fewer than 20 students
• Alumni giving rate

No change occurred in two:

• Proportion of faculty members who are full-time
• Graduation rate performance (US News developed a formula to predict a six-year graduation rate.  It compares a college’s actual rate to its predicted rate and the difference is US News’ measure of graduation rate performance.  For the 2008 edition, it looked at the graduation rate for the class that completed six years in May 2006, i.e., the cohort that entered in August, 2000.

Wittenberg improved in two of the four indicators for which US News does not publish data:

• Faculty compensation
• Spending / student

No change occurred in two:

• Proportion of faculty members with terminal degrees
• Student-faculty ratio

These data are summarized below:

As noted above, US News groups the ranking indicators into seven categories.  Our rank in each category is as follows:

Peer assessment                             97
Ranking indicator used:  peer assessment survey

Graduation and retention            121
Ranking indicators used:  six-year graduation rate, freshman retention rate

Faculty resources                             121
Ranking indicators used:  classes <20 students, classes with >49 students, faculty compensation, student-faculty ratio, proportion of faculty who are full-time.

Student selectivity                          99
Ranking indicators used:  freshman standardized test scores, freshmen in the top 10% of their high school class

Financial resources                          101
Ranking indicator used: spending/student

Alumni giving                                     167
Ranking indicator used:  alumni giving rate

Graduation rate performance    167
Ranking indicator:  graduation rate performance

Wittenberg’s Performance Compared to the Other Liberal Arts Colleges

Box plots for 9 of the 11 indicators that were published appear below (box plots were not produced for either graduation rate performance, which is a statistic of doubtful value, or standardized test scores, which were not provided in a form that could be plotted).   For each indicator, a box plot was created for the top half of the Top Tier (this used to be known as Tier 1), the bottom half of the Top Tier (which used to be known as Tier 2), and for Tier 3.  Wittenberg’s values from the 2007 edition and the 2008 edition were also plotted, to make it possible to compare those values to the distribution of values in each tier and to show how our values have changed from a year ago. 

Peer Assessment:  Wittenberg’s published value improved slightly from 2007, but is still below the Tier 2 median.

Freshman Retention:  Wittenberg’s published value improved slightly from 2007, but is still below the Tier 2 median and 25th percentile.

Graduation Rate:  Wittenberg’s published value improved slightly from 2007, but is still below the Tier 2 median.  It is located exactly at the 25th percentile.

Percentage of the Freshman Cohort in the Top 10% of Their High School Class:  Wittenberg’s published value improved from 2007, but is still below the Tier 2 median.  It is, however, above the 25th percentile.

Acceptance Rate:  Wittenberg’s published value improved slightly from 2007, but is still considerably below the Tier 2 median and 25th percentile.

Proportion of Classes Enrolling 50 or More Students:  Wittenberg’s published value improved from 2007 and is now comfortably above the Tier 2 median.

Proportion of Classes Enrolling Fewer than 20 Students:  The proportion of small classes declined enough to place us below the 25th percentile in Tier 2.

Proportion of Faculty Members Who Are Full-Time:  the proportion remained unchanged and is exactly at the Tier 2 median.

Alumni Giving Rate:  The rate declined enough to place us below the 25th percentile in Tier 2.

Cautions about Using the Data

One reason that discussions of US News can be complicated is that there are actually three sets of US News data:  the data we submit each year, the data US News uses to compile its rankings (which are rounded and can come from several years of data submissions), and the data US News publishes.  They are not the same.  Therefore, use the data with care, understand their limitations and their ambiguities, and keep the following in mind:

• As noted above, the six-year graduation rate published in the magazine is NOT the value US News uses to compile its college rankings.  The published graduation rate is for the last class to have completed six years since its initial matriculation.  The ranking indicator, however, uses an average of the graduation rates for the last FOUR classes to have completed six years since their initial matriculation.
• Five of the ranking indicators use averages of data from multiple years.  Consequently, a one-year improvement in our data might not raise our average or our indicator score.  For our average to improve, all that matters is for the newest value to be higher than the value it replaces, whether or not it is higher or lower than the previous year’s value. 
• The data can be several years old.  We complete the US News survey in April, which means that statistics that are available only at the end of an academic year (e.g., graduation rates) use end-of-year data from the previous year.  Rankings that are based on data from several years reach even further back.  For example, the six-year graduation rate that is used to calculate the college rankings is an average of the rates from the freshman cohorts that entered in 1997, 1998, 1999, and 2000. 
• The indicators carry different weights in US News’ ranking calculations.  Even large improvements in some indicators would have little or no impact on our overall ranking. 
• Our final ranking depends on the performance of the other colleges in our category.  We can improve our ranking scores and not gain any ground.  We would actually lose ground if the colleges ranked near us improve their scores more than we.  If they were to show changes similar to ours, then our rank would not change.  If they were generally to do better or worse, then our rank would change accordingly.   This year’s results illustrate the point.  Our overall score last year was 46 and our rank was 117.  This year, our overall score was unchanged at 46, but our rank declined from 117 to 118.  To add to the confusion, despite our drop in rank, we moved up from Tier 3 into the Top Tier of colleges.  Go figure!

Significance of Wittenberg’s Overall Score

As most of you know, US News has been severely criticized for its “Best Colleges” survey and rankings.  The consensus opinion among institutional researchers and others involved in educational assessment is that the overall scores are fairly meaningless, at least as measures of the quality of the education they provide. 

• They are based almost exclusively on measures of inputs (how much money a college has, how well prepared its students are when they enter, etc.). 
• They ignore most of the things that we know contribute to student learning (the amount of writing that students do, how often they participate in class discussions, etc.). 
• They do not try to measure how much a college’s students actually learn or how much they change, or in what ways they change. 

The consensus opinion is that there is neither a theoretical nor empirical justification for US News’ choice of variables or its methods and there is further consensus that its ranking formula does not produce a valid measure of institutional quality, institutional effectiveness, student success, or student learning.

Significance of Wittenberg’s Rank

Needless to say, if the overall scores have little meaning, then the rankings they create have even less.  In particular, small differences in rank or small changes in rank from year to year have no significance whatsoever.  Movement up and down in the rankings from one year to the next is extremely misleading and does not necessarily denote any change in institutional quality or even a change in a college’s score relative to the top scoring institution.  Depending on the number of schools that had the same converted score or a nearby score in the previous year, a school whose score moves only slightly up or down in one year can experience a large change in rank, a small change, or none at all.  Not surprisingly, a school whose score does not change at all can still experience a change in rank. 
For example, when there are a large number of tied scores, a school whose score moves up or down just one or two points can experience a large change in rank.  If one of the schools with a score of 83 in the adjacent graph were to drop to 82 in the next year, its rank would drop by nine places.  Instead of just six schools having higher scores, there would be 15 schools with higher scores.  However, if the single school with a score of 84 were to drop to a score of 83 in the next year, it would still have only 5 schools ranked above it.  Its rank would not change. 
Anomalies such as these are pervasive.  In the 2008 edition, for example: 

• Bennington’s raw score dropped 3 points from 2007 and its rank dropped by 15 (from 91st to 106th).  However, even though its raw score also dropped 3 points, Wesleyan University’s rank dropped only one place (from 10th to 11th). 
• Coe’s raw score dropped by 2 points and its rank by 11, whereas Middlebury and Swarthmore’s rank remained unchanged, even though their raw scores also dropped by 2. 
• Mount Holyoke and Barnard both lost 1 point in their raw scores and their rank dropped by 4, whereas Carleton, Claremont-McKenna, and Vassar, which also lost 1 point in their raw scores, actually rose in the rankings by 1. 
• Of the 29 schools whose raw scores did not change, 14 dropped one or two places in the rankings, 5 stayed at the same rank, and 10 rose in the rankings from 1 to 7 places.  
• As noted above, Wittenberg’s score stayed the same, its rank dropped, yet it rose from Tier 3 to the Top Tier. 

The fact that most schools move up or down in the rankings from year to year (many quite significantly)[see footnote 1], and the fact that these movements are almost unrelated to changes in their overall scores, and the fact that there cannot have been significant, objective changes in institutional quality in such a short time, demonstrate the obvious:  

There is neither a theoretical nor an empirical basis for believing that differences in overall scores of just a few points have any significance or justify assigning different ranks. 

The consensus opinion is that differences in overall scores of less than 10 to 15 points are insignificant and that US News should recognize this and reform its rankings accordingly. 

Wittenberg did move into the Top Tier in the 2008 edition of the rankings.  Since Wittenberg’s overall score did not change and its rank actually dropped, our move up into the Top Tier was the result of the Tier’s growing in size, from 110 schools in 2007 to 125 schools in 2008.  The reasons for this are unknown, although the total number of schools in the “Best Liberal Arts Colleges” category increased by a net of 35 (30 schools were dropped from the category and 65 were added).  It was reasonable for each tier to grow somewhat.  It also possible that the changes in overall scores among the schools in the lower ranks of the Top Tier were too great for US News to tolerate without an adjustment of some sort.  Without a change in the size of the Top Tier, it is likely that too many schools would have either moved to the very bottom or dropped into Tier 3 (27 of the 36 lowest ranked schools in the Top Tier in 2007 dropped in rank this year). 

Why US News Matters

Despite the flaws in the “Best Colleges” survey, it cannot be ignored.  Although the overall scores and the rankings have little or no meaning, the data that go into them have considerable meaning.  US News asks many of the right questions, even if the report it produces with the answers it receives is not a terribly useful document. 

Withal, there are many people who use it.  Prospective students (and, one can surmise, their parents) use the “Best Colleges” survey and their use has increased steadily over the last decade.  Even though nationally only about 16% of freshmen report that national magazine rankings were “very important” in their college decision, the proportion varies by institutional type, selectivity, and governance.  Slightly over 30% of freshmen at highly selective, private, four-year colleges report that rankings were “very important” in their college decision.  Use also varies by race, ethnicity, intended major, and standardized test scores.  Minority students, students intending a major in a technical or profession-oriented field, or who earned a high score on the SAT or ACT are much more likely than other students to report that rankings were “very important” in their college decision [see footnote 2].  What we believe to be the case turns out to be true – desirable students interested in our type of institution pay attention to the US News rankings.  If for no other reason, we probably need to pay attention as well.

Footnotes:

1. In the 2008 edition, of the 125 schools in Tier 1 and Tier 2, only 14 stayed at the rank they held the previous year. 

2. Higher Education Research Institute, “College Rankings & College Choice: How Important are College Rankings in Students’ College Choice Process?” HERI Research Brief (August 2007).