What Do We Rank When We Rank Colleges ? Who Determines How and Who Benefits ? : Student Empowerment and the Development of Alternative College Rankings

This article describes the life-cycle thus far about how a class of undergraduates and their professor have been developing a new college ranking approach based on the college's cultivation of civic intelligence. The approach was educational in that we learned by doing what it might take to actually develop a rubric in general (although specifically for ranking colleges). We also found it to be empowering since it was largely a student inspired and directed project. it also fostered critical thinking and social critique based on a bottom-up approach to evaluating orthodox higher education.


Introduction
This story began when my students and I at the Evergreen State College began looking into college rankings.We started this inquiry more or less because of an article on the "Smartest Colleges."As a faculty member (and alumnus) at a non-traditional liberal arts college I am aware that we do things differently and consequently I look somewhat dubiously on conventional ranking systems for colleges.With the class focus on "civic intelligence," approaches that purport to measure abstract social concepts can be relevant to us, especially if they could help us in furthering our understanding of civic intelligence.This short article describes how that somewhat casual initial inquiry led to a more purposeful project with substantial goals far beyond our pay grade.It highlights several of the interesting aspects of our project and its implications for educational activities in the classroom and beyond.

What is the Value of Colleges and Universities?
Every year 20 million people apply to colleges in the United States.To help them identify the schools they'd like to attend many of the hopefuls consult one or more rankings.Although there are many alternatives (such as Greenest Colleges, Best Party Schools, Best Value Colleges, and Colleges That Change Lives) many, if not most, people turn to the US News and World Report's (USNWR) annual college rankings ("The Best Colleges") as their go-to guide.
When we dig deeper into how these ratings are devised we uncover some interesting factoids.Stanford University (and Alice Lloyd College, a small Christian school in rural Kentucky), for example, lead the United States in the percentage of applicants they reject (95%).Harvard's alumni harvest is the most impressive ($650 million in 2015) and MIT graduates tend to earn the highest salaries right after graduation ($110,200 annually on the average).
Although all of that information is actually factored into the ratings, we may ask (as many Radical Teacher readers have undoubtedly also asked) how much it actually tells us about which colleges are "best"?And best for whom?Without actually thinking about it, many people accept information like this as meaningful, legitimate, and authoritative when they rely on college rankings to make important decisions.
While the need to simplify the process of college selection for potential applicants is real, the reliance on approaches like the USNWR's may be problematic.For one thing, these rankings may be failing to advise students thoughtfully.The damage, however, may be more broadly significant: degrading our vision of education, perpetuating social privilege, skewing education towards a market orientation away from public problem-solving, and helping, even, to discourage real learning in schools and classrooms-including, for one thing, downplaying the educational importance of time not spent in classrooms.

Civic Intelligence at Evergreen
Over the years my attention has been increasingly drawn to the question of how groups of all types and sizes, from a handful of people to entire countries or the world, address shared problems.There is no question that some groups do this better than others.For over a decade I've been using civic intelligence as the name of that social capacity or phenomenon.The "amount" of civic intelligence the group has is reflected by the extent that they succeed in addressing problems they face efficiently and equitably and that they have the knowledge, skills, attitudes, social relations, and other resources that are likely to be useful in the face of future challenges (Schuler, 2001).Civic intelligence exists to some degree in all groups and to the degree that it is applied will determine how humankind addresses issues such as climate change, whether wars can be avoided, and how equitable societies are.Thus the quantity and quality of civic intelligence will determine quality of life and possibly even survival itself.
Over the years as my students and I considered civic intelligence the more we realized that studying it was not enough.Civic intelligence must actually be practiced through thought and action if a deep understanding of its potential, challenge, and significance is to be realized.One implication of this is that we ask questions and seek answers rather than just read about cases in a book.Hence we frequently look at our own circumstances, including how education is approached at our own school, the Evergreen State College.Evergreen is a public liberal arts college in Olympia, Washington whose basic philosophy of interdisciplinary studies integrates theory and practice and is strongly rooted in the progressive education tradition of John Dewey, Jane Addams, and others.Evergreen strives to serve non-traditional and other marginalized groups and approximately 50% of Evergreen's students are at or below the federal poverty line.

Evergreen Students Tackle Ranking
As part of our exploration a few years back in our Social Innovation and Civic Intelligence program, which looked at various ways that social reforms were enacted, my students and I became acquainted with a recent news story on the "Smartest Colleges" in the Unted States.The study was conducted by Lumosity, a "brain training" company.The "smartest" college designation, which was determined by how well students performed on a variety of online puzzles, was awarded to MIT, with Harvard and Stanford the first and second runners up.I prepared a short presentation to help us understand and critique the various approaches to college rankings.As we dug deeper into this we were somewhat surprised to see the often flimsy foundations (generally invisible and unquestioned) for products that had such profound implications.It was also illuminating to learn about the sporadic shenanigans of colleges in their struggle for higher scores (waiting to accept students with lower GPAs until after the rankings had been published, for example).
Looking deeper at the individual indicators that are used to support the ranking revealed that many of them help enforce social norms such as individualism, privilege, and elitism.We couldn't help but notice that many of the indicators were biased against Evergreen's philosophy but also against the circumstances of many of our students.One of the dubious elements is the use of the alumni giving rate.In some of the rankings this directly equates the quality of the school with the wealth of its students and their families.For that reason colleges who want high rankings should think twice about allowing students from low income families to sneak past their ivy covered gates.
Similarly, looking at the rejection rate as an indicator of high quality education should encourage a rational school to do everything in its power to garner many more applications than it could ever accept.Another indicator used by USNWR is based on how quickly the average student graduates.What's wrong with this?In the first place, that information doesn't really seem relevant: It not clear how much (if at all) a student might suffer if they attended a college where all students don't complete their degree in exactly four years.More significantly, however, it discriminates against economically disadvantaged students.
It is basically telling colleges not to accept students who are in more precarious positions economically, because they are more likely to interrupt their education or take a less than a full load in any given term, due to health or job related issues that more advantaged students are better defended against.

Looking deeper at the individual indicators that are used to support the ranking revealed that many of them help enforce social norms such as individualism, privilege, and elitism.
Prominent approaches such as USNWR's reliably rewarded the elite, well-funded institutions.At the same time, they also seemed too narrow and diverted attention away from more important perspectives.One of the students in the program suggested that we embark on an alternative ranking project.This was definitely in accordance with our focus on civic intelligence.It would help us focus on what a college could do to cultivate citizens who are interested in working for the common good and what they could do to better equip students for this critical role.Society needs citizens who can help come to terms with "wicked problems" (Rittel and Webber 1973) such as inequality, oppression, climate change, and environmental degradation that defy simple analysis and that citizens must actually help address (and not just through voting) if genuine change is to be achieved.We wanted to challenge the mistaken and dangerous idea that finding answers to our complex social problems is either irrelevant to higher education or that the answers will simply "emerge" via elites, the market, or just plain good luck.Based on that we felt that a more valuable orientating question for a college ranking would be something more like the following: "What are colleges doing to help cultivate citizens who are more likely to feel responsibility towards their fellow citizens and have the 'democratic faith' that John Dewey (1980) valued?"

Ranking Process and Indicator Development
The students and I were intrigued with the idea of our own ranking project and decided to jump in.Our analysis led to the desire to design something that represented our views, not the implicit views of the economic elites.We wanted to identify indicators that would encourage educational systems that were more equitable and empowering for students and, at the same time, that would encourage the development of traits in citizens that would help address pressing social needs.Our goal was to envision and articulate measures that colleges could answer somewhat definitively and honestly based on verifiable evidence.
Our ranking project work was inserted into our other classroom activities several times over the following term.For our first homework assignment everybody brought in specific ways that they believed the civic intelligence of a college could be demonstrated.As we worked individually, collectively, and incrementally over the course of several weeks, our vision of a ranking system that highlights civic intelligence coalesced into five categories and some specific ways to think about them.
(1) How does the college conduct its own affairs in civically intelligent ways?
This perspective focuses on the college as an institution, specifically on its administration, transparency, governance, and organizational structure.
(2) What does the college do to promote civic intelligence among students?This perspective includes activities in the classroom as well as other activities that take place outside the classroom such as informal and formal student organizations and activities.
(3) How does the college cultivate civic intelligence in the community?This examines how the college cultivates civic intelligence in the community and to what extent the college influences the wider world.It looks at the prevalence of students at the college who are engaged in internships with educational, service, or non-profit organizations and whether there is a legacy of non-profit groups in the community.(See, for example, the Sustainability in Prisons Project, http://sustainabilityinprisons.org/ that was launched at Evergreen.)(4) How does the college addresses significant societal issues and needs?This refers primarily to how well and to what extent the college performs in terms of broad social expectations.A college, for example, that accepted a large number of students who are statistically more unlikely to graduate runs the risk of receiving low marks in many ranking systems.But if the college educates these students and graduates them in higher number and they secure meaningful employment those schools should receive high marks.(5) What were the enduring lessons in respect to civic intelligence that the college imparted to its graduates?Addressing this question probably means learning about relevant activities, attitudes, awareness, skills, or even social imagination when students enter the college and when they leave, including perceptions as well as actions, probably by gathering feedback from graduates at regular intervals.
While the criteria and the indicators are still preliminary, the fact that the undergraduate students developed the framework above and a variety of indicators helps demonstrate new possibilities for college rankings as well as the non-traditional educational approach that prompted this work.

Further Refinement
Since our initial effort the project has moved forward somewhat.I discussed the project with Evergreen's Director of Institutional Research and Assessment.I also contributed a blog posting about our work for the Social Innovation Generation group (Schuler 2011).I had been hoping (perhaps not so realistically) that potential coconspirators would get excited about the project and together we would incite the rest of the world's colleges to drop everything and re-orient their efforts to the study and cultivation of civic intelligence.
Recently several students from the Civic Intelligence Research and Action Laboratory (CIRAL) that I facilitate at Evergreen became interested in picking up the work where we left off.CIRAL provides an open framework for students to work together on research and action projects that they have developed themselves (Schuler, 2016).The next task was identifying some indicators to focus on.In thinking about that we realized that it would be possible to draw other students into the project while drawing on the "wisdom of the crowd" as well.That would be in keeping with our focus on civic intelligence and the interplay between group and individual cognition and collective intelligence.We developed a survey containing the initial list of indicators and asked the students in the Global Hunt for Civic Intelligence program to rank them in terms of relevance to civic intelligence in higher education.
The idea of putting the survey on our CIRAL FaceBook page came several days after we had developed it.The FaceBook approach was more or less an afterthought but it helped raise consciousness and extend the idea generation to students from previous civic intelligence programs.It also helped surface three quite valid new indicators.Two had to do with financial transactions (how the money is allocated at the college and where the money comes from) while the other one dealt explicitly with preparing students to do civic intelligence work.Although obtaining comments (or new indicators) was not part of the original motivation for using social media, FaceBook's commenting feature promoted commentary on the subject.For example, when we asked people to note their preferences among the indicators in category 5, it prompted this richer response from a former CIRAL student: This is a question I keep bumping into, as I feel that while Evergreen has avenues for self-directed study and perhaps group-directed study, by the time students get to Evergreen their internal-authority has been trained out of them…I am reaching the end of my time at Evergreen and wish I knew at the beginning what I know now.This is one way this college fails to conduct its own affairs in civically intelligent ways that support and promote civic intelligence among students-there is little scaffolding in place to make new students fully aware of the opportunities available to them and guide in the intimidating task of figuring out how to take advantage of these opportunities.This response prompted the creation of another indicator in time for it to be added to the survey that was handed out in class.This indicator, "College provides necessary knowledge, skills, and other preparation to study and practice civic intelligence," although added after some initial responses to the survey had been made, ended up being the most popular within its category.It also has prompted me to rethink some of ways that we organize our CIRAL work.This indicator could also play a very strong role in the civic intelligence of Evergreen if it were considered holistically, possibly by using the fairly extensive list of "enablers" of civic intelligence (Schuler 2014) that my students and I developed over the years, to develop programs, workshops, and learning objectives on campus.

Similarly, the schools themselves are trapped into ranking
systems that often reinforce standards and pedagogy that are not keeping pace with today's challenges.

Learning, Critique, and Power
One perspective on power is that it constrains how groups of people are formed and what the group can do; it determines the challenges and opportunities that the groups will encounter, including, significantly, the rewards or punishments that are received from taking various actions.In the case of education, students are generally trapped into a system not of their making where their paths are largely circumscribed.At the same time, the net worth of their work is collapsed into a single letter or numeric score.Similarly, the schools themselves are trapped into ranking systems that often reinforce standards and pedagogy that are not keeping pace with today's challenges.It is a rare endeavor indeed when students are engaged in the classroom in critical reflection on the conditions under which their education is conducted.
Although we did not originally examine the reasons why this exploration might be valuable educationally, several lessons can now be identified, even as the project carries on.One capability that can be improved by the exercise, which may be the most important, is self-efficacy, both individual and collective, the belief that obstacles can be surmounted (Maddux 2009) not foregrounded as a "learning objective" or something that the student is graded on, although it is necessary in the real, i.e. non-academic, world where problems often must be embraced, rather than assigned, and are often "wicked" (Rittel and Webber 1973), rather than handcrafted by teachers into bite-size chunks.
During the exercise, we also reviewed other relevant assessment rubrics including the National Survey of Student Engagement, Self-Assessment Rubric for the Institutionalization of Service-Learning in Higher Education, and Evergreen's Self-Study on Core Themes & Indicators.It could be argued that we should have looked at those before we developed our provisional set of indicators; after all, Why reinvent the wheel?Regardless of the unsurprising fact that we found no rubrics based specifically on civic intelligence, we were not interested only in rubrics, but also in the process that creates them.In other words, we were interested in thinking as much about how the wheel came to be as the actuality of the wheel itself.
Looking at ranking systems with a critical eye provides an interesting and relevant focus for student inquiry.For one thing, many, if not most, of the students will be familiar with them.This exercise helped hone our critical inquiry skills and undertake social critique.It helped us question implicit givens such as social, economic, and political imperatives, to get a better look at the man behind the curtain.This exercise helped us see the ongoing maintenance of the normative framework that is necessary to sustain the systems.
We had a chance to see how concepts are developed and cultivated over time, to essentially "do" social science.By developing a ranking system (albeit an unfinished one) we got a good idea about the process that anybody would essentially go through.By necessity we developed creative, collaborative skills as we went along.Also, because we picked up the project again after a lag, the message came through that projects can essentially be thought of as open-ended and somewhat never-ending; they don't end when the term ends.When we reached one milestone (small or large) the next step was more easily seen.Thus students were left with increased confidence that they could successfully undertake efforts typically considered to be untouchable.
Not only was the project proposed by students, it was moved along by students at every stage.The categories, for example, emerged (and were refined) after everybody had brought in a handful of ideas that were written on the board.Every student had a chance to move them around, combine them, develop provisional categories, and ultimately agree on the categories discussed above.From the onset to (at least partial) fruition the project was student-led and inclusive, not directed from the top down.Moreover, by embarking on a project at the beginning we got a much better understanding for how projects like this are done in the "real world"-including the decisions behind the choices, and the pitfalls and challenges of the project.
When we saw the need we developed a critical stance and initiated a counter-project.Students were engaged in developing utopian ideas about how education could or should be conducted.Hence, their self-efficacy and social imagination were encouraged.Opening up the idea of rankings in a broader way to question their validity and impact, the work implicitly became a bottom-up critique of institutionalized education.The project helped surface the assumptions, stated (and unstated) purposes, and social implications of major college rankings specifically but also about issues of research in general.It also helped us develop a preliminary framework for the colleges that we'd like to see, a vision that could be introduced into a more public discussion.
Improving Civic Intelligence in Higher Education?
While a college may gain a higher ranking by attempting to replicate the characteristics of elite institutions as much as possible, America's democracy depends on the civic intelligence-which includes creativity, skills, compassion, and many other characteristics-of everybody-not just a select few.We are continuing to refine our initial framework to create a solid rubric that colleges could use to conduct a self-evaluation in relation to civic intelligence.The most important thing might be that students become cognizant of their role in their own education and that of others.We like to think that it is possible to increase civic intelligence and to reduce civic ignorance.One of the most important lessons of civic intelligence is that practicing civic intelligence is one of the best ways to learn it.
The rankings that we developed are intended to be aspirational, to encourage the improvement of civic intelligence of the world's colleges and universities.We believe that if colleges and universities were to explicitly acknowledge-and "own"-in a deeper way their responsibility and their dedication to cultivating civic intelligence, societies in the twenty-first century would likely be far better equipped for twenty-first century realities.The point is to encourage colleges to think in these terms and it may even be possible that other students (and their faculty) can develop new systems which are comparable to ours.
The election of 2016 raises new questions and suggests new challenges to the theory and practice of civic intelligence.Whether a country survives and thrives or whether it self-destructs ultimately depends on the civic intelligence of its citizenry.This depends, to a large degree, on its educational systems.The rankings that we use to evaluate our educational systems need not be tacit enablers for elitism, inequality, and the status quo.With thought and effort, they can reflect broader issues that increase our chances of working together for the common good.For this reason we encourage educators to facilitate exercises like this in their classrooms.Beyond that, however, we encourage students and their professors to promote ranking systems like the one we have been discussing here while pushing reforms within their institutions.