DEBATE CENTRAL
Debating Resources for the World since 1994
There is a debate community, and there are several debate communities. The smaller communities make up the larger and exist in a kind of tension with them, when participants are faced with resource choices between engaging the "national circuit" and remaining regionally competitive. I realize that in several communities, this choice is not absolute. In others, it is quite determinative of the quality and character of squads and their goals.
This paper is really a collection of speculations, some meditative and others anecdotally empirical, concerning the current practice of mutual preference judging (MPJ) at NDT and CEDA debate tournaments. Participants in MPJ debate rank critics on a list provided to them before the tournament. At some point in the tournament, such as after the first two rounds, MPJ greatly increases the chances of debate teams receiving critics in front of whom they "prefer" to debate. I invite colleagues to research any of the speculations to follow. I make no claim to their absolute truth.
Mutual preference judging is the practice of allowing teams to rank critics based on teams’ preference for that critic. The tournament software then assigns judges based on each of the two teams highest possible preference. Using his own computer program, employing an ordinal "A, B, and C" scale of critics, Gary Larson achieved ninety percent mutual "A"’s at the 1998 CEDA National Tournament in Rochester, New York. Some discussion of the effects of Mutual Preference Judging has taken place on the EDEBATE list serve. Those in favor of MPJ have made the following arguments:
Those opposed to MPJ have made these arguments:
During the past five years or so, policy debate communities in the United States have consistently supported MPJ. A majority in support of the practice have patiently heard, and gently rejected, arguments against it. Some in the vocal anti-MPJ minority have speculated that its supporters wish to avoid a meaningful and public debate about it. One concern is that those against MPJ will themselves be perceived as "bad judges" and hence receive low ranks from nationally competitive debate teams.
Based on my own experiences and conversations, as well as aggressive solicitation for feedback from participants on the EDEBATE list serve, the following are my speculations.
Speculation #1: Competitors will support MPJ more often than judges
Mutual preference among competitors for their umpires makes logical sense. In this sense, the community, and even individuals in it, are often divided between their own roles as judges on one side, competitors on the other. Some in the community have suggested that both sides are competitors, reasoning that since judges have the power to "rank" debaters, that power should be reciprocal. Michael Hester sardonically writes:
if the harms of mpj justify its abolition, why stop there? why not just get rid of all aspects of competition? this isn't a redherring to deflect attention away from MPJ. i'm making this comparison to support my main argument: the debate community values competition. our every gathering is an endorsement of it. (Mike Hester, "Re: MPJ thoughts," 28Oct 1999 Posting to EDEBATE Listserve)
Competitor empowerment, based on a system which allows both teams to "select" a preferred critic, is laudable, attractive, and intellectually challenging. Those who seek excellence in the game would support MPJ. Those who seek comfort, consistency and predictability, all of which favor competitors, would choose such a system. A survey testing this speculation should be easy to assemble.
Speculation #2: Many coaches and judges are also competitors at heart, and will thus support MPJ
MPJ is designed to be fair and objective within the competitive limits of academic debate. Before we begin the barrage of anti-MPJ arguments, a long post by Gary Larson, debate’s leading tabulation programmer, is extremely telling. Larson reminds us that judging assignment has been "unfair" in worse ways.
Truly "random" judge assignment was rarely used at large tournaments, even during the height of CEDA’s egalitarian period. Instead of mutual preference of the participants we typically had "tabroom" preference. In most cases where "manual" assignment of judging took place - early on with cards, later often with TRM, it was not the case that tabrooms just shuffled the deck and assigned whoever came up. In many cases, tabrooms were explicit (though not often publicly vocal) about their goal of having the "best" judges in the "best" rounds. In other cases the outcome was to assign judges "randomly" but to "fix" the egregious cases. In the worst case, judge assignments could directly favor teams associated with tabroom participants or elites whose interests would be "protected." Even in tournaments with STA and its viciously random judge assignment algorithm, most tournament directors I worked for preserved the right to fix the inevitable "tragedies" that random assignment would produce. It was a significant early struggle to convince CEDA leadership that judge assignment at nationals should be "entirely" random (even assuming strikes). At the first two CEDA Nats run on STA, the tournament director reserved the right in elims to "adjust" "bad" panels created by the overtly random computer.
Larson continues:
It is ironic that given my early crusade to implement "random" rather than "tabroom preference" judging at large CEDA tournaments, that I am now the one who has doomed us all with the possibility of ordinal preference. Throughout the past decade my campaign has not been for random, or MPJ or ordinal but rather for the elimination of tabroom interference in the judge assignment system. It is my contention that none of us are either wise enough or fair enough to decide for others who should judge each or any debate at a tournament. And if we aren’t (and probably never were) really committed to truly random assignment, computer-mediated MPJ of some form is FAR better than the alternative. Again ironically, if a system is to be criticized for entrenching elites, the alternative of "tabroom" preference is probably the worst alternative of all.
Insofar as the alternative to MPJ is tab room preference, there is no question that MPJ is fairer. Moreover, it is difficult to discern the fairness or integrity of "random" judging when there seems to be community consensus that there is a standard of excellence by which the best and most competitive rounds should be adjudicated. Randomness is at least as arbitrary as MPJ in result, if not in design.
I write the preceding argument out carefully because it underscores two important things about the pro-MPJ position. First, defenders of MPJ are not unconcerned with fairness or representation. Second, defenders of MPJ want to hear the community integrity argument within the context of a community of excellence. There must be a means by which judges who do not render decisions reflective of the arguments in the debate can be kept from determining the success of competitors who often spend thirty, forty or more hours per week preparing to debate. Hard-working debaters need hard-working judges.
If MPJ really is fair to everyone, then fairness may trump other concerns, even pedagogical ones. If it is not fair, if it instead (or simultaneously) entrenches inequalities arbitrarily, then we need to re-think its practice.
Speculation #3: MPJ has a balkanizing effect on the debate community
Any system of judging based on debater preference means some judges will be excluded. There are several problems of exclusion unique to debate communities. First, mutual preference exclusion is often arbitrary, because it is often based on inexperience with a critic, rather than experience. This means stereotyping can occur. Gina Lane observes:
When we define our judging pool in such a narrow way, there is no way to ever have experiences with a diverse pool of judges in order to see if preferences change. And, in the process of selecting the narrow A group, many stereotyped generalizations will be used to exclude judges regardless of their actual abilities. In other words, a team's lack of actual experiences with many judges means the categorization of these judges as non-preferred is made on the basis of stereotypes. Under the preferred A pool system, this stereotyping will be persistent and entrenched, because a team will rarely experience the judging of critics they don't prefer and will therefore not have the opportunity to reconsider whether or not reality matched the stereotype. If it is true that we are excluding critics from debate rounds on the basis of these stereotypes, then there are a host of implications -- a lack of role models for nonrepresented groups in the community to emulate; a lack of diverse perspectives and insights; lost opportunities to communicate with each other in order to bridge differences and increase understanding. And (gasp!) we might even discover that these 'non-preferred' people are good critics as well.
Second, beyond the cultural and ethical implications of exclusion, the marginalized are also at a competitive disadvantage. David Steinberg notes:
Small regional programs, and those that do not travel or network much, are left out of effective decision making in completing the MPF forms. They do not know the judging pool. So the system magnifies the advantages of the advantaged, and emphasizes a feeling of disenfranchisement.
Since coaches are also competitors, part of the team, their exclusion from judging "good" teams deprives them of development.
The coaches of the small regional programs and new coaches without much exposure (as debaters or judges) are ranked low, and therefore do not judge the most accomplished debaters, except as mutual C's. Tournaments try to avoid mutual C's. So if I am a new or unknown coach, most in need of exposure to the top levels of debate, I am told by the elite that I am not worthy, and I am not likely to judge top rounds. I am not a member of the club, and not invited to join. This retards my development as a judge and coach, since when I do judge the experienced teams, I learn a lot that I can teach my debaters. Such a perception (not unique to judge assignment, but reinforced by it) drives program directors to Parly, NEDA, and generally out of our flavor of the activity...
A reputable "preferred pool" begins to look like an elite. This reinforces negative self-images and deprives some programs of growth. Dean Gundlach asks:
Who gets in? Former competitors from elite programs. Coaches from elite programs. Former National Champions. Sounds like a good pool to me. However this may also reinforce stagnation of programs and organizations.
So apparent is the competitive disadvantage of MPJ-marginalized schools that some have gone so far as to call MPJ "artificial," as when Terry West speaks of the
artificial constructs such as mutual preference [which] judging constrain judges from some schools from being able to hear rounds with teams (often from being in the same divisions) their entered students may encounter. That's fundamentally unfair. In fact, it's downright shameful, and about #3 on my perpetual list of reasons why mpj works only for the Chosen Ones.
I call these results "balkanization" in two respects. The individual balkanization has already been chronicled. But there is also a balkanization of judges and preferred teams. If I am a good judge, some teams will know that, but not all. Last year at a "national" tournament, I judged West Georgia versus Alabama, Alabama versus Cornell, and Cornell versus West Georgia. In elimination rounds, it was Cornell and Alabama again, and then Cornell once more.
As judges become "Balkanized" into particular clusters of teams, generating new, discursive debate communities, those at the margins of these communities will be the "rejects" of MPJ-based debate. They are the silenced victims of a drive towards homogeneity, interpreted by the elites in the debate community as "stability." Accordingly,
This search for stability inevitably breeds a system wherein certain judges are preferred by a certain level of teams that reinforces stereotypical perceptions on an elite CLUB that no one can penetrate and views that judges are not biased. Can we really say that after judging a certain type of round over and over judges don’t (sic) begin to be affected by their own round experience?
Critics of MPJ do not assert that it is the primary reason for community fragmentation, but it adds to a list of challenges for small programs, including
MPJ, budget considerations, disclosure, coaching, national vs regional, policy vs value, and a whole lot more. Almost all of these have been written about extensively on this list... what remains to be seen is tangible solutions.
Speculation #4: Many coach-judges are alienated by competitors’ seeming preference for younger critics
I have personally heard several coaches refer specifically to ageism in debate, and perhaps this, too, is inevitable in a competitive, youth-based activity. But an academic, intellectual activity? The alienation of older critics, many of whom are tenured professors whose professional growth necessitates decreased involvement with weekend-to-weekend debate, hurts the academic image of debate communities. Tom Preston observes that
no activity in academia can afford to drive away its senior faculty members. Yet mpj as practiced now has the propensity to do just that, and the inverse academic hierarchy it creates in our subdiscipline often makes us a laughing stock within and without the communication discipline–listen to the former debate coaches at NCA who are now chairs and deans as they talk about the status of debate, and you'll see what I mean--although some are supportive, many are quite antagonistic.
Those wishing to make academic debate more academically respectable face an uphill battle if they simultaneously support MPJ. In this sense, the perspective of "older coaches" makes sense: In no other competitive activity, athletic or intellectual, can competitors choose their umpires. The inversion of learning potentially created by such freedom, combined with the complete dismissal of the idea that debaters should, at least sometimes, expect to practice public (read slower!) advocacy, further alienates us from our non-debate peers, limiting our own universe of discourse.
Speculation #6: MPJ may improve judging by rewarding accountable, constructive feedback and decisions from judges who want to be chosen by teams
The merits of the traditional strike system, a limited number of strikes commensurate with the number of judges in the tournament, were based upon the idea that we deem certain judges less desirable to hear our teams. This implies that we deem other judges more desirable. We ought to examine our reasons for doing this in the first place. One reason is that we expect a certain performance from a critic. She must check her biases at the door, keep an accurate record of the debate, vote only on relevant and timely placed claims, and render a decision which both criticizes and encourages debaters. These do not seem like unreasonable claims.
If competition encourages self-improvement, then every judge, not simply the "elite," should benefit from MPJ. We have no data on whether this is true; it is certainly not true for those who leave the policy debate community. But should we refuse to consider the argument that judging should be "better?"
Speculative Conclusions:
Debate communities ought to discuss MPJ. We should raise questions about the experiences of judging, the effects of certain structures of competition upon our organizations, and the philosophies of competition we support.
At the same time, we should not confuse the issues. All coaches want competent judges hearing their teams. A system wherein teams can know they will be debating in front of critics with whom they are comfortable has almost as much pedagogical merit as a system which encourages adaptation and open-mindedness. And despite what we tell our debaters, there are bad judges–judges who don’t try, who bring an inordinate amount of bias into the debate, who are not constructive in their comments and criticism. Maybe the answer is a permutation. Perhaps, however, it is something better than anything we have now–something we have not imagined yet–as Jeffery Hobbes suggests:
I still think it is unfair to say that critics and educators who are against mutual preference systems are guilty of a double standard. Perhaps, they are arguing for a higher standard of accountability than a popularity contest--accountability based on actual performance with the potential to become better at their life work.