The role of program committees

by Oded Goldreich

The only role of a program committee (PC) for a conference is to select a program for that conference. This tautology is often forgotten, and the purpose of this short essay is merely to discuss this triviality (as well as its consequences).

The role of a PC is merely to select a program. It is not the role of a PC to rank papers by some objective criteria (putting aside the question of which criteria this may be and whether a meaningful evaluation can or should at all be based on objective creteria). The role of a PC is to select among submissions a subset of papers that are believed to be most adequate for presentation in that conference.

On the selection process

The selection is a compromise among different evaluations of individual PC members. As any compromise, such a selection does not have a rationale (although individual evaluations may have one) other than the rationale of social compromise. The (essential) irrational nature of the compromise is amplified by the fact that the evaluation of scientific work is not absolutely objective (but rather depends on subjective scientific views, beliefs and criteria).

The selection is based on partial reviews. Due to time constraints, the abovementioned individual evaluations are typically based on partial review of the submissions that does not allow for absolute verification of the results and comprehensive evaluation of their merits, not to mention the lack of a posteriori perspective on the impact of the work. Thus, even the individual evaluations are far inferior to what may be obtained in an ideal review process, let alone one carried out several years after the paper is published.

Decisions are binary - but merits are not. Even taking the wrong (see above) assumption that the PC can rank the submissions according to some objective linear order (by absolute, objective quality), this does not mean that a single threshold can provide a meaningful qualitative distinction between "good" papers and "bad" ones. Thus, the gap in quality between accepted papers and non-accepted ones, if at all existing, is typically quite small.
Furthermore, typically, there is a large number of submissions that are interesting enough to be included in the program but are not sufficiently interesting so that they must be included in the program. Thus, any decision regarding these submissions is within reason. Typically, the relation between the number of these submissions and the desired length of the program dictates that only part of these papers be included in the program (but one has to remember that also the part that is not included consists of papers that are not much different in quality than those that were included).

Still, PC are typically required to make a choice (i.e., to select for the program only a subset of the submissions). So a choice is indeed made as described above (in lack of better alternative). In view of the above, it is important not to attach to this selection (of a program) any semantics beyond the trivial one (i.e., a selection of a program). Thus, when referring to paper not selected to be included in the program, it is suggested not to use the term "rejected" but rather to use the term "not selected to be part of the program" (or at least the term "not accepted"). This suggestion is not made for sake of nicety and/or politeness but rather to reflect better the true nature of the decision.

On by-products of the selection process

Typical by-product of the selection process are "reports" or "reviews" written by the PC members and/or their sub-referees. It is indeed a question what to do with such by-product. My own suggestion to a PC would be:

Following the suggestions, such "comments to authors" may be useful (while minimizing the side effect of annoying the authors). I'd also suggest to explain the above guidelines in the preamble sent by the PC chair to the authors (explicitly telling them that the comments are not supposed to provide a justification for the PC decision). (Recall that, in my opinion, typically no justification can be provided for the PC decisions beyond the tautological fact that the PC (or a majority in it) decided this way.)

How to try to do the best selection

It all starts with selecting the PC itself. It is common wisdom that the PC common expertise should cover the various sub-areas that are part of the conference scope. More importantly, the PC should consist of people who are qualified to do the job (i.e., evaluate scientific work, exercise scientific judgement, be honest and open to discussion). What is often forgotten is that these qualifications (e.g., good judgment) are only weakly correlated with other qualities such as creativity (which is almost irrelevant for the PC's work). Furthermore, personality features like morality are even less correlated to creativity. Thus, the most accomplished researchers are not necessarily the best potential PC members: having achieved important results does not necessarily imply having good judgment, and certainly says little about moral characteristics. It is possible to make important research contributions while lacking good judgment (especially, in a broad sense) and having poor morals, but a good PC member must have good judgment (in a broad sense) and fair morals (e.g., firm commitment to truth, being willing to invest much time for the common good of the community, etc).

Operation of a PC: See my suggestions (including the use of numerical scores with clear verbal interpretation and the use of ex-submission information).

Evaluating the output of a PC (i.e., its selection). Given the dichotomy between the continuous scale of merits and the binary nature of the decision, a PC cannot be evaluated based on its performance on borderline cases. Furthermore, one should allow for small margins of error even in the operation of a good PC. Thus, a valid complaint against a PC is its failure to decide correctly on a clear-cut case (which unfortunately happens too often, because most PC are poorly selected and thus do a poor job).

Baruch Awerbuch's Maxim

The decision of a PC is a random variable with an expectation determined by the submission's quality and a variance determined by the PC's quality.

Back to Oded's page of essays and opinions or to Oded's homepage.