What's wrong with STOC/FOCS? [2009]

by Oded Goldreich

Preface: This position page is intended to explain my current feelings, views and actions regarding STOC/FOCS. That is, my primary motivation here is not to convince others that my views are correct, but rather just to explain my own views and actions to people who do not agree to them. Needless to say, I will be delighted if other share my views, because this will provide basis for a change.

Postscript (2012): My later essay On Struggle and Competition in Scientific Fields is very related to the issues discussed in the current webpage. In fact, Sec. 2.2 details the foregoing discussion, after a theoretical framework is presented in Sec 2.1.

For sake of clarity, I will use various methodological dichotomies, which confront notions that do not exist in reality in a pure form. In reality, the conflicting notions are mixed, and the issue is one of balance between them.

My first distinction is between competition and contents. Indeed, any competition refers to some contents, and any human activity can be viewed through the lens of competition. Still, there are contexts were the competition aspect is more dominant and others where is its less dominant. My starting point is my subjective feeling that, in recent years, the competition aspect became more dominant in STOC/FOCS.

Note that I'm not saying that the competition aspect was not present in STOC/FOCS in the past. On the contrary, my own experience (which dates to the early 1980s) is that STOC/FOCS were always marked by a (flavor of a) competition. But, in the recent years, I see a greater obsession with the competition aspects and a decline of interest in anything that is not competition-like including the actual contents. Concretely, I hear and overhear more discussions of which paper "got in" and which did not "get in", which paper got which awards, and which is better than which. And, hear and overhear less discussions of the contents of various works, what makes them interesting, and what can be "carried home" from them.

Can I prove, using hard facts, that the above is the case? Certainly no. But, assuming the distinction between competition and contents, one can easily imagine circumstances in which the competition aspect is less dominant and others where it is more dominant. Thus, the question of whether this aspect can be made less dominant in STOC/FOCS makes sense in any case. It is also common sense that making the competition aspect less dominant provides room for other aspects; specifically, to keen interest in the actual contents of scientific works.

An alternative way of demonstrating what is currently wrong with STOC/FOCS is to make a distinction between the interests of the authors/speakers and the interests of the readers/attendees. In PCs (and in outside references to the PCs), you often hear references to the legitimate (or illegitimate) interests of the authors, and the entire discourse of fairness evolves around these interests. In contrast, you rarely hear discussion of the interests of the conference's attendees, which is indeed very odd since the conference is supposed to serve the attendees.

Competition promotes superficial measures and attitudes, and thus the dominance of the competition aspect in STOC/FOCS (as well as the decline of interest in the contents itself) yields disregard of conceptual aspects and focus on technical ones. This leads to the second distinction that I wish to make: a distinction between the conceptual and the technical.

Each real scientific work exhibit a mix of a conceptual message and technical details that support and instantiate this message. I wish to stress that the conceptual message need not be a novel model or definition; this message may be very "technical" in nature (e.g., a novel technique, or even merely the fact that the complexity bound on some problem was improved). Still, there is always a conceptual message (which may be less interesting or more interesting), or else the technical details are meaningless (at the last account). On the other hand, a conceptual message with no technical details does not appear in the domain of science.

Likewise, the evaluation of scientific work (see related posting) is typically a non-trivial function of both its conceptual message and its technical details; that is, in typical cases, none of these two aspects should dominate the evaluation. Unfortunately (at least according to my feeling), recent PCs of STOC/FOCS have shown little interest in the conceptual messages of the various submissions, and have confined their attention to evaluating the technical details (and specifically their "difficulty"). This is related, in my opinion, to the current dominance of the competition aspect, because technical difficulty is easier to evaluate (and argue about) than the importance of a conceptual message. Indeed, most reviews and discussion tend to focus on the technical difficulty of the various submissions. In general, I note an increase in the fraction of superficial reviews, which is to be expected when the focus of the review process is to serve a competition.

To demonstrate my claim regarding the bias of recent PCs towards technical difficulty, let me consider the profile of typical papers accepted at STOC/FOCS (i.e., the bulk of the program). Recalling that the bulk of the accepted papers are not significantly better than many of the non-accepted submissions (see related posting), I note that submissions of average technical difficulty are prefered to submissions of lesser technical difficulty that carry a much more important/interesting conceptual message. That is, an "advantage" in technical difficulty carries much more weight than a significant advantage in the conceptual message (which, at best, is discarded as having secondary importance).

[Indeed, a new venue in which I'm involved, called Innovations in Computer Science (ICS), intends to give higher weight to the conceptual message of the submissions. The name chosen for ICS is supposed to reflect this commitment.]

Answers to some FAQ and some Additional Thoughts

Q: Can you offer any evidence for the changes you claim re STOC/FOCS?

The most objective evidence regarding the competition aspect is the relatively recent establishment of many new awards (by the TOC community), and in particular the introduction of the "best paper awards" at STOC/FOCS.

As for the disregard for the conceptual message, this is evident in almost every STOC/FOCS review that I saw in recent years. I wish to stress that I am referring to the reviews themselves and not to the PC decision (which may be unrelated to all the reviews that one may see from outside the PC). My point is that almost all the reviews that I saw showed no interest at all in the conceptual message and were focused mainly on evaluating the novelty and difficulty of the technical development. I wish to stress that I don't dismiss technical novelty, but I reject it as a SOLE criteria for evaluation of submissions.

Q: As a person who, in the past, defended the value of the "standard research" (including the "unglamorous technical work" involved in it), it is odd that you now advocate venues that promote the opposite (e.g., ICS focusing on "new directions").

I think the texts of ICS are being misunderstood. These texts are not dismissing the value of the "standard research" (including the "unglamorous technical work" involved), but rather say that works that follow a non-standard path (i.e., due to their innovative character) are being dismissed when they lack perceived technical difficulty. But, in my opinion, the problem is wider -- as explained above: The issue is what I call a balance between the weight given to the conceptual message and the technical details.

Getting back to ICS, my reading of its texts is more general; that is, I read that ICS claims that there is a value also to conceptual steps that are not coupled with hard technical difficulty. We (ICS) do not dismiss the latter, but are concerned at the common dismissal of the former!

In any case, I see a world of difference between trying to force one vision of what TOC should be doing (e.g., application-driven research) versus arguing for pluralism (in which both conceptual and technical steps are appreciated)!

Luca Trevisan (in his blog in theory [June 9, 2009]): Still, I think that initiatives such as ICS will not correct an imbalance, but rather will create an imbalance, penalizing the necessary, hard, and unglamorous technical work...

We differ in our view and analysis of the current reality as well as with respect to our concerns regarding the future. I think that there is a significant imbalance in the review process (and it reflects a similar imbalance in the attitudes of the community), and I hope ICS and/or something else can correct it. I see absolutely no danger of "penalizing the necessary, hard, and unglamorous technical work" in the near future. I promise that if I ever see such a danger, then I'll fight against it (no less than anybody else).

Q: Could you clarify your distinction between conceptual message and technical details.

Indeed, a common misunderstanding regarding my views is that a conceptual message must be a novel notion or model or a new approach to a problem. I wish to stress that the conceptual message may also be more technical in nature. It may be a new technique (assuming that its essence can be communicated in high-level language) or merely a statement of a new result. (In all cases, the question is what is the importance of the conceptual message communicated by the paper.) To illustrate my point, let me pick Johan Hastad's celebrated paper Testing the Long Code and Hardness for Clique [STOC'96]. This highly technical work is also loaded with extremely important conceptual messages, including Needless to say, on top of this there is also a hairy Fourier analysis, which I call the technical details of this work. Indeed, Johan's paper would have no problem being accepted to current STOC/FOCS, because its great conceptual contributions are coupled with a difficult technical proof. What concerns me is papers of significant conceptual contribution (although not of Johan's paper) that are not coupled with a difficult technical proof. That is, referring to my favorite grading scale, I claim that while papers with a 6-7 grade on technical difficulty and a 3-4 grade on their conceptual message are often accepted, papers with a 7 grade on their conceptual message and a 4 grade on their technical difficulty are rarely accepted.

As a final comment regarding technical difficulty, let me note that technical difficulty is often misinterpreted as using and/or referring to "non-elementary mathematics". Although there is a positive correlation between the two, they are not identical. Consider, for example, Ran Raz's proof of the Parallel Repetition Theorem or Johan Hastad's proof of the Switching Lemma.

Additional discussion of the recent preoccupation of STOC/FOCS's PC with technical difficulty appears in Salil Vadhan's comment [June 12, 2009]). Below, I reproduce extracts from his comment.

... there are many different kinds of [scientific] contributions: making progress on known important problems, introducing new models and questions, developing new techniques, bringing simplicity and clarity to previously complex/confused areas, drawing new connections between topics, etc. ...

Note that some of the above kinds of contributions might be called "technical" (e.g., making progress on existing important problems, developing new techniques) and others "conceptual". But purposely missing from my list is the question of how "difficult" or "easy" the paper is. Indeed, I feel that a paper's "difficulty" is orthogonal to its value, and should not be a significant criterion in deciding whether to accept it. Instead, we should be trying to assess how much we *learn* (or will learn) from a paper, how it contributes to advancing the state of knowledge in the field. We may learn a lot from a difficult paper because of the significance of the final result or because of techniques developed along the way, but either way, the paper's difficulty is not the *reason* for its value. To some extent, the same holds for simplicity - if we prefer simpler solutions (when they exist), it is because they tend to clarify our understanding, tend to be more efficient/practical, etc.

N.B.: Even if one may claim a positive correlation between learning from a result or a proof and the difficulty of the proof, the two things are not identical. Thus, we should consider what we learned from the paper and not whether the paper is difficult.

Q: Where are the potential ICS papers, which were not accepted to STOC/FOCS

Some of these have appeared in special area conferences and some have not appeared in any conference. In both cases, the impact of these works on TOC has been reduced and/or delayed. The question of whether a certain work should appear in a TOC-wide conference or in a special area conference is addressed in my opinion page Where to submit.

Q: Why not try to change STOC/FOCS, rather than start a new conference?

I agree that one should try to change STOC/FOCS itself (as well as other conferences that are also effected by the same set of wrong attitudes, which in turn are rooted in the attitudes of the community at large). In fact, some of us tried to influence STOC/FOCS in the past and will continue trying in the future, although our success so far has been very limited (to say the least). Indeed, the current attempt to start a new conference, committed to different attitudes, does not contradict the attempt to promote these attitudes in the existing conferences. On the contrary, the "challenge" posed by the new conference may help in effecting the latter change...

Q [Jan'10]: Did the 1st ISC meet your hopes?

No, at least as far as my impression of the operation of the PC is concerned. While the program of ICS'10 looks very good, and while one can sense a more conceptual profile in it, my feeling is that my main concern regarding "attitudes of competition" etc was not address. In retrospect, it seems that a deeper consensus on the goals and a more conscious effort of the steering committee is necessary for effecting such deep changes in attitudes.

A thought [Feb'10]: The career promotion competition

It seems adequate to refer to STOC/FOCS (at their current disposition) as biyearly career promotion competition. From this perspective, it feels inadequate for me (or any other Full Professor) to participate in it. Indeed, my tentative decision is not to participate in this competition any more, except when the paper is co-authored by individuals seeking promotion.

A related essay [Nov'11]: On the status of intellectual values in TOC

My essay On the status of intellectual values in TOC addresses issues that are related but not identical to those raised above. The term "values" is adopted from Sociology, where it is defined as the set of beliefs of a society (regarding what is correct, good, and/or desirable). By "intellectual values" I mean a specific type of values; that is, those that advocate curiosity, study, and understanding. In particular, I believe that the TOC community holds (and should hold) both intellectual values and instrumental values. The issue at hand is the balance between them.

In the essay I study the status of intellectual values in the TOC community during the last three decades. Specifically, analyzing the motivational parts of papers that appeared in several STOC proceedings, I found evidence to my feeling that the importance attributed to intellectual values has declined in the last decade (or so). The said evidence is conditioned on a number of assumptions, which are spelled out in the essay. I then discuss three theories that may be used to explain the decline of intellectual values in TOC (or rather three phenomena that may cause this decline).

Most relevant here is a sociological theory regarding the evolution of scientific fields and the competition in them. It asserts that as a field become more successful (or, actually, is considered so from the outside), the competition within the field intensifies, and this creates pressures towards ``objective'' measures of accomplishment that can be reviewed from the outside. Such measures are typically oblivious of intellectual contents. Thus, under the reign of (externally monitored) competition, intellectual values decline.

Another related essay [Jan'12]: On Struggle and Competition in Scientific Fields

My essay On Struggle and Competition in Scientific Fields is very related to the issues discussed in the current webpage. In fact, Sec. 2.2 details the foregoing discussion, after a theoretical framework is presented in Sec 2.1.

Back to Oded's page of essays and opinions or to Oded's homepage.