Conferences vs. journals in computing research
Such is the title of Moshe Y. Vardi Editor's Letter in the most recent Communications of the ACM (v52i5).
I'll excerpt it a bit:
What I'm referring to is the way we go about publishing our research results. As far as I know, we are the only scientific community that considers conference publication as the primary means of publishing our research results. In contrast, the prevailing academic standard of "publish" is "publish in archival journals." Why are we the only discipline driving on the conference side of the "publication road?"
Conference publication has had a dominant presence in computing research since the early 1980s. Still, during the 1980s and 1990s, there was ambivalence in the community, partly due to pressure from promotion and tenure committees about conference vs. journal publication. Then, in 1999, the Computing Research Association published a Best Practices Memo, titled "Evaluating Computer Scientists and Engineers for Promotion and Tenure," that legitimized conference publication as the primary means of publication in computer research. Since then, the dominance of conference publication over journals has increased, though the ambivalence has not completely disappeared. (In fact, ACM publishes 36 technical journals.)
*snip*
My concern is our system has compromised one of the cornerstones of scientific publication—peer review. Some call computing-research conferences "refereed conferences," but we all know this is just an attempt to mollify promotion and tenure committees. The reviewing process performed by program committees is done under extreme time and workload pressures, and it does not rise to the level of careful refereeing. There is some expectation that conference papers will be followed up by journal papers, where careful refereeing will ultimately take place. In truth, only a small fraction of conference papers are followed up by journal papers.
Years ago, I was told that the rationale behind conference publication is that it ensures fast dissemination, but physicists ensure fast dissemination by depositing preprints at www.arxiv.org and by having a very fast review cycle. For example, a submission to Science, a premier scientific journal, typically reaches an editorial decision in two months. This is faster than our conference publication cycle!
So, I want to raise the question whether "we are driving on the wrong side of the publication road." I believe that our community must have a broad and frank conversation on this topic. This discussion began in earnest in a workshop at the 2008 Snowbird Conference on "Paper and Proposal Reviews: Is the Process Flawed?" (see http://doi.acm.org/10.1145/1462571.1462581).
Interesting. I'm curious about all this and I wonder if any of the computing people out there who are reading this share Yardi's ambivalence. It's always seemed to me that the computing community's tendency to self archive on their own web space has been a great strength, probably leading to a somewhat lower probability of an arxiv-like system coming in and taking over like with physics. From what I've seen, probably 80-90% or more of most conference proceedings are available via authors' web pages.
Posts
12 comments:
This is a question well worth asking. You would think that as in earlier times, Program Committees met together, maybe there was something special (and positive) about that. I understand that most of this takes place online today; very few in person PC meetings.
A tradition, once established, is very hard to alter.
I think Lance Fortnow has written some stuff related to the broad question. See his blog.
Thanks, Carol.
I do generally follow Computational Complexity, but it's fallen off my radar recently. I'll have to add it as an imaginary friend in my Friendfeed!
I think it's important to distinguish between conferences. A full, 8 or 10 page paper at the conferences of the Association of computational linguistics or at the ACM computer-human interaction conferences are very hard to publish. Rejection rates are around 80% and several conferences allow a right to reply to reviewers now. The proceedings are also archived very carefully. Finally, I remember a time when computational linguists were at arxiv, too - cmp-lg ; not very active these days afaik.
Hi Perceval,
Thanks for the comment. It's my impression too that many CS conferences do have very strict peer review.
Yup, I can confirm many CS conferences are very intensively peer-reviewed, the world wide web conference http://www2009.org/ for example is presitigous with a 90% rejection rate.
I'm going to jump in and second (third?) what perceval and dullhunk said. Conference reviews are very strict and very thorough, even with the time pressures. This is especially true with the top-tier conferences (20% or below acceptance rates), but I've also found that the second- and mid-tier conferences (20-50% acceptance rates) and smaller workshops also take the peer review process very seriously. In my own experience, the reviews I've received (and given) for conference papers are definitely on par with those I've received (and given) for journal papers.
Hi Jane,
Thanks. Given the response here and on Friendfeed, I'm beginning to wonder if the original article overstates the worry about peer review a bit. It seems that most of you actual CS academics are fairly pleased with the state of peer review in your field.
See also essay by Dan Wallach (Rice U CS) apropos conferences http://tinyurl.com/dn7frz
Conference reviews are very strict and very thorough, even with the time pressures.I'd STRONGLY disagree with that, at least in theoretical CS. I have in mind the FOCS and STOC community. For those conferences, the reviewing is marginal at best and in no way comparable to a respectable journal. It's highly selective, in the sense that the acceptance rate is low, but not very careful or thorough. For example, it is understood that actually checking whether a highly mathematical paper is correct is too much to ask under the time constraints, so decisions are based on the reputation of the author and whether the proofs sound plausible. Partially or totally mistaken submissions are published far more often than they should be. The original idea was that this wouldn't be a problem: conference papers were supposed to be loosely refereed research announcements, and weren't supposed to count for anything if the authors didn't follow up with a journal paper. However, in practice people frequently don't publish journal versions, so the whole system breaks down. I personally know of half a dozen cases in which people proudly list seriously mistaken conference papers on their CVs and web pages, with no indication that there are major errors or how to fix them. (In one case the mistake is completely unfixable, but the author feels he is not being dishonest in not retracting the paper: after all, all he says is that it was published in a certain conference, and indeed it was. If the readers assume that it was correct, how is that his problem?) In fact, the official FOCS/STOC policy is not to publish corrections or errata, since those are supposed to be dealt with in the journal papers.
I have no personal experience beyond the FOCS/STOC community (and closely related conferences such as SODA, CCC, etc.). CS conferences may work better in less mathematical areas, where it is easier to read and evaluate papers quickly. However, note that low acceptance rates mean nothing whatsoever about the quality of reviewing. Rejecting 90% of the submissions doesn't mean the remaining 10% were wisely chosen.
Part 2 of previous comment:
One key difference between conference and journal refereeing is the ability to enforce changes. As a journal referee, I frequently recommend accepting a paper conditionally on certain changes. For example, clarifying a confusing part of a proof. With theory conferences, I can't do that. I can indicate changes I wish the authors would make, but then it's all up to them. Program committees are highly reluctant to reject papers on the grounds of lack of clarity, since that makes authors angry. (In practice, the PC is blamed more for rejecting good papers by well-known people than for accepting bad papers.) What happens is that poorly written or confusing papers (perhaps of dubious correctness) get accepted, with the authors being advised to clarify but not forced to. Responsible authors do their best, but others completely ignore the issue.
Submission deadlines and page limits contribute a lot to this. There's a long tradition in theoretical CS of writing the entire paper the week before the submission deadline, with the writing often extending to the last hours. This is just begging for bad writing and mistakes. The way they page limits come in is that they provide a convenient excuse for not going into detail about crucial technicalities. If people actually wrote a lengthy journal version that corrected all errors and explained everything thoroughly, it would be great. However, once the conference paper is published, the authors have received 95% of the intellectual credit, so they often don't bother to document everything thoroughly enough to receive the remaining 5%.
As you can see, I'm pretty upset about this. It makes me really angry, since the main argument in favor of conference publication seems to be rapid dissemination of research results. In the age of the internet, that's ridiculous. No conference in theoretical CS is as fast as the arXiv or reaches anywhere near as many people. There's a lot of value to meeting in person, giving talks, etc., but it should be decoupled from publishing research papers.
Thanks, Anon. I very much appreciate what you have to say.
I suspect it may have an awful lot to do with the particular sub-discipline.
I read your post as a result when looking for information. I should say that as a grad student I see a couple of good reasons why people rather apply to conferences than journals and probably part of the reason why the acceptance rate of conferences has decreased:
First, journal papers require harder work. They need to be a finished article with full results it takes around 6 up to 1 year to publish results in a journal.
2. Traveling. Why investing so much effort on a journal paper, when a conference implies you traveling somewhere, knowing people and actually visiting places to present your partial results.
I don't know but it seems clear to me that grad students prefer traveling (all paid) to exotic places rather than rewarding themselves with an actual journal publication.
Post a Comment