22

I have found that the most notable TCS conferences like FOCS, STOC, SODA, and ICALP are single-blind. That is the authors do not know the identity of reviewers; however, the reviewers know the identity of the authors.

I believe that the most ethical way of reviewing is the double-blind review, where neither the authors nor reviewers know the identity of each other. Then, why the top TCS conferences are just single-blind? What are the good reasons for doing it?

The only reason that I could think of is that a reputed author's name or institution could boost the confidence of the reviewer about the claims made in the paper. But again it can introduce bias.

Note: There are few conferences that do follow double-blind reviewing like ESA. Then, why this practice is not followed by the other top conferences?

Inuyasha Yagami
  • 1,531
  • 1
  • 8
  • 25
  • 4
    SoCG is currently having a vote centred around this question. Will be interesting to see the results and what the CG community thinks about that. – Tassle Jun 08 '21 at 13:15
  • 7
    Also there is a problem with posting your paper on arxiv when the reviewing process is double blind. – Bartosz Bednarczyk Jun 09 '21 at 04:33
  • 8
    None of these arguments against double-blind work if we do not require the process to be perfect, but just to aim to reduce the initial bias. I think there are no downsides in double-blind if it amounts to just omitting names and affiliations in the submission to the conference server. This is how the lightweight double-blind works in e.g. ESA, STACS, LICS, and PODC. – Laakeri Jun 09 '21 at 13:37
  • 3
    @Laakeri: I completely agree and would upvote this 100x if I could (the phrase I've been using is "don't let the perfect be the enemy of the good"). The main issue is then logistics, especially around handling conflicts of interest while maintaining double-blind. But I have found out from some organizers who have done this lightweight double-blind that there is a fairly easy way to handle this in easychair. I hope more conferences do this in the future. – Joshua Grochow Jun 09 '21 at 17:02
  • 1
    CCC is still not double-anon either. (Re my previous comment: I've asked one of the organizers I spoke to if they would feel comfortable posting here.) – Joshua Grochow Jun 10 '21 at 00:34
  • 3
    STOC is supposedly going double blind (forced to by ACM). – Yuval Filmus Jun 11 '21 at 16:53
  • 3
    Some AI conferences are now going triple blind: PC members don't know who the reviewers are. – Yuval Filmus Jun 11 '21 at 16:53
  • You mention that reputation can help a paper. Similarly, bad reputation can work against a paper. If researcher X is known to have errors in their papers, the PC might review their papers more carefully. – Yuval Filmus Jun 11 '21 at 16:56
  • 1
    There is an unstated assumption here that not only does OP think that double-blind is best practice, but in fact double-blind is objectively the best, and so the real question here, is why have TCS conferences "not seen the light". I think that a better way of stating the question would be, why did most TCS conferences not switch to double-blind, although many other CS conferences have. – Yuval Filmus Jun 12 '21 at 15:32
  • 1
    One could ask, why are reviews in TCS conferences not made public? I believe that this is the most ethical choice, and the only reason I can think of for TCS conferences not switching, is that reviewers are less likely to be honest (but this is offset by some other, unstated advantages, that are obvious). – Yuval Filmus Jun 12 '21 at 15:34
  • 6
    FWIW SODA this year (SODA 2022) is doing a "lightweight double-blind reviewing process" (https://www.siam.org/conferences/cm/submissions-and-deadlines/soda22-submissions-deadlines). – Neal Young Jun 14 '21 at 13:56
  • @YuvalFilmus: Public review solves some problems but not others. For example, public review would allow post hoc checking of bias, whereas double-anon is supposed to reduce bias before decisions are made. Now that some TCS conferences are starting to do double-anon, it would be great to see some data on the outcomes, e.g. measures of bias before vs after going double-anon, over several years. – Joshua Grochow Jun 14 '21 at 15:35
  • 3
    To the people who voted to close b/c answers would be opinion-based: I disagree. Q1 is about why conferences aren't yet double-anon. This could, in principle, be answered factually by people involved in organizing those confs. Q2 is about "good reasons for doing it". While theorems in this area are impossible - it's about the science of people - there can be good reasons that are as un-opinion-based as any when it comes to the science of people. – Joshua Grochow Jun 14 '21 at 15:40
  • 1
    I think it is a mistake NOT to have a double-blind review for the CS theory conferences. There has to be a rebuttal process as well because some reviewers write whatever nonsense and baseless comments they like. – Arash Rafiey Jun 15 '21 at 02:57
  • 3
    Also FWIW this site (which I curate) tracks double-blind reviewing status of top conferences across areas of computer science: https://double-blind.org. In addition, it contains links pointing to resources related to double-blind reviewing (academic papers, articles, and blog posts). – EmeryBerger Jun 17 '21 at 17:12
  • 1
    @YuvalFilmus: would you happen to have a reference about how ACM imposed a double-blind policy on STOC? I was not aware of ACM imposing this kind of choices on member conferences. – a3nm Sep 16 '23 at 22:03
  • @a3nm If I remember correctly, it was mentioned on Slack or equivalent, which wouldn't necessarily count as a reference. – Yuval Filmus Oct 29 '23 at 13:50

2 Answers2

16

This is not an answer to the question but too long for a comment by 1253 characters.

STACS 2021 used light double blind review. We used easychair for it, which provides some functionality for double blind review. You need the executive licence, which costs some money. Costs depend on the number of submissions, I think for STACS it was around 600 British pounds.

Overall, the process went quite smoothly. As a PC co-chair, you can of course see the authors. So I actively only handled very few papers, mostly those that no one wanted to have and there, I was rather far away from the topic. Instead I focused more on monitoring the process, stimulating discussion etc.

Authors can declare conflicts of interests when submitting. I had the feeling that this was used in an honest way. Furthermore, PC members get to see a list of all authors (without the paper titles) and can declare conflicts of interest against authors. This eliminates essentially all conflicts with PC members. (As a PC chair, you can see the list of all COIs. The authors' declarations and the PC's declarations matched with very little noise.)

More problematic are the subreferees. Easychair blocks referee request to authors of a paper automatically (if you use the email addresses from the list provided by easychair). Subreferees were asked to declare COIs if they were aware of it. In most cases, this worked well. There were a few cases (maybe 5 out of > 600) were I had the feeling that the subreferee correctly guessed the authors and wrote a biased (very positive) report. One could have a person (or committee) here, who sole purpose is to look for such conflicts but who is not involved in the PC work.

In a questionaire fill out after the notification, about 85% of the subreferees said that they did not actively searched for the authors. On the other hand, about 50% were able to guess the authors of at least some of papers they reviewed.

Markus Bläser
  • 2,938
  • 18
  • 19
10

As of 2023, the situation has changed, and all conferences mentioned in the question (and, indeed, all major generalistic conferences in TCS that I am aware of) are now using double-blind reviewing:

See also the website by @EmeryBerger tracking this information: double-blind.org.

However, as far as I know, the same is not true of TCS journals.

a3nm
  • 9,269
  • 27
  • 86