55

Every time I referee a paper, I dream of a system which would allow me to ask the author a question without troubling the editors. It would save time for everyone involved, most importanly the referee; in my case it would probably half the refereeing time.

Here is the system I envision: together with the paper, the journal would give the referee a link to a secure web form where the referee could type a question which would automatically be emailed to the corresponding author, who in turn could reply in a similar web form. Of course, the editors should have access to the logs of these exchanges, and the system should be set up so that the exchanges (if any) can only be initiated by the referee.

Question. Has this ever been implemented? Are there reasons this cannot (or should not) be implemented in mathematics journals?

Igor Belegradek
  • 28,360
  • 1
    This is a bit of a blog question, given to discussion rather than to answers. I'm extremely interested; I'm not downvoting; but I'm voting to close. Because I can't resist stating an opinion: as an author, I would make an idiot of myself on such a system, because I'd feel pressured to give quick, wrong replies to show confidence in my result. I often e-mail complete rubbish, for example. A system which favours a gradually thought-out response to a referee report works to my advantage. – Daniel Moskovich Jan 02 '11 at 19:48
  • 22
    I cast a vote against closing. – Andrés E. Caicedo Jan 02 '11 at 19:55
  • 2
    @Daniel, I do not suggest a "live chat" between a referee and an author. There is no time pressure for the author to reply immediately. – Igor Belegradek Jan 02 '11 at 19:58
  • 1
    The idea suggested is a good one, but I don't actually believe in the usefulness of anonymity for referees. It either impairs the flow of communication or is bypassed (how many people here have written/received a message along the lines of: I was reading your interesting preprint, and I have the following questions...) – Igor Rivin Jan 02 '11 at 20:25
  • @Igor Rivin: I send those emails all the time, nothing to do with refereeing. – Andrés E. Caicedo Jan 02 '11 at 20:31
  • @Andres Caicedo: you mean you send such email but NOT when refereeing? – Igor Rivin Jan 02 '11 at 20:34
  • @Igor Rivin: :-) – Andrés E. Caicedo Jan 02 '11 at 20:40
  • @Igor Rivin, some think that "anonymity for referees" makes the referee report more objective. Of course, a referee could ask questions privately like you suggest, but those questions are sometimes ignored especially if the issue is tricky or sensative. On the other hand, nobody can ignore a question from a referee. – Igor Belegradek Jan 02 '11 at 20:41
  • 8
    When I'm refereeing, I often have the urge to create a fake gmail account and email the author my questions. However, I've never had the guts to actually do this (and probably never will). – Andy Putman Jan 02 '11 at 21:11
  • 2
    Some fields (e.g. maybe biology) use double-blind refereeing, so the referee doesn't know who the author is. Primus (in which I have a forthcoming paper) is a math education journal that does that. This would make Andy Putnam's suggestion unworkable without going through the editor. I like that suggestion. Might some editors be offended if the referee did this without consulting the editor first? – Michael Hardy Jan 02 '11 at 21:33
  • @Daniel: I would assume that, in most cases, things that are not clear to the referee are usually not actual gaps in the proof but bad writing on the author's side. This isn't a question of "confidence in the result". – darij grinberg Jan 02 '11 at 21:34
  • 1
    @Michael: An honor/shame culture isn't quite the best environment for mathematical research. If referees cannot directly communicate with authors because editors might be "offended", we've got an inefficient system - that's all. – darij grinberg Jan 02 '11 at 21:37
  • I agree with Andres Caicedo. I've sent e-mails asking questions about preprints, but, as far as I can remember, never when I was the referee. When I'm refereeing, I send my questions via the editor. – Andreas Blass Jan 02 '11 at 21:44
  • 7
    As for Igor's question (or rather, proposal), of course I sympathize with it, but I reject its very premise - that we need to fix the peer review system of mathematical journals, rather than replacing it by a completely new one. What comes to my mind is, for instance, an arXiv-like database with phpBB2-like discussion, MO-like voting, and some additional structure such as the following one: (1) People with high rep or academic endorsement can have their comments appear directly next to the paper, so instead of the rather small information that we get from a journal publication ("this paper... – darij grinberg Jan 02 '11 at 21:45
  • 20
    @Darij: given some of the low-quality stuff I see on the arXiv, and read about as appearing in the proceedings of a faraway country of which I know nothing, do you really want a publishing model where everything is discussed and voted on? and do you believe that the highest-voted questions and answers on MO are really 7 times better than others? – Yemon Choi Jan 02 '11 at 21:47
  • 9
    I'm not a big fan of anonymity, but if you want junior faculty at all involved in the refereeing process, it is indispensable. How are you going to get someone's candid opinion otherwise? – Thierry Zell Jan 02 '11 at 21:49
  • 4
    ... lives up to the standards of this journal, as confirmed by two anonymous referees who will never get to take any responsibility for their review and probably never get any financial or other benefit from it either, except for a boost of their egos, but they have skimmed throug this paper and found the pictures cool"), we would get something of much more honesty and therefore of much more use: "(insert name here): liked the proof of Theorem 1, haven't read further; (insert different name here): seems correct, but proof of Theorem 6 relies on unwritten finite-generated assumptions... – darij grinberg Jan 02 '11 at 21:50
  • 4
    Put another way: I am not wholly convinced that popularity/trendiness = worth – Yemon Choi Jan 02 '11 at 21:50
  • 1
    etc.". You could google for the names and find out how much these comments are worth. (2) Viewers should be able to view weighted votes. Of course, this should not be done ad absurdum (too much weighting would allow for de-anonymization), but I want to know how much academically endorsed members value some particular paper, and not just how much the average visitor values it. – darij grinberg Jan 02 '11 at 21:52
  • 4
    I like Igor Belegradek's suggestion. I'd prefer, though, that the editor not merely have access to a log of the interchange but be notified (by e-mail) that an exchange is going on. Presumably, the web system could be organized so that, when it sends a question to an author, it also sends the editor a message saying "referee n of paper m by author x has sent the following question to the author" (and similarly for an author's replies). This wouldn't require any action by the editor (except maybe moving the message to an appropriate folder, or deleting it as the system would keep a copy). – Andreas Blass Jan 02 '11 at 21:53
  • 1
    @Yemon: I have addressed this in the last part of my comment. I am not totally convinced that it is going to work, but at least we can easily do much better than a blind internet popularity contest. Also, I am not sure about your "low-quality stuff I see on the arXiv" remark. My experience with cranks and math spam is that, usually, it is ridiculously easy to filter it out. – darij grinberg Jan 02 '11 at 21:56
  • 2
    Andreas, I believe the editor should have more control of the process. For example, a referee who may be floundering with the paper should probably not be refereeing it. The editor should know this, and the author should not, unless the editor thinks the author should. This is not always because of the quality of the information, but always because of the price of the ink or the electrons. So a simple notification to the editor is not enough; the editor should decide if the questions are appropriate to pass on. Gerhard "Ask Me About System Design" Paseman, 2011.01.02 – Gerhard Paseman Jan 02 '11 at 22:28
  • @darij: our comments crossed so I think I understand your position a bit better now. With regards to my comments about the arXiv, I was not referring to crankery, but to low-level specialist stuff that is either incorrect or missing the point, or just IMHO rather superficial -- you will forgive me if I don't actually point the finger at anyone right now. (My point is that I fear the signal/noise ratio will get too low; and if one corrects for this by letting Eminent Specialists wield quality control, then you seem to be back where you started with the traditional model.) – Yemon Choi Jan 02 '11 at 22:32
  • @Yemon: There still is a difference to the traditional model: In my model, there is no "editor" who is supposed to supervise the process and be the main link between author and reviewer. This difference is central to the question at hand. – darij grinberg Jan 02 '11 at 22:35
  • Discussion on this question opened on meta: http://tea.mathoverflow.net/discussion/876/on-refereeauthor-communications/ – Daniel Moskovich Jan 02 '11 at 22:43
  • 3
    @Gerhard: You seem to assume that the editor is able to distinguish "the referee is floundering with the paper because he is not competent" from "the referee is floundering with the paper because the paper is badly written" or "the referee is floundering with the paper because of some trivial misunderstanding which would be immediately cleared up by a 2-lines email exchange between referee and author". I doubt this is the case, and whenever it is not, your approach will often turn out to be a waste of time and e-ink in itself. I think the editor SHOULD be able to read the emails between ... – darij grinberg Jan 02 '11 at 22:43
  • ... the author and the referee (already for transparency reasons; corruption is quite an issue in life sciences), but if we force all correspondence between author and referee to pass through the editor as a relay and potentially become stuck at that relay, we are letting the bureaucracy take over the process. – darij grinberg Jan 02 '11 at 22:44
  • It is amazing to me that there are no journals implementing a scheme like the one proposed. Do any of the referees here have any insight as to why not? – Dr Shello Jan 03 '11 at 01:45

7 Answers7

15

I did exactly as others have suggested: created a fake email account (this was pre-Gmail; I think I used "sithmail") called [author name]reviewer, and corresponded that way.

Actually, I did this not to ask about the paper, but to strongly recommend to the author that he put his article on the arXiv before signing anything (and pointed out that the editor himself frequently posted to the arXiv). I didn't see any point in putting the editor into a difficult position, about whether he was allowed to suggest to the author that the author arXiv the article.

Allen Knutson
  • 27,645
12

For many years now, I have refereed only non-anonymously. It has not been uncommon that the authors of a paper I recommended with reservation—or, especially those I rejected—to write me directly, and we subsequently engage in an online (but off-journal) dialog to improve their paper, often for another venue. Almost uniformly these dialogs have been productively positive. Easy for me to say, because of course I hold all the power strings.

Nevertheless, what Igor B. suggests seems to me would be a significant improvement on the current system. My experience with the more one-sided communication I have experienced predicts that it could work well.

Joseph O'Rourke
  • 149,182
  • 34
  • 342
  • 933
  • 2
    It would be desirable to have a system that would work regradless of personal bravery. Also I know for a fact that some editors consider an anonymous referee (or anonymous recommendation letter writer) to be much more objective. – Igor Belegradek Jan 03 '11 at 04:37
6

The review system for research projects in Italy (during the last few years; it is possibly going to change now) was based on comparing the opinions of two different anonymous referees. At some point the two referees have to exchange opinions with each other and reach an agreement, while remaining anonymous. In practice, this is achieved by channeling communications through a website based at a central location. I post my comments, the other referee can read them and post his comments, etc. Since we both know that our exchange is watched by some official at the Ministry, we feel some pressure as to not revealing our identity. I think this works pretty well, and should work also for peer review.

Actually, several journals have already an electronic system for reports; at the moment it is a little awkward to use it as a communication tool, because of the many steps involved (the messages must be approved through the chain referee->editor->chief editor->editor->author or something like that), but it should be quite easy to make it faster and transform it into a supervised direct channel between referee and author.

Piero D'Ancona
  • 8,775
5

I think nothing must be changed in the present procedure.

1) We send a paper to a journal when we think that the paper is ready for publication, if it is not, we don't send it. We spend more time on it, if necessary we left the paper 1 or 2 weeks in our drawer before sending it, just in case.

2) Once the paper is in the hands of the referee, the referee judges a work assumed to be complete, not a draft. If it is not accepted the first time but may be improved, it's referee's job to tell what to do and the journal board's job to be the intermediary.

3) After one or two rounds the paper is definitely accepted or rejected. That's the way it works.

The referee is not a teacher, he is a peer and judges as a peer.

Stefan Kohl
  • 19,498
  • 21
  • 73
  • 136
Patrick I-Z
  • 2,259
  • 11
    I only said that my job as a referee would be much easier if I could ask questions with no hassle. Currently, I often do not ask questions because this is complicated, and involves troubling busy people. As a result I waste a lot of time. The existing system surely works, but it does so at the expense of people like me. – Igor Belegradek Jan 02 '11 at 22:25
  • 6
    Also at the expense of the readers, who would mostly profit from the unasked questions (were they asked). – darij grinberg Jan 02 '11 at 22:37
  • I am also a referee, sometimes, and I prefer the way it works, don't take offense :-) If you think that something is wrong in a paper but the paper is good just write down your suggestion (it happened to me recently, I accepted after one round), if you involve too much time, you are a professional, then the paper is not good, reject it. It may be also because you are not the right person to refer the paper, just send it back. That is my way ;) – Patrick I-Z Jan 02 '11 at 23:03
  • 1
    The referee-author comminication I suggested would be an extra feature, which a referee may or may not use; those comfortable with the current system could referee exactly as before. – Igor Belegradek Jan 03 '11 at 04:24
  • Hi Igor, I understand your position. I had to contact, in the past, a young author to improve the redaction of his paper, the result deserved to be published but the paper was a little bit messy. I found a way to do. My point is that it everyone in this situation may find his way, according to some elementary rules of deontology. It may be not necessary to formalize that in the process of referee, I am afraid that it will complicate a rather simple process, we are used to, without a real gain. But this is just my opinion :) – Patrick I-Z Jan 03 '11 at 16:42
5

Anyone using one of the suggestions based on "create a fake free email account" should watch out for the possibility that the free email provider will include your IP address in the message. If it does, it is trivial for the recipient (here, the author) to determine which university owns that IP address, which would probably let them make a very good guess at your identity (check the faculty web page for Podunk State University and see who is the expert on reticulated splines). So that would give only the illusion of anonymity.

For example, Yahoo Mail does include the sender's IP address. Gmail apparently does not do so currently, or at least doesn't include it in any obvious way (it could be encoded in some way that is well known but not by me).

Nate Eldredge
  • 29,204
1

My opinion is based much more on an idealization of the process of refereeing, rather than the actual process itself, so this opinion may have little worth. It appears to me to have much common sense, so I will share it anyway. (This may all be in whatever procedure or policy manual for referees the journal has.)

The goal of the process is to have someone spend resources putting ink to paper. There are other goals, such as scientific progress, but let us take the economic viewpoint. The representative of the people spending these resources is the editorial team. The team is involved in the process of deciding how to arrange the ink. Your questions and eventual evaluation are key in this decision. I suggest that you send a dated letter (or email) which they can easily extract and forward to the author. In short, they need to be actively involved in the forwarding of such questions.

There are several techniques you can use which ease the process. For one, leave any reference to yourself out of the letter, so the editorial team can put such in if needed.

If you leave whitespace between the questions, you encourage the answerer to fit their answer in that space. Having the answerer meet the challenge of giving a concise answer usually promotes proper answering, and adds value to the process.

Nowadays, many people use a computer to search for a substring. Even so, use every appropriate landmark to help locate the relevant text, as well as an appropriate string, e.g. "Near the end of your second section, just before Lemma C, you have the phrase '...'. What were you thinking when you included that phrase?" .

With the letter to the author, include remarks intended only for the editorial team, saying what will happen if the questions get answered promptly, e.g. your referee report will come six weeks earlier, or you anticipate a particular problem which may evaporate if the right answers come, otherwise you will recommend the problem be fixed. If the editorial team knows explicitly how passing the letter on will benefit them, they will be more motivated to pass it on and save time.

Arrange it so that you only need to send two or three such letters at most. This will involve phrasing your questions properly and accumulating them so that this part of the process is economically advantageous. Also, while becoming a referee, make sure you understand and agree with enough of the editorial process, so that it will allow timely transmission of such questions and answers.

Gerhard "Ask Me About System Design" Paseman, 2011.01.02

Gerhard Paseman
  • 12,968
  • "Arrange it so that you only need to send two or three such letters at most." This is not always possible (answers to questions often provoke new questions) and often far from desirable (some papers are considerably improved in proof - this happened with some of mine, for example -, and improving a paper is quite difficult if you can just exchange 3 letters...). – darij grinberg Jan 02 '11 at 22:52
  • darij, the real world is different from my ideal, of course. If two or three letters results in significant improvements, then I view it as a stage where the old version is rejected, and the process begins anew with the improved version, and another two or three letters as needed. Of course, the real world will do what it wants anyway, common sense or no. From an economic standpoint, two or three letters seemed appropriate to place in my idealized version. Gerhard "Ask Me About System Design" Paseman, 2011.01.03 – Gerhard Paseman Jan 03 '11 at 09:22
0

How about this:

  • First adopt Andy Putnam's suggestion in the comments above: The referee sets up a pseudonymous gmail account to correspond with the author.

  • If the referee's anonymity matters, one would want to use a different pseudonym for each new paper one referees. Otherwise, even if one can't be identified, authors might figure out that the referee named Rollland is the same person as the referee of some other paper, also named Rollland.

  • Some system (a web site) would be available to pick fake names from (and tag them as already taken, if they are) so that the referee's particular personal style of choosing various fake names would not get decrypted. Maybe just 9-digit numbers or the like.

  • If many referees do this and editors object, then their standard boilerplate requests to referee might include their policy that this is not to be done.

All this assumes refereeing continues to be single-blind. The desirability of blind refereeing is debatable. And it won't work with double-blind refereeing unless the editor assists somehow.

Michael Hardy
  • 11,922
  • 11
  • 81
  • 119
  • 1
    What would force the author to answer any questions from a fake google account? Questions from a referee cannot be ignored. – Igor Belegradek Jan 02 '11 at 22:15
  • @Michael : I've never heard of a mathematics journal using double blind refereeing. – Andy Putman Jan 02 '11 at 22:54
  • 1
    @Igor : Of course, you would identify yourself as the referee... – Andy Putman Jan 02 '11 at 22:54
  • @Andy: The one I mentioned is actually a math-education journal, Primus. So now you've heard of that.

    @Igor: "Force" is a bizarre word here. The author has an interest in being understood. If the referee says "Why did you decide to include an account of X in this paper?", the author might want to answer "You misunderstood. I didn't mean X; I meant Y." What author would need to be forced to do that? Admittedly, this does raise a question of how the author knows the person claiming to be a referee really is a referee. I can think of ways to address that.

     – Michael Hardy Jan 03 '11 at 01:01
  • ....such as the author giving the referee an ID code at the time of submission. – Michael Hardy Jan 03 '11 at 01:01
  • 4
    Long ago, the Proceedings of the AMS experimented with double-blind refereeing. I refereed one paper that way, but I had, just a few weeks before, heard the author speak on that work. So my report began with something like "If the author is X, accept subject to the following suggestions; if the author isn't X, put him in touch with X, who presented the same results at ..." – Andreas Blass Jan 03 '11 at 02:43
  • Even after the Proceedings abandoned the idea that all refereeing should be double-blind, authors retained (for a while --- I don't know if the option still exists) the right to ask for blind refereeing by submitting a second, blind version of the paper. During part of that time, I was on the Proceedings editorial board, and I don't remember getting many requests for blind refereeing, but there was at least one from a crank. – Andreas Blass Jan 03 '11 at 02:47
  • 3
    @Michael: I have several concerns with the "fake google account" idea. First, the communications are neither approved not supervised by the journal. Second, the author may not believe that this is not a prank, and there is a real referee on the other side. Third, a message from a fake account might never pass the spam fitler. – Igor Belegradek Jan 03 '11 at 04:32
  • @Igor: I addressed the problem of approval by the journal; it might become standard practice for the editor to inform potential referees of whether the practice is allowed. I also addressed the problem of the prank by suggesting maybe an ID code given by the author to the editor to pass on to the referee. – Michael Hardy Jan 03 '11 at 06:45
  • 1
    Double-blind refereeing seems problematic, because it would be easy to take a good guess at the author by seeing whose name appears most often in the references. And you can't very well blind those, because the referee needs to be able to read the cited papers. How did Proc. AMS deal with this? – Nate Eldredge Jan 03 '11 at 08:45
  • 1
    @Nate: I'm not sure the Proc. AMS dealt with the self-citations issue at all; I think they just wanted the "blind" version of the paper to omit the author's name and affiliation and to make the obvious changes in the text (change "I proved" or "the author proved" to "Blass proved"). An author who really wants blind refereeing should probably avoid excessive self-citation. (I think that, during the time when blind refereeing was mandatory, rather few authors really wanted it.) – Andreas Blass Jan 03 '11 at 15:43
  • 1
    @Nate: "Would" is a bit of an odd word to use here. Why not look at what actually happens in those journals that use double-blind refereeing (not math journals, as far as we've seen here, unless Primus counts) instead of talking about what "would" happen? Anyway, is someone actually proposing double-blind refereeing? Clearly if you had double-blind refereeing, then this whole thing would not work without the journal itself setting up some channel for communications. – Michael Hardy Jan 03 '11 at 17:30