| 1 | Nov 2, 2011 3:38 AM | Maybe DBR-style review process is good for "outsiders"? Could be interesting to check this statistically. |
| 2 | Nov 1, 2011 3:08 PM | The overall philosophy that double-blind review is just intended to introduce *some* uncertainty about author identity and that this can be implemented *just* by removing author names and self-citing in third person -- initiated by Norman Ramsey for his ICFP and carried on admirably for POPL this year -- seems very attractive. What's distressing is how hard it turns out to be to implement. You start out saying "This is the whole policy," but then people start raising questions about particular cases (posting to Types OK?), and you start having to make difficult calls, and you start writing FAQs, which get longer and longer, ... and pretty soon it's a hassle again.
The other, more insidious, point is that DBR makes dealing with COIs much more challenging and time-consuming to deal with. In particular, PC members who notice that a paper could really use an outside review by some particular expert can't just ask that person directly; they have to ask the chair to ask the expert, in case the expert is actually the author. Maybe chairs are totally willing to do this, but I rather suspect the effect will be fewer reviews by "the real experts" on a given paper (e.g., the authors of the work on which this work is based, or its direct competitors, etc.) when those experts don't happen to be on the PC or ERC in a given year. This would be a shame. |
| 3 | Nov 1, 2011 1:47 PM | I think it is a good idea. Let us continue it! |
| 4 | Nov 1, 2011 8:47 AM | I have a concern about the double blind process if it continues. As my paper was rejected, I am afraid to talk to people about it incase POPL is DBR next year. However, talking to people is how you improve research. |
| 5 | Nov 1, 2011 8:36 AM | The process seemed fair and well-executed. |
| 6 | Nov 1, 2011 7:55 AM | it's a big change |
| 7 | Oct 26, 2011 1:41 PM | I was a PC member of POPL'2012. I found DBR is good from the viewpoint of a PC member since I was more careful to review the papers and paper bidding. I also found it is very different when we give the initial scores. I also found it is easy to review the system-language papers under DBR but it is hard to review theoretical papers under DBR. As the author of (number redacted), I am not happy under DBR by the reasons that we did not get the expert reviews as we usually had. (But I know our paper can be improved). |
| 8 | Oct 25, 2011 3:45 PM | I think double-blind reviewing takes away the distraction of who is writing a paper, making it easier for reviewers to concentrate on the paper itself and thereby improving reviews. |
| 9 | Oct 25, 2011 9:41 AM | DBR does not add much value over and above SBR as it is
quite easy to guess who the authors are from the style of
research, the results, citatations, etc.. The core problem is
the shoddy level of reviewing which is mainly because of three reasons:
1. Reviewers don't want to spend enough time reviewing the
paper. As a result if a contribution get slightly technical
there is a good chance of it being rejected. POPL/PLDI
have been reduced to essay writing contests. Thus
readability is of primary concern as opposed to scientific
content.
2. There are huge biases in the reviewing process and people
blatantly exercise their biases (this has been noticed by
by this authors during PC work and has been confirmed
by other researchers as well including some who were on
the POPL 2012 PC).
3. Reviewers are not held accoutable for their reviews. If
they truly believe in the nonsense they write then
they should stand by it by putting their name on it.
Personally, I feel there has been a fundamental shift
in the kind of work that is being accepted at top conferences.
Incremental well-written papers that can be reviewed over
lunch stand a much better chance that papers aspiring for
fundamental advances in the state-of-the-art (which is
what my motivation was when I took up research).
As a result of my experiences I feel sticking to this
line of work is a waste of my life and will actively be
looking for opportunities in other (professional) fields. |
| 10 | Oct 24, 2011 3:12 AM | I thought that the process worked well. |
| 11 | Oct 24, 2011 3:07 AM | Nothing to say. I would have appreciated a survey that asked for feedback about the rebuttal process though. |
| 12 | Oct 23, 2011 5:16 PM | I feel that DBR should be the norm because the content of papers should stand by themselves independent of the reputation of their authors; and SBR should be the exception. Revealing names at the time of author-response is also a good idea because one can finally clarify any open questions by direct reference to additional/previous work and such. |
| 13 | Oct 22, 2011 7:36 PM | Reviewers should sign their reviews, or at least their identities should be revealed after final notifications. |
| 14 | Oct 22, 2011 4:26 PM | Well done! |
| 15 | Oct 22, 2011 2:38 PM | I think hiding authors mostly eliminates reading bias. While reading a paper, if related to familiar prior work, one can often figure out a few of the authors regardless. I also don't like that double blind discourages discussion of work; in my opinion publication should be the finalization of a work that has already been commented on by a variety of others. |
| 16 | Oct 22, 2011 2:02 PM | An alternative review format which I've heard proposed recently, which admittedly has its own potential faults, is completely open reviewing, or single-blind in which the identity of the reviewers is known by the authors. A potential advantage of this format is that reviewers are held accountable for quality reviews. While I don't claim that quality reviews were an issue for my POPL 2012 submission, I think that most researchers have, at one point or another, gotten reviews that clearly showed lack of care or effort. Thus I think that it's worthwhile to at least discuss this format. |
| 17 | Oct 22, 2011 8:17 AM | One reviewer criticized us for not submitting source code, full test results, etc. which we actually did! It turns out that he/she wasn't given access until after having submitted his/her review. It wouldn't have changed the outcome for the paper though, but it was still suboptimal: (1) that a reviewer is unaware of the rules, and (2) to receive such criticism.
To make it constructive: one can imagine supplementary uploaded with the submission, such that it is available to a reviewer while reviewing. |
| 18 | Oct 22, 2011 5:56 AM | I think the quality of review was very high, and it felt that somebody had actually read the paper. |
| 19 | Oct 21, 2011 6:05 PM | DBR is good only if the reviewers never know the authors otherwise it can just be some form of hypocrisy. I would prefer reviewers to sign their reviews and have the courgare of their opinions. |
| 20 | Oct 21, 2011 5:16 PM | its good, keep it. |
| 21 | Oct 21, 2011 4:59 PM | The problem was not with DBR -- but one reviewer was clearly (name redacted) -- who did not read the paper -- made up a story on what the paper was about -- criticized it and then cited 6 of his own paper for citation one of which is his (paper topic and publication venue redacted). It is not clear among scholars how that paper passed the quality scrutiny of [a top conference] when it would not even pass a workshop on (topic redated). He not only wanted us to cite 6 of his papers -- there was no relevance of those papers to the content of our paper. Such reviewers must be banned. |
| 22 | Oct 21, 2011 4:56 PM | Our review scores changed (went down) after our names were revealed with NO added feedback. That is unacceptable.
That's not necessarily a problem with the DBR mechanism, but was a problem with this POPL PC. |
| 23 | Oct 21, 2011 4:09 PM | Keep it for next year. |
| 24 | Oct 21, 2011 3:58 PM | Is there a very strong case for DBR? If not, drop it. Or, be radical and go for NBR - no blind review, i.e., signed reviews! :-) |
| 25 | Oct 21, 2011 3:55 PM | From my limited experiences, the biggest problem I've ever faced was that the double-blinding was just not working. I wonder if it helps to ask reviewers to take a guess on who the authors are, along with their confidence level. Those who guess correctly with high confidence don't have to be disqualified, but their reviews should not be used as the sole basis of decision if possible.
One would immediately be alarmed by the possibility of reviewers "playing dumb" in their guesses, but it should increase reliability in cases where reviewers turn out to be benevolent. I think that's a step forward if it can be made to work. |
| 26 | Oct 21, 2011 2:35 PM | Perhaps we should qualify the reviews and drop the least useful one. As deciding the percentage for quality reviews there will be a priori. Currently there are four reviewers per paper. One of our reviews was topical, anecdotal, and last minute by a non-expert. Shouldn't there be a mechanism in place to save both the time of the reviewer and the author(s), and prevent such useless reviews? For POPL at least? |
| 27 | Oct 21, 2011 2:27 PM | Restricting the list of external referees may yield to have a non-expert ones. |
| 28 | Oct 21, 2011 2:27 PM | The DBR process improves reviews quality. It however felt that the efforts spent detailing answers for reviewers' interrogations and misunderstandings were vain. |
| 29 | Oct 21, 2011 6:42 AM | I think double-blind reviewer does not improve the quality of the selection
process much because well-known authors (for whom the reviewer might have an
a priori good opinion or would spend more time on his review) most likely be
recognized because their research areas or preliminary versions of their
work are known. Hence, the unrecognized authors will most likely remain the
younger researchers.
On the other hand double blind reviews require a (slight) extra-work for
authors and a lot of extra-works for the program committee especially in
selecting sub-reviewers so that altogether the process is not worth. |
| 30 | Oct 20, 2011 3:28 AM | In my case, I think that it was easy to guess who was the authors (or at least part of them) since we are developing our own programming language. Therefore DBR only introduces drawbacks since we cannot give access to the compiler and to the examples.
Nevertheless the way DBR was quite reasonable, in particular due to the sentence: "Nothing should be done in the name of anonymity that weakens the submission or makes the job of reviewing the paper more difficult". |
| 31 | Oct 19, 2011 6:24 AM | I was really happy with the DBR-style review process. |
| 32 | Oct 18, 2011 11:11 AM | I don't have a strong feeling either way, and as an author I did not feel it made much of a difference.
One thing that I would like to change if DBR is to be kept:
Please allow for anonymous submission of appendixes, source code and the like, so that they is available to the reviewers when they first read the paper. The more theoretical papers can often not be reviewed properly without the supplemental material.
Important statistics I would like to see are:
1. How frequently PC/ERC members changed their scores after the author identities were revealed? If this happens too often (i.e., more often than in past SBR conferences), this might indicate that the purported objectiveness due to anonymization is lost. I suspect this was largely not the case, but it would be nice to have some data.
2. How often did it happen that PC/ERC members would have bid to review paper X because they were experts on the topic, but did not recognize that the paper was relevant, because it happened to have a strange title and the author names were hidden. |
| 33 | Oct 18, 2011 8:17 AM | Having an ERC seems like a very good idea, regardless of whether the reviews are single or double-blind. You should continue to use one. |
| 34 | Oct 17, 2011 9:52 PM | DBR is better than nothing, but far from improving the quality of many of the very strange reviews I (and my colleagues) have been seeing, both as authors and PC members including POPL's. I wholeheartedly request that we switch to the PVLDB "journal model" for the health of our community. |
| 35 | Oct 17, 2011 5:42 PM | This is a good direction in improving the quality of accepted papers, but it remains to be seen if it has an effect. |
| 36 | Oct 17, 2011 5:23 PM | no-blind reviewing is without doubt the only sensible form of reviewing. Anything else makes it easy for referees to do a poor or dishonest job. For POPL, this is perhaps less of an issue compared with other conferences since the standard of reviewing is generally high. |
| 37 | Oct 17, 2011 4:07 PM | I don't think you'll get any objective kind of feedback out of this kind of survey.
The process was well thought out and executed. It was lightweight for the authors.
The jury is still out, though, on whether DBR or SBR gives less biased / more objective results - if that's what is desired. |
| 38 | Oct 17, 2011 3:13 PM | My experience was very positive. The referees' comments where very helpful and to the point. Conference chair was very helpful as well. The process was as smooth and fair as it could be. |
| 39 | Oct 17, 2011 1:42 PM | To me, DBR-style review seems to work without major hindrance. Reviewers should not know at first which are the authors of the paper they will review. |
| 40 | Oct 17, 2011 12:51 PM | It was alright,but I did not think it was necessary. |
| 41 | Oct 17, 2011 12:07 PM | The paper was based very closely on our previous work; the formalism would have been immediately recognizable to anyone who had read the previous work. So I'm not sure whether the anonymization worked. |
| 42 | Oct 17, 2011 11:08 AM | The next step is to assign scores to the reviews and to have them edited. |
| 43 | Oct 17, 2011 10:41 AM | Not about DBR, but about the reviewing process in general:
(1) The quality of the reviews was overall extremely high. Thanks You!
(2) The response/reaction to the rebuttal (or better lack thereof) was extremely disappointing. |
| 44 | Oct 17, 2011 10:00 AM | Conferences are not a game to be played, they are a representation of the work that people in the field are doing. Authorship matters. |
| 45 | Oct 17, 2011 9:54 AM | DBR is great! It should also be blinded during PC meeting! |
| 46 | Oct 17, 2011 9:50 AM | I think all criticism I have ever heard about DBR was exaggerated. It really isn't that hard to send an anonymous submission.
On the other hand, I also think the benefits are greatly exaggerated; several times I had the occasion to look anonymous submissions in my field (requests for comments, subreviews, etc.) and each time I was able to accurately guess the authors.
Bottom line is, I'm happy to let anyone who is more opinionated fight for they point of view, I just don't think it's an issue that affects our work as researchers much. |
| 47 | Oct 17, 2011 9:48 AM | 1. Reviewers are overloaded and do not carefully read the papers. The size of the committee should be increased. I heard PC members got ~25 papers to review. If that's true, it's unreasonable to expect anyone to review 25 POPL submissions carefully and thoroughly. Perhaps the steering committee should put a limit (say, to 10) papers?
2. My earlier suggestion about making reviews reviewer-private and author-public would improve the chances that reviewers really change their reviews as the result of the rebuttal. In my limited experience, I know of only ONE case where rebuttal changed anything (something observable by the authors). IMO, reviewer-private reviews would improve the efficacy of the rebuttal. |
| 48 | Oct 17, 2011 9:24 AM | Stop DBR.
Reviewing papers only by PC and ERC can introduce more biases based on topics, even if slightly less biases based on authors. It makes the POPL look
like a closed community, of narrow interests. |
| 49 | Oct 17, 2011 9:04 AM | I'm not sure if it's practical, but it might be good to maintain the blinding through the PC meeting. |
| 50 | Oct 17, 2011 8:59 AM | DBR is an improvement on SBR, but it is time to question the current implementation of the SBR process in the light of the incentives structure it provides to reviewers, and as a consequence, authors. The requirements on review quality should be nearly as high as those on submission quality. At the moment, there is no feedback loop that prevents bad reviews. This in turn forces an author to spend time and space on marketing a submission to safeguard against bad or lazy reviewers.
In the current system, there is no accountability for reviewers and hence no incentive to provide good feedback.
Everybody knows that our conference system needs an overhaul. Nobody wants to be the first to do it.
Here's a practical small-step suggestion that should not offend anyone. Introduce an optional "For the Public" section in reviews and simple guidelines about what the supposed contents of this section. The public part of the review will be published anonymously online unless the PC decides not to do so. This will provide additional useful information for a potential reader. It will also send the message that reviews are taken seriously as academic work in their own right. |
| 51 | Oct 17, 2011 8:58 AM | Please clarify the policy for previously-released software (especially for open-sourced software). |
| 52 | Oct 17, 2011 8:37 AM | I liked the DBR-style review process. It increased my confidence in the reviews, and I did not have to made adjustments to either the manuscript or personal actions. It is certainly worth it to consider this style for future editions of the conference. |
| 53 | Oct 17, 2011 8:35 AM | I like it! |
| 54 | Oct 17, 2011 8:35 AM | At least one of the reviewers made technically inaccurate statements. In particular, they said previous work was polynomial, when the authors in the paper said their work was exponential. When pointed out in the rebuttal by giving an actual quote in the paper, the reviewer stood by their comment. It was completely ridiculous. |
| 55 | Oct 17, 2011 8:33 AM | Nice implementation of DBR that succeeded in minimizing the burden of DBR for authors. I am still uneasy about the amount of effort it took the PC and esp. the PC chair to implement it. |
| 56 | Oct 17, 2011 8:21 AM | DBR is a red herring.
What we really need is a way to drastically improve the quality of reviewing.
I consider the reviews for one of my papers borderline incompetent. Specifics: a reviewer argues that a an abstract machine with complex states is easier to use than a logic of equality to reason about -- hold on to your seat -- observational equivalence. In my 25 years of working on this problem on and off, I have never seen such a judgment. The rest of the review was much worse.
The reviews for the accepted paper were equally mixed. How can someone make blanket statements such as "In five years all papers will be verified in Coq." As you may have heard by now, even the Coq paper we investigated had a flaw (in an unverified section). And that doesn't even take into account Xavier's release of a provably correct C compiler -- with a bug. See PLDI '11.
I could go on, but there's no space left in this box and no time on my clock :-)
|
| 57 | Oct 17, 2011 8:19 AM | I think that the DBR-style review process is an excellent thing in itself, and should be used and promoted in other conferences (the LICS conference will apply it this year). On the other hand, I think that it would be interesting to carry on in this direction of fair and open reviews, by allowing the authors to write an official answer to the final decision, and to give an official score to each of the reviews of their paper. Let me take my case as an example. There is little doubt that one of the reviewers seriously misbehaved, and wrote a negative review on purpose, without properly understanding my work. I was really embarrassed to complain in my answer to the intermediate reviews, because I felt that such a reaction would probably kill the submission. On the other hand, now that the paper has been rejected, I feel that there should be an official space to indicate to the PC (not necessarily the ERC) members that this specific reviewer seriously misbehaved. Since much of our research life is based on reputation, I think that it would help to prevent such misbehaviors. Accordingly, I think that it would be interesting to ask every PC/ERC reviewer to indicate whether he has (even the slightest) conflict of interest with the reviewed paper. In that way, the other PC/ERC members would be aware that the review may be tainted by this conflict of interest. |
| 58 | Oct 17, 2011 8:16 AM | Overall I think it's a good idea, but it does hurt certain types of papers -- particularly papers that are the continuation of prior research, where the author may have additional insight to convey based on that past work. |
| 59 | Oct 17, 2011 8:13 AM | My expert reviews from POPL were very thorough. Perhaps this is a result of involving the ERC more closely and also in choosing experts carefully.
I expect that "past reputation" plays a role in the PC meeting, especially in borderline cases: you are more likely to vote in favour of someone you respect than someone you don't like. For DBR to be properly effective and useful, you would need to keep it right through to the meeting.
Overall, the fact that I missed out on some interesting informal discussions means that I have an overall negative feeling about DBR. |
| 60 | Oct 17, 2011 8:11 AM | Maybe the rebuttal should be anonymous too.
I also don't like that reviewers remain anonymous, or at least I see some tensions with that. But that's another story. |
| 61 | Oct 17, 2011 8:06 AM | The DBR requirements impaired our submission, cost significant thought in the late stages of paper-writing, and were pointless - as reviewers (at least, the expert reviewer) was in no doubt who the paper was by. |
| 62 | Oct 17, 2011 8:01 AM | If DBR is to be used again in the future, I would like to see some tangible evidence that it has an effect on anything. If not, then why bother with all the complication? |
| 63 | Oct 17, 2011 7:59 AM | I have comments on authors response. Is there any statistics about how many grades are improved after authors response? After author's response, there is no further response from reviewers, no change of grade, no change of their original comments. So I wonder whether it is useful for authors to response, or such thing just happened to few people. |
| 64 | Oct 17, 2011 7:56 AM | Besides the issue with the experimental setup that makes it
easy to identify a group,
the concepts described in our paper are present in open-source
software repositories, as names of classes/packages etc.
DBR makes sense only if the blindness can be assured.
The authorship is revealed after the reviews are submitted.
It should be kept double-blind throughout, if at all. |
| 65 | Oct 17, 2011 7:50 AM | How about trying the other extreme: having no anonymity at all, for a completely open process where reviewers take responsibility for their reviews? (Please note that I am not complaining about the reviews for my paper. I would have preferred them to be more detailed, but the rejection was well documented.) |
| 66 | Oct 17, 2011 7:47 AM | My publications are theoretical/Mathematical, but I think POPL is fast becoming a conference that is less and less relevant.. POPL commitee responses were so poor I did not even bother to send a response to the reviews. I seriously doubt I will send papers again, and I was hesitant even before. |