| 1 | Oct 13, 2011 5:11 PM | Bit of a false dichotomy there. If tiered reviewing were to become common within SIGPLAN, I would be disappointed. The amount of work required to review a clearly bad paper is negligible. I don't see that tiered reviewing solves any problem for us. |
| 2 | Oct 6, 2011 8:25 PM | Especially, given the papers with weak scores that got in as someone was excited. |
| 3 | Oct 6, 2011 5:43 PM | That's a weak no; I'm not sure. |
| 4 | Oct 6, 2011 4:14 PM | Trying to obtain the minimal number of reviews is a good idea. |
| 5 | Oct 6, 2011 12:41 PM | Really bad papers typically get really short reviews - thus not that much work is actually wasted |
| 6 | Oct 6, 2011 12:01 PM | A "clearly bad paper" is often less work for the reviewers. |
| 7 | Oct 6, 2011 9:36 AM | I think the responses here are leading, i.e., weighted toward the survey author's point of view. To be less leading, "wasted" could be replaced by "additional", and so on. Neither of the choices reflects my point of view, so I choose neither. |
| 8 | Oct 6, 2011 8:49 AM | It would have been useful for a small subset of the papers that we could easily predict would wind up with all "1"s. |
| 9 | Oct 6, 2011 8:43 AM | I do no feel strongly about my answer to this question; either option can work. |
| 10 | Oct 6, 2011 8:36 AM | Potential injustice: experts are often buddies w/o COI. |
| 11 | Oct 6, 2011 5:46 AM | Reviewing a bad paper can take a lot of time (and a lot of agonizing), in the end for nothing because one does not learn anything. [Associate Editors for TOPLAS] have the system of Quick Reviews. It works and it is quite effective. |
| 12 | Oct 6, 2011 5:35 AM | On the other hand, I think it could be very useful to have access to previous reviewers' factual commentaries about the paper -- mistakes in proofs, confusing typos, etc. I think these would improve the quality of reviews and reduce wasted effort, without introducing bias. |
| 13 | Oct 6, 2011 4:58 AM | I think no POPL submission should be reviewed less than 3-4 times, but tiering could be used to adaptively channel effort to controversial papers. |
| 14 | Oct 5, 2011 7:29 PM | An unfairly worded question since my 'no' on two-stage reviewing is more that as PC/ERC member I prefer the scheduling flexibility of having a longer time before one deadline rather than two separate deadlines where I can't do my second batch until later. Being on a PC is a ton of work, so I'd prefer all the scheduling flexibility I can get. |
| 15 | Oct 5, 2011 6:11 PM | With tiering we have to make sure the first 1-2 reviews are REALLY from experts. This is difficult in a broad venue like POPL and I worry that many good papers will draw the short straw and get rejected erroneously. The current setup (2012) worked great. No more tiering please. |
| 16 | Oct 5, 2011 4:14 PM | On the bad papers (those with 3 bad reviews), asking for a 4th one was probably a waste of time. |
| 17 | Oct 5, 2011 3:41 PM | I think each paper should receive at least 3 reviews. Beyond that, tiering makes sense. |
| 18 | Oct 5, 2011 3:31 PM | Giving papers the chance to have a champion is important. There is value to the authors in the feedback given, even for bad papers. Perhaps alleviate the excess work by encouraging reviewers to spend little time on seemingly bad papers, revisiting later if needed. |
| 19 | Oct 5, 2011 2:40 PM | Three reviews by default might be enough... |
| 20 | Oct 5, 2011 2:12 PM | Please see my previous comment. |
| 21 | Oct 5, 2011 1:57 PM | The 1-4 scale may not have enough information in it to reject papers with few reviews. |
| 22 | Oct 5, 2011 1:54 PM | Tiered reviewing is an excellent way to reduce reviewer load and focus most intensive reviewer efforts on papers above the bar. |