| 1 | Oct 13, 2011 5:19 PM | I'm glad it's over :-)
More seriously, I'd like to see the reviewing form refined a bit. I'd like better language on some of the choices---for example, "accept and will advocate" and "accept but will not advocate". I also think it would help to explain to the reviewers how the multiple-choice questions are intended to be used.
I would also urge future program chairs to think carefully about what 'scores' they wish to expose to authors. In my view, scores should be seen as an aid to the program committee in making decisions, and not as a meaningful product of the review process. In this view, no good can come of sharing scores with authors.
If the scores are kept private, then a significant amount of busy work (adjusting scores after the PC meeting) can be avoided. And if the scores are private, authors can't complain about them :-) |
| 2 | Oct 10, 2011 6:31 AM | Mike did an excellent job. |
| 3 | Oct 6, 2011 8:29 PM | Physical PC meetings are great but too expensive (global warming, cost, time). We should stop this and find a better alternative (video conferences?). |
| 4 | Oct 6, 2011 5:49 PM | I have never liked the 1-4 overall rating scheme. I'd prefer finer distinctions. As I mentioned earlier, I strongly dislike rejecting papers based solely on the absence of strong ACCEPT ratings. Evaluations of papers change too much as a result of discussion. |
| 5 | Oct 6, 2011 4:36 PM | I thought it was well done. But even as an ERC member, a lot of work. There will be a natural tendency for the ERC to be roughly the same from year to year, which is problematic on grounds of workload. |
| 6 | Oct 6, 2011 12:05 PM | The selection process seems good enough, given the supply/demand constraints. However, POPL may want to think about separating selection from dissemination, as argued here:
http://www-verimag.imag.fr/~tripakis/papers/lets.pdf |
| 7 | Oct 6, 2011 11:13 AM | Mike, once the process starts, don't ask the PC/ERC to give input and vote on procedural matters. The PC chair should make the decisions. When in doubt, stick with the a priori agreed upon process. |
| 8 | Oct 6, 2011 10:48 AM | Overall, it was very pleasant experience! |
| 9 | Oct 6, 2011 9:46 AM | I felt that the process was very burdensome, and regulation-filled, especially compared to other PCs I've been on. I would like to see some evidence that all these machinations achieved something before I'd be willing to advocate for such a heavyweight PC process for future PCs. I'm not sure how the kind of comparison of outcomes I seek could be possible, though. |
| 10 | Oct 6, 2011 9:06 AM | Overall, it was very well done, including the PC meeting discussions. |
| 11 | Oct 6, 2011 8:53 AM | From my vantage point, the process ran really smoothly. I was very impressed with how much attention you paid to so many papers, not to mention all the other administrative issues. Thanks for all the hard work! |
| 12 | Oct 6, 2011 6:04 AM | The fact that Michael was constantly in our neck made us work harder and be very serious and conscious about our reviews ("if the boss is serious, ...").
Great PC experience! Thank you, Michael! |
| 13 | Oct 6, 2011 5:01 AM | (Comment redacted.) But I imagine this is irrelevant to this survey. |
| 14 | Oct 6, 2011 2:18 AM | It was very enjoyable, especially under such professional guidance! |
| 15 | Oct 5, 2011 11:58 PM | Great job overall! I enjoyed it and think the results were very good. |
| 16 | Oct 5, 2011 11:06 PM | Here's a suggestion: have the discussion lead for each paper write a summary of the PC discussion, coordinate with the other reviewers to make sure it's accurate, and post it as a comment. Kinda like they do at NSF. I think that having the discussion lead have this responsibility will make it sure it gets done. And I think it should be done for all papers, even ones that are good. |
| 17 | Oct 5, 2011 10:45 PM | no |
| 18 | Oct 5, 2011 9:48 PM | I'm in favor of even stronger blinding: reveal the authors to reviewers only if the paper is accepted. (And resolve conflicts lazily.) |
| 19 | Oct 5, 2011 9:03 PM | Mike, you took it very seriously, and implemented significant improvements. Thank you for your efforts! |
| 20 | Oct 5, 2011 7:33 PM | It's gotten too complicated: getting committed people to do timely responsible reviews is sufficient and necessary -- all the meta-discussions are healthy to a point and then a distraction. |
| 21 | Oct 5, 2011 6:16 PM | Nope -- great job Mike! |
| 22 | Oct 5, 2011 4:57 PM | First, a *big* thanks to Mike for tirelessly working on making the process as fair as possible.
While I cannot say this objectively, I do feel that many good papers were rejected, some
even with (many) high scores (rest of comment redacted).
If I recall correctly, Mike emailed at some point that he could accept upto 49 papers, but
only 44 were accepted. Cursorily reading discussions of papers that were ranked highly
initially but ultimately got rejected, it seems like these papers suffered from an unusually
high degree of scrutiny. I heard from someone on the POPL'11 PC that the then chair,
Mooly Sagiv, gave a speech before the PC meeting started on the virtues of being positive,
and that the speech had the intended effect on the PC members of not being overly critical.
After all, almost any paper can be killed by finding some shortcoming or the other. Overly
critical reviews not only frustrate and drive even good researchers out of the field, but also
make the entire community overly critical over time (many reason this way: why should I
accept this paper which appears inferior in quality to my paper which was rejected?). Please
promote and propagate the efforts of people like Mooly Sagiv, Thomas Ball, Alex Aiken,
Martin Rinard, etc. on fostering more acceptance in our community.
Other than this complaint, once again a big thanks to Mike for a job well done! |
| 23 | Oct 5, 2011 4:18 PM | Revealing the authors of all the reviews to all members of the ERC might not be the best policy (the ERC includes a large fraction of our community). |
| 24 | Oct 5, 2011 3:53 PM | Overall, this year's review process is the best I have seen in recent years. The only improvement I can suggest is to do better *reviewer assignment* for each paper.
One way to do this is that right after the popl deadline, the PC chair can divide all the submissions into smaller clusters based on areas and let PC members and/or ERC members suggest a list of qualified expert reviewers. This information is then gathered and passed to the PC chair who can then filter out the conflicts and feed into whatever algorithm the PC chair is using. In some sense, PC members are playing the role of associate editors for a traditional journal. |
| 25 | Oct 5, 2011 3:39 PM | The numeric scores other than overall and expertise were irrelevant. Providing authors with the numeric scores with the reviews for author response seems to distract authors and complicate later changes of position. |
| 26 | Oct 5, 2011 3:17 PM | Many, but I would prefer to discuss them privately with the PC chair. |
| 27 | Oct 5, 2011 2:44 PM | The software (HotCRP) was quite good. |
| 28 | Oct 5, 2011 2:00 PM | Overall it went well. |
| 29 | Oct 5, 2011 1:56 PM | Mike did a great job. |
| 30 | Oct 5, 2011 1:54 PM | The ERC did not work very well at all. Everything else worked excellent. |
| 31 | Oct 5, 2011 1:54 PM | I think the guardian process worked very well and that the quality of reviews was very high. Discussions happened well before the PC meeting, as a result the PC meeting ran very smoothly.
I think it would be good to implement some process to allow authors to give feedback on the quality of reviews. |
| 32 | Oct 5, 2011 1:53 PM | My only grouse with the review process was the many options on the review form. Including novelty, importance, conviction and clarity is overkill. Overall merit + reviewer expertise should suffice. The rest of the argument for or against a paper is more nuanced and should be made by the main content of the review, not by these scores. |