| 1 | Oct 13, 2011 5:15 PM | I really dislike the ERC mechanism, for a variety of reasons. But in the current political climate, it is by far the best solution to the problem of PC submissions. (And in my opinion, solving this problem is the *only* way to justify "blessing" a picked set of external reviewers as an ERC.) |
| 2 | Oct 6, 2011 5:46 PM | I didn't really experience that this year. My past experience with using the ERC is that I wasn't confident that acceptance standards were consistent; I worry about accidentally applying lower standards to PC submissions. Using the ERC would probably still be my second choice. |
| 3 | Oct 6, 2011 4:15 PM | For a physical PC meeting, I would rather disallow PC submissions, because otherwise it is awkward at the meeting. For a virtual PC meeting it is not a problem.
Using an ERC is not a good idea. It means that PC papers are subject to a totally different process to any other submissions. That's not fair. For instance: they are not subject to the kinds of analysis that go on at the PC meeting; ERC members cannot collectively discuss PC papers (because of anonymity, we could not see the discussions) so the decision is made by three or four people rather than the whole committee. As I said, PC papers are subject to a totally different process to any other submissions. Moreover the "Programme Committee" is not entirely deciding the programme, which is strange.. |
| 4 | Oct 6, 2011 12:45 PM | It's complicated :-) I don't think POPL should disallow PC papers, but ERC members may have similar conflicts to PC members so having PC papers being reviewed by ERC members is probably not an ultimate solution to the problem. At the same time, I don't think this problem is that big - so using a combination of all three methods is probably OK. |
| 5 | Oct 6, 2011 11:12 AM | Have a phone conference to resolve the few contentious papers (remainder redacted). |
| 6 | Oct 6, 2011 10:47 AM | I think we should consider discussing borderline PC papers at the PC meeting (with any conflicts leaving the room). The ERC should still recommend accept or reject but leave the final decision to the PC to calibrate across submissions. To make this work, the discussion lead ERC member could call into the meeting to present the ERC discussion and conclusion. Each ERC member when accepting the position would essentially agree to being able to call in perhaps during the first morning of the PC meeting. |
| 7 | Oct 6, 2011 9:44 AM | I disagree with the premise of this question. Why is a PC member reviewing a PC submission different from anyone else reviewing it? Someone not on the PC cannot advocate for or against a PC submission directly, but the discussion should be based on the PC member's review, not on the additional arguments they may make on the fly. |
| 8 | Oct 6, 2011 9:04 AM | I don't think it is good to prohibit PC submissions -- they should represent your core audience.
Ad-hoc reviewers may have a harder time with calibration.
Holding PC to a higher standard is difficult to ensure. |
| 9 | Oct 6, 2011 8:45 AM | Any solution except forbidding PC submissions is ok. |
| 10 | Oct 6, 2011 5:55 AM | I prefer to not sit across the fellow PC member whose paper I just rejected.
The ERC members should hold to a (much) high higher standard.
One more thing: The decision about acceptance of PC submissions must be fixed before the PC meeting. I know of one case where a PC member was extremely pushing rejections, and, as I only later learned, the PC member had two submissions. |
| 11 | Oct 6, 2011 5:47 AM | As an ERC member reviewing a couple PC papers, I felt we did not have sufficient perspective on the acceptance standard to make an accept/reject decision. I suppose this was exacerbated by the fact that there has been a push to accept more papers at POPL, and the overall standards are less clear. |
| 12 | Oct 6, 2011 4:59 AM | It would have helped me to know the policy for PC submissions (i.e., that the reviews were all ERCs *and* the papers would be accepted or rejected separately, nit at the PC meeting). |
| 13 | Oct 6, 2011 2:17 AM | This is hard, not sure what is best here really. |
| 14 | Oct 5, 2011 11:58 PM | Not having the PC have to deal with PC papers did simplify some things, but it's not clear that it's worth the overhead of introducing an ERC. |
| 15 | Oct 5, 2011 9:46 PM | I was on the PC and had a paper rejected. I much preferred to have it rejected by the ERC than at the PC meeting. |
| 16 | Oct 5, 2011 6:16 PM | The PC should be purely electronic, in which case these
issues don't arise (i.e. you can safely have PC submissions
without worrying about conflicts.)
I've heard the arguments for physical meetings (mainly
many changes over the raw score based acceptance) but its
not clear that
(a) these changes would not happen over a well reasoned online discussion
that does not suffer from the arbitrary timing constraints imposed by a physical
meeting,
(b) the changes actually make the program _better_,
Unless one can demonstrate the result is strictly (and vastly) superior, I don't see the justification for the travel and carbon costs. |
| 17 | Oct 5, 2011 3:36 PM | Calibrating PC papers with others is important. It is not entirely clear to me that PC reviewers of PC papers presents a higher conflict of interest than PC members with submissions reviewing other papers. |
| 18 | Oct 5, 2011 3:18 PM | I do not have a firm opinion on this question. Using an ERC, or some other external reviewers, or PC members to review PC submissions are all fine -- what matters is to find expert reviewers, whereever they are (the ERC is a way to cheaply expand the pool of expertise available to the PC, so it is a good idea). |
| 19 | Oct 5, 2011 3:16 PM | Why not treat PC submissions the same as any other? I fail to see any reason for treating PC papers any different than any other. It's just as awkward to review a paper by a PC member as it is to review a paper by a close friend of a PC member. Where do we draw the line? |
| 20 | Oct 5, 2011 2:44 PM | Any answer other than "allow PC members to review PC papers" seems OK with me. |
| 21 | Oct 5, 2011 1:59 PM | Use PC reviewers and mostly the same standard as for other papers. But identify them as PC papers in discussion. |
| 22 | Oct 5, 2011 1:57 PM | also not allowing PC submissions seems good |
| 23 | Oct 5, 2011 1:56 PM | PC members reviewing PC papers are also fine. |
| 24 | Oct 5, 2011 1:53 PM | Allow PC members and ERC members to review PC papers, and hold them to the same standard as all other papers. |