| 1. Were you on the PC or the ERC? | |||
|---|---|---|---|
| answered question | 75 | ||
| skipped question | 0 | ||
| Response Percent | Response Count | ||
| PC |  | 33.3% | 25 |
| ERC | | 66.7% | 50 |
| 1. Were you on the PC or the ERC? | |||
|---|---|---|---|
| answered question | 75 | ||
| skipped question | 0 | ||
| Response Percent | Response Count | ||
| PC | | 33.3% | 25 |
| ERC | | 66.7% | 50 |
| 2. Please indicate your current opinion about which choice (among only these two) is best: single-blind reviewing (SBR) as is typically employed by POPL, or double-blind reviewing (DBR) as we implemented it this year, which involved revealing authorship after first review, and using guardians to help find expert reviews midway through the review process. Your answer should reflect your perception of the best choice on balance, based on which process you think is overall the most fair, most accurate, most enjoyable, etc. (You will have an opportunity to rate particular aspects of the review process later.) | |||
|---|---|---|---|
| answered question | 67 | ||
| skipped question | 8 | ||
| Response Percent | Response Count | ||
| Traditional single-blind reviewing (SBR) | | 29.9% | 20 |
| Double-blind reviewing (DBR) as per POPL this year | | 70.1% | 47 |
| Optionally provide reasons or qualifications for your choiceview | 31 | ||
| 3. How has your opinion about DBR changed as a result of your experience with POPL'12? | |||
|---|---|---|---|
| answered question | 64 | ||
| skipped question | 11 | ||
| Response Percent | Response Count | ||
| My opinion of it improved a lot | | 23.4% | 15 |
| My opinion of it improved a little | | 60.9% | 39 |
| I think less of it than before | | 10.9% | 7 |
| I think much less of it than before | | 4.7% | 3 |
| Optionally expand on your answerview | 20 | ||
| 4. Do you have any suggestions for how we might have handled DBR-style reviewing better? | ||
|---|---|---|
| answered question | 26 | |
| skipped question | 49 | |
| Response Count | ||
| view | 26 | |
| 5. Papers were assigned a "guardian" whose job was to review the paper early, and then help find an expert reviewer if the guardian himself was not expert. Overall, do you recommend this idea? | |||
|---|---|---|---|
| answered question | 62 | ||
| skipped question | 13 | ||
| Response Percent | Response Count | ||
| I would recommend it for both SBR and DBR processes | | 87.1% | 54 |
| I would recommend it only for DBR; for SBR there's no need for it | | 6.5% | 4 |
| I would not recommend it | | 6.5% | 4 |
| Comment if you wishview | 20 | ||
| 6. In a sense, guardians imposed a kind of tiered reviewing: some reviews are completed during the first phase, and the result helps determines whether further reviews are needed with the second stage. Do you think that tiering should have been used more generally, e.g., to avoid reviewing a clearly bad paper as many as four times? | |||
|---|---|---|---|
| answered question | 64 | ||
| skipped question | 11 | ||
| Response Percent | Response Count | ||
| Yes, I think tiered reviewing of some sort could have worked | | 48.4% | 31 |
| No, I think the wasted work is negligible next to the potential injustice | | 51.6% | 33 |
| Comment further if you wishview | 22 | ||
| 7. We decided to make supplemental material only available after you submitted your review, to emphasize to the authors that the submitted paper is what we are reviewing, and to avoid the need to anonymize this material. How do you feel about this policy now? | |||
|---|---|---|---|
| answered question | 61 | ||
| skipped question | 14 | ||
| Response Percent | Response Count | ||
| It's the best choice for DBR | | 27.9% | 17 |
| It's the best choice whether using DBR or SBR | | 27.9% | 17 |
| A better choice would be to make supplemental material available during the first review (anonymizing in a DBR setting) | | 16.4% | 10 |
| The best choice for DBR would be to allow both anonymous and non-anonymous material to be submitted | | 9.8% | 6 |
| We shouldn't allow supplemental material | | 18.0% | 11 |
| Optionally provide reasons or qualifications for your choiceview | 15 | ||
| 8. On balance, are you in favor or against the ERC mechanism (detailed questions below)? | |||
|---|---|---|---|
| answered question | 63 | ||
| skipped question | 12 | ||
| Response Percent | Response Count | ||
| In favor | | 88.9% | 56 |
| Against | | 11.1% | 7 |
| 9. PC submissions present a potential conflict of interest when other PC members can review them. Given your past experience and your experience from this year, how should we handle PC submissions? | |||
|---|---|---|---|
| answered question | 62 | ||
| skipped question | 13 | ||
| Response Percent | Response Count | ||
| Use an ERC, as this year | | 64.5% | 40 |
| Use ad hoc external reviewers | | 3.2% | 2 |
| Allow PC members to review PC papers, but hold to a higher standard | | 12.9% | 8 |
| Do not allow PC submissions | | 9.7% | 6 |
| Other | | 9.7% | 6 |
| Comment on your answerview | 24 | ||
| 10. External expert reviews may be needed for papers outside the PC's expertise. How should we handle finding external expert reviews? | |||
|---|---|---|---|
| answered question | 62 | ||
| skipped question | 13 | ||
| Response Percent | Response Count | ||
| Use an ERC only | | 1.6% | 1 |
| Use an ERC with occasional outside solicitations (as this year) | | 82.3% | 51 |
| Use only ad hoc solicitations (as with POPL last year) | | 14.5% | 9 |
| Some other way | | 1.6% | 1 |
| Feel free to expand on your answerview | 15 | ||
| 11. Do you have any other thoughts or suggestions about the POPL'12 review process? | ||
|---|---|---|
| answered question | 32 | |
| skipped question | 43 | |
| Response Count | ||
| view | 32 | |