CFP Reviewer best practices
This document is intended for CFP reviewers at PyCon India.
CFP reviewers review the talks submitted at PyCon India and cast their votes. The final list of selected talks is based on these votes.
CFP reviewers, being experts in their domains, should evaluate talk proposals based on technical depth. They should also consider presentability, i.e., whether the proposal suits a conference talk.
The review process comprises three phases:
- Conducted by: the volunteers of the CFP workgroup.
- Aim: Complete and detailed proposals. Volunteers work with the authors to make sure proposals are complete in all respects.
- Finishes when: The proposal has sufficient details, outline, and slides.
- Conducted by: the CFP reviewers (who are technical experts in specific domains)
- Aim: Review the proposals on technical merits, and vote on them
- Finishes when: the CFP reviewer votes on the proposal
- Conducted by: the volunteers of the CFP workgroup
- Aim: Make sure talks are ready for the stage
- Finishes when: The speaker presents the talk before the workgroup
- Volunteers will set up reviewer accounts. These accounts are authorized for specific categories alone (See The categories section below)
- To see all the proposals that you can review, head to this page. Volunteers may also suggest proposals that are ready for review.
- Prefer reviewing proposals that are more ready first. These proposals contain sufficient details, including the slides too. Please look at Phases-1 in The entire review process section above.
- Feel free to ask for details/clarifications by using comments. See The Comments section below.
- Proceed to vote when satisfied that the content is sufficient. There are six voting levels (from -2 to 3), each weighing differently.
- Please add a meaningful comment while voting. In case of a tie, we'll use these comments to compare.
- Please avoid using the most negative vote (-2) too often. It should be used only for talks that may damage the conference.
There are three kinds of comments - the Comments, the Reviews, and the Reviewers Talk.
- Comments: can be seen by the author, other reviewers, and logged-in users. An email is also sent to the author.
- Reviews: can be seen by the author and other reviewers. And an email is also sent to the author).
- Reviewers Talk is not seen by authors and can be used to leave notes for other reviewers.
The best is to use Reviews. Please feel free to ask/suggest the proposer if you have any questions/suggestions. These comments are in Markdown format supported. Please make sure to write accordingly.
- Reviewers are assigned to specific categories (like
Core-Python,Data Science, AI/ML, etc.) and are therefore authorized to review only the talks in their categories. - The corresponding name can be clicked on the listing page right below the talk titles to see all the talks in a specific category.
Examples: Core-Python, Data Science, AI/ML
- You can vote after you are fully satisfied with the proposal review.
- Try to avoid as much bias and comparison with other proposals as possible. You should score a proposal on its merit.
- You should use your best judgment to score from -2 to 3.
- After giving a score, you need to comment on the proposal on the following guidelines for every proposal you review:
- Why do I think hundreds of people should spend 30 minutes watching this Talk?
- Why do I think the proposal should not be selected?
- In a few talks, you may need help to add good points or, in other cases, any bad points. That's ok. Please feel free to leave it blank.
The reviewers' names will be kept secret, especially from the authors. This is to make sure there is no undue influence on the reviewers. The reviewers will be thanked and felicitated after the CFP process completes. Please don't reveal your identity to the authors.
Reviewers are not supposed to be submitting talk proposals themselves. They should also keep themselves free from conflicts of interest - by not reviewing talks from their friends and colleagues.
PyCon India's code of conduct applies to the reviewers.
Contact the CFP team at [email protected], or on Zulip chat
Remember how you will Review will affect thousands of attendees and their valuable two days. Please be sure to use this responsibility wisely. -- @zerothabhishek, PyCon India 2019 CFP lead
