Change the future

PyCon Program Commitee Guide

This guide details how the PyCon Program Committee operates. It's designed to be a working document for the PC itself to follow, and also for prospective PC members to get an idea of what being on the PC entails.

This is a living document; it describes the PyCon process as it is "right now"; it'll be updated continuously as our process grows and changes.

The program committee's job is to craft the best possible program for PyCon out of the pool of proposals submitted. It's a tough job: PyCon 2012 received roughly 400 proposals for only 95 slots. The committee tries to go beyond simple review: committee members interact with proposal authors and try to help them put forward the best possible proposals. Often the final form a talk takes is shaped by feedback, comments, and criticism from the committee.

Joining the committee

Committee membership is open to anyone - more voices, more diversity of viewpoints, make for a better conference. To join the program committee, sign up for the pycon-pc mailing list. We use the mailing list to coordinate all the rest of our activities.

Committee membership is a time commitment. Expect to spend something approaching 4-8 hours per week, August through December, with the bulk of that work coming towards the end of November as the final selections are made. The bulk of the work involves reviewing proposals on the PyCon website (see initial review, below) and then later participating in IRC meetings (Kittendome and Thunderdome, below).

Committee structure, consensus, and voting

The committee's led by a chair and co-chair. In 2013, the chair is Jacob Kaplan-Moss; the co-chair is Luke Sneeringer.

Whenever possible we strive to operate in a way drawn from the open source process that's worked so well for Python. That means that we'll try to find rough consensus where we can. This doesn't scale for review itself, so during the review process, we'll use voting and majority rule to select the bulk of the talks.

Ultimately, the responsibility for the program falls on the chair, so the chair has the final say. The chair is the Benevolent Dictator for the PyCon Program, if you will. However, the chair will try to use chair fiat as little as possible.

The review process

After someone submits a talk proposal, it goes through three rounds of review:

  1. Initial review takes place on the website. The committee reviews each proposal, leaves feedback for the author, and votes on the proposal using an "identify the champion" system.

  2. Once the call for proposals closes, IRC review begins. The first round, nicknamed "Kittendome", reviews each talk in isolation, determining if the proposal would be a "good talk" for PyCon. Some talks will be rejected at this point; the rest will pass on to…

  3. The second round, nicknamed "Thunderdome". The remaining talks (typically about 1/2 - 2/3rds of proposals) will be grouped by topic into groups of 3-5. During Thunderdome, we'll discuss one group at a time, and select the best talk from each group.

When Thunderdome completes, we'll have the list of selected talks. Authors will be notified of acceptance or rejection, the selected talks will be published, and the conference schedule will be written.

2013 Timeline

This is a rough idea of the schedule we're looking at:

  • Website reviews: now - Oct 8 (CFP closes Sep 28).
  • Kittendome meetings: Oct 8 - Nov 11
  • Thunderdome meetings: Nov 12 - Nov 30
  • Talk list published: Dec 1
  • Schedule published: Dec 15

Initial review

Once talks start to roll in, we'll review them using tools on the website. This happens concurrently with the CfP - we don't wait for the call to close to start this work. The committee has two goals here:

  1. Give feedback to authors and help them improve their proposals (authors can edit proposals up until the CfP closes).

  2. Get an initial set of votes from as many committee members as possible. This will inform the later meetings.

We try to make sure that each talk gets reviewed by at least 3 committee members (and ideally more).

Voting

The process we use to assess proposals is based on the Identify the Champion process. The idea is to to assess each talk and identify "champions" and/or "anti-champions" -- people who will argue strongly for or strongly against a talk in initial review.

The document linked above uses an "A/B/C/D" scale for reviewing; we use the same idea, but use "+1/+0/-0/-1" labels (after the Apache voting system). These votes mean:

  • +1 -- I will champion the talk at the program committee meeting. The topic sounds good and the proposal is solid. This vote means you're willing to put your reputation on the line in favor of the proposal and its author, and that you believe PyCon would be worse off for not having the talk.

  • +0 -- This topic is good and the proposal is solid. The speaker is capable of correcting any deficiencies, and I think a significant number of PyCon attendees would want it available. I don't feel strongly enough about it to serve as its champion at the program committee meeting.

  • -0 -- I am not in favor of this talk as proposed. The talk, the topic, or the proposal has problems that concern me. I'd rather see it not accepted.

  • -1 -- This talk or its proposal has significant problems. I believe that PyCon would be worse off for having it, so I will argue strongly to reject this talk.

If you don't know enough about a topic to judge a proposal's coverage of it, or it's a topic you tend to actively avoid, you don't have to recuse yourself from voting on that basis. You can still judge the structure of the proposal, the likelihood that the speaker can cover the material proposed in the time allotted, whether the topic is likely to appear to others, etc.

Criteria for evaluating talks

It's hard to come up with an objective set of criteria for evaluating a talk. Ultimately, scoring talks is a subjective endeavor; we're all trying to select what we think are the "best" talks for the conference. We don't expect everyone to use the same set of criteria, but some valuable questions you might ask yourself as you review are:

  • Would I go to see this talk?
  • Are there a significant group of likely attendees that would want to see this talk?
  • Does the proposal seem well-thought-out?
  • Does it seem like the author knows enough about the topic?
  • Is the proposal too ambitious for the time allotted?

"Obvious" accepted/rejected talks

After the initial review, there will be a small set of talks that are "obviously good" or "obviously bad". We'll set a threshold, by committee consensus, for talks that are automatically rejected or get to automatically skip ahead to Thunderdome. This is usually something like "reject all talks with 3 or more -1s and no +0s or better" or "accept all talks with at least 4 +1s and no -1s".

These accepted talks aren't a free pass to PyCon - it just means that the talk goes directly to Thunderdome. A reject is final, however; it weeds out the few obviously-bad proposals that nobody on the PC wants to support.

IRC Meetings

Next, we hold meetings in IRC. (The committee chairs can help any committee members who aren't familiar with IRC.) Meetings run for an hour, and are usually scheduled a week or two in advance.

Program Committee meetings are held in IRC on Freenode. If you're not familiar with IRC, ask Jacob or Luke and we can help you get set up.

We use Doodle.com to propose and vote on times for the meetings. Expect meetings to run about an hour, maybe stretching to two if needed (although we'll try to keep those to a minimum).

We'll announce each meeting's itinerary in advance on the PC list, including the list of talks to be covered.

Kittendome

In the first round of meetings, we go through each proposal individually. The here is simple to determine if a given talk -- reviewed in isolation -- is potentially good enough for PyCon. That is, in the first round we review talks strictly on their merits and avoid holding them up against similar talks (we do that next, in Thunderdome_).

We'll review talks, one at a time. We'll give a couple minutes for the champion (identified by the website votes) to make an argument for the talk. Then, a few minutes of general debate. Finally, we'll vote yay/nay on each talk. Majority rules - a majority of "yay" votes will indicate that the talk is accepted -- it moves onto Thunderdome. A majority of "nay" votes indicates that the talk is rejected -- it's out. The chair will break ties.

Meeting format

Meetings will be in IRC; we use an IRC 'bot to keep us on track. A Kittendome meeting will look like this:

First, the bot will announce the talk is under consideration::

<pycon_bot> === Talk 456: Monitoring parrot liveliness - http://us.pycon.org/2012/review/456/ ===
<pycon_bot> (http://us.pycon.org/2012/review/123/ will be next)
<pycon_bot> If you are (a/the) champion for #456, or willing to champion it,
            please type a succinct argument for inclusion of this talk. (2 Minutes).
            Say 'done' when you are finished.

A champion — usually someone who's given the talk a "+1" on the website -- then has 2 minutes to give a succinct argument for the talk. Many champions prepare these summaries in advance (the schedule goes out a day or two prior to the meeting)::

<champion> Love the approach. Very clear what the talk will accomplish, and
           how, and in a valuable topic. done.

The bot then announces general debate::

<pycon_bot> === General Debate (3 minutes) for Talk: #456 ===

Members debate, and anything goes at this point (but please see the meeting guidlines below).

Once debate concludes, it's voting time::

<pycon_bot> === Voting time! yay/nay votes for talk #24 ===
<pycon_bot> Please do not speak after voting until we've gotten our report.
<champion> yay
<otherperson> nay
<someoneelse> yay

After a short voting period, the bot summarizes the results::

<pycon_bot> === Talk Votes on #456: 3 yays, 10 nays, 0 abstentions ===
<pycon_bot> The nays have it.

The chair records the final decision. Rinse and repeat.

Thunderdome

After round one has ended, we're left with a set of talks that are all "good enough" for PyCon. Unfortunately, this'll typically be more than double the number of talks we can actually accept, so we pit similar talks head-to-head and try to select the best.

In each "round" of Thunderdome we review 3-5 talks, roughly grouped by theme/content/topic/etc. Out of each group we'll get one winner, possibly some damaged talks, and some talks that are out. We'll do this by discussing, and then the committee will vote for the talks they like. Members may vote for any number of talks (including none or all of them).

The winner is the one talk from the group that absolutely should be at PyCon. It advances and is safe from future cuts. It's on the program! There doesn't have to be a winner from each group, but there probably should be. The votes identify the winner: it's the talk with the most votes, but only if it received a thumbs- up from at least 75% of the committee. In some cases there may be multiple winners, but this can't happen too often.

Damaged talks are ones that have been hurt by Thunderdome but aren't quite out of the running yet. They'll be put into the damaged pool and if at the end we've still got spaces left we'll fill those spaces from the damaged pool. There may not be any damaged talks in a given group, or all talks may end up damaged. Damaged talks are are any talks that didn't win, but did receive votes from at least 50% of the committee.

Talks are selected by fewer than 50% of the committee in attendance are out – they will not appear at PyCon.

The result of Thunderdome will be the ~100 talks that'll appear on the PyCon program.

Thunderdome process

Like the first round we'll use a bot to keep us on track.

The bot will announce each group::

<pycon_bot> === Thunderdome for “Birds of a feather” begins now! ===
<pycon_bot> Talk #123: Modelling flight patterns of swallows
<pycon_bot> Talk #456: Monitoring parrot liveliness
<pycon_bot> Talk #789: SciPy for aerodynamics engineers
<pycon_bot> You now have 2 minutes to review these talks and collect
    your thoughts prior to debate. Please refrain from speaking until
    debate begins.

Members will have 2 minutes to read talk descriptions, collect their thoughts, and formulate arguments; please try to stay silent during this time so everyone can think.

The bot will then announce 5 minutes of open debate::

<pycon_bot> === General debate (5 min) for “Birds of a feather” ===

Members debate, and anything goes at this point (but please see the meeting guidlines below).

Then we vote! Vote for as many talks as you like, but please keep in mind the very limited number of spots. Voting will look like this::

<pycon_bot> === Voting time! ===
<pycon_bot> Enter your vote in the form “123, 456, 789”. You may vote for as many talks as you like, but please keep in mind the limited number of available slots.
<jacob> 123, 456
<jesse> 456
<tim> 456
…

The bot will tally the scores, telling us the results::

<pycon_bot> === Results ===
<pycon_bot> WINNER - Talk #456: 9 votes (90%)
<pycon_bot> DAMAGED - Talk #123: 5 votes (50%)
<pycon_bot> OUT - Talk #789: 3 votes (30%)

Remember there may be no winner, and there may be any number of damaged talks.

Finally, the chair will record the talks as in, damaged, or out, based on the voting.

Meeting guidelines

To keep things moving smoothly and quickly, please try to follow these guidelines during all IRC meetings:

  • The chair will send an agenda for each meeting to the mailing list in advance. This'll list all the talks we plan to discuss. If you're a champion for one of the talks up for that meeting, please try extra-hard to make it to the meeting! Otherwise we'll have to "channel" you based on what you wrote on the website, and that won't do your voice justice.

  • Please use the time before each meeting to refresh your memory on the talks for each meeting; you'll only have a short period of time during the meeting to read each talk. Please come prepared.

  • Please stay on topic. We have a lot to do, and not much time to do it. Please direct any unrelated chatter to private chat, other chat rooms, or the mailing list.

  • For Kittendome, focus only on the talk itself, not on other talks. If there's another, better talk on the same topic try to ignore it; we'll sort it out in Thunderdome. In this round, just deal with each talk on its (de-)merits.

  • In Thunderdome, focus on the talks in each group, and not on other groups, other talks that didn't make it to Thunderdome, talks you wish had been submitted, etc. It's impossible to evaluate every talk against every other talk, so we have to work in groups.

  • Disclose bias if it exists. If the speaker is a friend, colleague, coworker, competitor or nemesis that's completely okay, but tell the group. If you wrote a piece of software covered by a talk, say so. Bias, in and of itself, isn't really an issue -- we have a big enough pool of reviewers that it'll balance out. But if you are biased, tell us so that we know where your review is coming from.

  • No talks will be tabled once the meeting begins. Thus, if you feel strongly about a group, topic, or talk then make a special effort to come to the meeting! If you absolutely can't make it, email the chair and a talk can be moved to a different meeting, but please let the chair know well in advance.

  • Generally, debate will not be extended. Again, we have too much to get through to spend extra time. However, if there's a particular contentious debate, the chair may choose to extend debate, so ask the chair if you think the group deserves extra time.

  • The chair gets the final call. Most of the time the votes will identify a clear winner, and the chair should follow the votes 99% of the time. However, if there's a tie, or if votes are really close, or if the voting is otherwise not quite clear, the chair gets to make a judgement call. This should happen rarely, but it will happen.

  • Decisions are final. Once a decision has been made, discussion should end and we'll move on. If you feel a decision was made in error, bring it up on the mailing list or via private email to the chair(s).

  • If a talk under discussion has been submitted by you, you'll need to recuse yourself. Do this by leaving the chat room; the chair will invite you back when discussion is over.