Guide: Running a Winner Selection Call with the Judging Panel

Running a Post-Judging Conference Call with the Panel

If after reviewing the judges’ scores you feel that an additional conference call is required to finalize the winners, you will want to schedule the meeting right away to ensure all (if not most) of the panel is able to attend. You can schedule individual calls with judges if they are unable to attend the group meeting, if time permits. Otherwise, you can get their opinions via email so they can be shared with the group. Below are the template agenda and “How To” guide to run the post-judging call.

Agenda for Post-Judging Discussion

• Comments from Judges on the Submissions and the Overall Experience
• Overview of Scores
• Selecting a Winner (or Winners)
• Next Steps (provide Voting or Winner Announcement information)

How to Run the Post-Judging Discussion

Before the call, the crowdsourcing project manager or meeting moderator will need to go to the View Submissions section of the HeroX page and use the orange 'Search & Filter & Sort' button in the upper right corner to filter by Score:

This way, you will be able to easily identify which submissions received the highest scores and how the judges individually evaluated each one.

In preparation for your Judging Panel call, we suggest that you:

• After sorting the entries by Score, download them as a CSV file via the “Export Entries” button under the “View Submissions” dashboard
• Review all of the judges’ scores across all submissions and highlight those that are outliers. For example, for a submission, you may find two high scores and one substantially lower score; or two low scores compared to a high score.
• Review an individual judge’s scores for all of his/her submissions. You may find that a judge is generous with scores across the board, whereas another judge offers lower scores across the board. This trendline across a judge’s set of scores will help you to compare across the outliers you have previously identified.
• Once you have done a complete review of all scores, highlight those submissions that you will like to discuss with the Judging Panel.

Prior to the call with the Judging Panel:

• Share the agenda for the call in advance with the judges
• Along with the agenda, send a list of submissions, indicating:
• Those with the highest scores
• Those with the lowest scores
• The submissions that require specific discussion
• If there are submissions that you would like particular judges to address during the call, it may be helpful to let them know in advance.

During the call with the Judging Panel, ensure that you have the following open:

• The agenda
• Your Master Scoring Sheet with all of the judges’ scores
• The submissions that require discussion, along with your notes
• The Judging Criteria

Be sure to record the call in case there are any comments or notes that you would like to be able to reference afterwards. Once the judges have joined the call, invite them to open the same documents.

When the judges scores align and there is general consensus on the winner(s), this should be a fairly simple discussion. Just confirm with the judging panel the top one, two, three (or however many prizes you’re awarding) submissions and ask if anyone has any opposition to any of those competitors receiving the prize. Then, continue to move through the agenda and make sure to allow the judges to ask any questions at the end.

If the scores are not in alignment, ask the judges to state their case on why they scored certain entries higher or lower than the other judges’ who reviewed the same submission. Go through each of the submissions that are up for discussion and seek consensus from from the judges’ as to whether or not this submission should be a contender for a prize.  

Once you have completed discussion of all of the necessary submissions, review final comments and determine the winners’ for the competition.

If the judges are not able to come to a consensus, the crowdsourcing sponsor’s primary stakeholders will have the final say in who gets awarded the money.