Understanding the detailed judging tally sheets

After every contest, the KCBS Representatives distribute various computer printouts to the teams.  These documents contain valuable information that should be understood by teams, judges and others.  The KCBS Computer program this year prints out three documents which are distributed after the results are announced:  

• "Tally Sheet" -- one per category, which shows each team's finish, team number, name, the six judges' total scores for that entry, and the total points scored for each team's entry.  An example is the Barbeque and Blues Tri-tip Tally Sheet here.
• "Overall Rankings" -- one per contest, which shows each team's overall finish in the Grand Championship contest, the team number, name, the total scores earned by the team in chicken (CK), ribs (RI), pork (PK), and brisket (BR), and the total points scored by the team toward the Grand Championship.  An example is the Barbeque and Blues Tri-tip Tally Sheet here.
• "Scoring Detail" -- one per team, which shows the team's team number, alternate number, name, status (A/P), and a table which shows, for each category, the three non-weighted scores (Appearance, Taste and Tenderness) awarded by each of the six judges.  After the awards ceremony, the "Scoring Detail" and "Overall Rankings" is usually given to each team. 

Each of these documents includes information that is very valuable to the teams, but it is presented in a way that might be confusing for many teams, including veteran competitors.  Because these documents contain such valuable information, teams should learn how to read and understand them.

Barbeque and Blues Tally Sheets

The tally sheets of the Barbeques and Blues contest, which was held on June 9, 2001 at Long Beach, will be used to explain how to read and understand the results.  The tally sheets for the Barbeques and Blues are available here.

The tally sheets for the individual categories (like Chicken) have several columns:

• Place -- the place in which the team finished (first, second, etc.).
• Team number -- this is the team number given to the teams at the event (the website eliminates this number only to save space).
• Team name -- starting next year, KCBS hopes to have in place a team name registration system so that the teams can be tracked accurately.  In the meantime, the limitations of the computer program require that all team names be in upper-case (capitalized) form and contain no apostrophe's, etc.
• J1 J2 J3 etc. -- these are the Judges.  Each judging table has six judges, and each judge scores each entry in Appearance Taste and Tenderness.  The numbers for each judge are the total of Appearance Taste and Tenderness under the old format (described below).  In 2000, the computer program was changed, but the Tally Sheets use the old format.  Remember that the lowest score is thrown out, and only the five highest judges scores are counted.
• Total points -- this is the total points awarded by the five highest judges.  Note again that the points shown on the tally sheets are not the actual points under the "new" system.  See below for more on this.

The old system for weighting scores

The Tally Sheets show the judges scores and totals computed under the "old system for weighting scores".  Under that system, the Taste Score is doubled and is added to the Appearance and Tenderness Scores.  So, for example, if a judge awards scores of 9-9-9 (a perfect score), then the computer will add them as ( 9 + (9 * 2) + 9).  Remember that the scores are always in the same order:  first Appearance, then Taste, then Tenderness, so the middle number always means Taste.  Therefore, under the old system, a perfect score by a single judge equals 36 points.

Similarly, a judge who scores 8-8-8 will total 32 points, and a judge who scores 7-7-7 will total 28 points.  Almost all scores will be in the range of 32 to 36 (the high range) and 28 to 32 (the low range).

Understanding scores between 36 points and 32 points

36 points equates to a score of 9-9-9.  This is a perfect score from a judge.

35 points means that the judge awarded a 9 in Taste plus a single score of 8 in either Appearance or Tenderness (but not both).  In other words, the judge's score card is 9-9-8 or 8-9-9.  The only way to find out which is to look at the scoring detail.

34 points means that the judge awarded either 9-8-9 (an 8 in Taste, which is weighted twice as much as the other scores in the old system, and 9s in Appearance and Tenderness), or 8-9-8 (a 9 in Taste two 8s in Appearance and Tenderness).  Again, the only way to discover which is by reference to the scoring detail.

33 points means that the score was either 8-8-9 or 9-8-8.  In either case, the Taste Score was 8, and another 8 was awarded in one of the other areas.

32 points almost always means that the judge scored 8-8-8.  However, it is also possible to total this with a score of 9-7-9.

31 points and lower always mean that the judge scored Taste with an 7 or lower, except if there was an Appearance Penalty (see below).

28 points and lower usually (but not always) means that an entry received an Appearance Penalty.  An Appearance Penalty means that each of the six judges at a table is required to score an entry's Appearance as a 1, because there was a violation of Rule 20, relating to the use of illegal garnish.  For example, if an entry box includes red-tipped lettuce (a violation of Rule 20), or if an entry does not contain the minimum six identifiable portions (a violation of Rule 18), each of the judges must score the Appearance as a 1.  If this happens, then the highest possible score is 1-9-9, which totals to 28 points.

4 points means that the entry was penalized with either an Appearance Penalty or a Disqualification Penalty, and the judge's score was 1-1-1.  If one of the six scores equals 4 points and the other scores are 28 points or less, then the cause was an Appearance Penalty (such as the Rule 18 and Rule 20 violations described in the preceding paragraph).  However, if each of the six judges scored the entry with 4 points, then this means that the entry was subject to a Disqualification Penalty, and each of the judges scores the entry 1-1-1.  For example, if an entry box contains a sliver of foil (a Rule 21 marking violation), then the entry is disqualified and all judges must score it 1-1-1.

0 points does not mean that an entry was disqualified!  When an entry is received at the turn-in table but then disqualified for some reason (like a Rule 21 marking violation), the entry will receive scores of 1-1-1 from each judge, so the tally sheets will report a total score of 4 points.  If the tally sheet shows 0 points, this means that no entry whatsoever was turned in.  Most often, this occurs simply because a team decided not to turn in a category, even though the team was registered for it.  In the Barbeque and Blues results

Perfect 180

A perfect 180 means that five of the six judges awarded an entry with scores of 9-9-9.  That is, the computer counts only the five highest scores by the judges (the low score is thrown out).  So if five judges award 9-9-9, then the computer equates this to a score of 36 points by 5 judges, and 5 times 36 is 180, a perfect score.

For example, at the Barbeque and Blues in Ribs, Mojave Hot Stuff received scores from the judges of 35, 36, 36, 36, 36 and 36.  Since the score by Judge 1 was the low score, the computer threw it out and counted only the five highest scores, which equals 5 times 36, or a Perfect 180.

Teams which score a Perfect 180 in a KCBS-sanctioned contest receive a Gold Start Award and recognition in the Bull Sheet.

Appearance Violation

The Barbeque and Blues in Ribs also shows the results of an Appearance violation.  The last team received scores of 04, 28, 04, 04, 04 and 04.  In this particular case, the team made a critical mistake in its ribs entry:  it failed to cut the ribs complete into separate portions.

When the entry was opened for the Appearance Score, it was evident that the ribs were not completely cut through.  Nevertheless, the judges awarded their Appearance Scores "as if" the ribs will easily separate when they are lifted.  After the judges gave their Appearance Scores to this and the other entries, the Table Captain passed the entry to the first judge to select a sample and pass the entry onto the next judge.  However, when the first judge lifted a single rib, all the ribs came with it (because they were not cut apart).  That meant that there were not enough pieces for the six judges, which is a violation of Rule 18 and required the KCBS Rep to instruct the six judges to change their Appearance Scores to 1.  (Judges may not change their scores except at the specific direction of the KCBS Rep and then only when the explanation for the change is written on the back of the score cards by the judges.)

Next, the judges had to score the entry for Taste and Tenderness.  However, only the first judge (who happens to be listed on the tally sheet as J2), had anything to taste, so only that judge could award a score in Taste and Tenderness, and the judge awarded the best score possible, which was 1-9-9 (which totals to 28 points).  The other five judges had nothing to taste, so they were required to score Taste and Tenderness with 1s, so their score cards were 1-1-1 (which totals to 4 points).  Thus, the scores of the six judges were 04, 28, 04, 04, 04 and 04.

The computer discarded one of the low scores and totaled the other 5 scores.  Notice that these should total to 48 points:  28 + 4 + 4 + 4 + 4 + 4.  However, the tally sheet shows that the total instead is 044.0002 -- where did this come from?  This requires an explanation and understanding of the new system.

The new system for weighting scores

The old system for weighting scores was described above and simply means that the Taste Score was doubled and then added to the Appearance Score and Tenderness Score.  The low score was discarded and the five highest scores were totaled, and the result was the total weighted score of the entry by the judges.

Two years ago, however, KCBS changed to a new system for weighting scores.  Briefly, the Tenderness Score was worth twice as much weight as the Appearance Score, and the Taste Score was worth twice as much weight as the Tenderness Score.  This can be illustrated as follows:  if an entry received a score of 1-1-1, it would be weighted as follows:  the Appearance Score would stay 1 point;  the Tenderness Score would be worth twice as much, or 2 points;  and the Taste Score would be worth four-times as much as the Appearance Score, or 4 points:  and together, they would total to 7 points (1 + 2 + 4).

This was a simple idea, but its execution became complex because KCBS opted to keep the same basic scores of 1 to 9, to keep a perfect score by a single judge (a score of 9-9-9) to equal 36, and thus to keep a perfect score by all judges (the 5 highest scores) to remain a Perfect 180.  To do this, the computer has to perform some heavy calculations, because the numbers are no longer even.  Instead, Rule 19 was changed to provide:  "The weighting factors for the point system are as follows:  appearance, 0.5714;  taste, 2.2858;  tenderness / texture, 1.1428."  

This means that the Appearance Score by a judge is multiplied by 0.5714;  the Taste Score is multiplied by 2.2858;  and the Tenderness Score is multiplied by 1.1428.  The resulting numbers are added together, and this equals the total score of that entry by that judge.  (You can use a calculator or pencil and paper to discover that a score of 9-9-9, when adjusted with these new weighting factors, will still total exactly 36 points.)

Only the tally sheet totals use the new system

Based on the new system, a score of 9-9-9 should total 36 points, which it does.  However, a score of 8-9-9 should total 35.4286, yet the tally sheets actually report this under the "old system" as 35 points.

Nevertheless, the computer does track the scores under the new system, even though it reports the scores of the individual judges under the old system.  This is a minor and temporary problem which is expected to be fixed in the next iteration of the computer program.  In the meantime, teams and judges can verify that the contest results are determined under the new system for weighting scores, as described above.

Report of scores awarded by individual judges

As described above, the KCBS computer program uses the new system for weighting scores, but the tally sheets report the scores of the individual judges under the old system.  There is no specific reason that KCBS does not want to disclose the actual scores by each judge.  That is, the computer program could report the scores in the form 9-8-7 rather than in its current form of 32.  Again, under the new system for weighting scores, the actual total of a score of 9-8-7 is calculated as:  (9 * 0.5714) + (8 * 2.2858) + (7 * 1.1428), which in this case equals (5.1426 + 18.2864 + 7.9996) or 31.4286.

Because the old weighting system permitted an easy translation of a judges total scores to the individual scores, it is expected that a revised KCBS computer program will include this information.  All teams deserve to know the individual scores issued to all other teams, because a blind judging system remains credible only if it is transparent, so that anyone can inspect the scores and verify the accuracy of the outcome.

This does not to say that the identity of the scores awarded by individual judges can or should be known to anyone.  There is no way that the computer (and thus no way that any of its output) can correlate the identity of a judge with the scores of that judge.  Many teams and even some judges believe that the tally sheets can be used to determine which judge awarded which score, but this is incorrect.  In fact, Judge 1 for one team is not the same judge as Judge 1 for another team.  Indeed, a judge sitting at a judges table will not have the same designation from one entry to another.  The computer does not input the score cards in any particular order, so that if "Fred" is Judge 1 in the Chicken Category, "Fred" might be Judge 4 in Ribs and Judge 6 in Brisket.  The score cards are not in any order, so the computer output does not order the judges' identities.

Scoring Detail

As explained above, the tally sheets report only each judge's total scores (computed under the old weighting system), such as 36 or 34, rather than the judge's individual scores, such as 9-9-9 or 9-8-7.

Under current procedures, the details of each judge's total scores of each of a team's entries are distributed to that team in the "Scoring Detail" printout.

Team Number and Alternate Number

Each team's "Scoring Detail" shows the team's Team Number and its Alternate Number.  The Team Number is the number assigned at random to each team in an event.  Team Numbers are usually assigned numerically, in the form 1, 2, 3, etc.  The Team Number is written on the top of each team's turn-in box (and teams are responsible to ensure that this number is written on its turn-in boxes).

When a team turns in its entry at the turn-in table, the KCBS Representative puts a sticker over the Team Number, so that the Team Number is covered and impossible to see.  On the sticker is a number, called the Alternative Number.  When the entry is shown to and scored by the judges, none of the judges knows that entry's Team Number (or name or other information).  Instead, the judges use the Alternate Number to identify and score the entry.  

There is a numerical relationship between the Team Number and the Alternate Number, but, until after the results are announced, the relationship is known only to the KCBS Representatives and the computer.  For example, the relationship may be as follows:  the Alternate Number equals the Team Number plus 101.  Thus, there is one and only one Alternate Number per team, but the use of this system helps to ensure the integrity of the blind judging method.  

After the results of the contest are announced at the awards ceremony, the relationship between the Team Number and Alternate Number is made public.  This also helps to ensure the integrity of the system, and it also permits judges to identify particularly good samples, so that they can extend their congratulations and even hire the team for catering.

Judge's scores

On the "Scoring Detail" sheet, after the Team Number, Alternate Number and Team Name, there is a table which looks like the following:

These are the raw, non-weighted scores by each of the six judges.  As explained above, the "Judge1" who judged Tri-tip is probably not the "Judge1" who judged Chicken, and so forth.

Veteran teams will study the "Scoring Detail" very carefully to learn what the judges liked and what may need improvement.  For example, most veteran teams have developed a consistent method for presentation which should ensure almost all 9s in the Appearance Scores.  If a team sees that its entry is receiving low scores in Appearance, then improvement in that area should be made.  Thus, veteran teams will look at their scores first in terms of their Appearance Scores, then their Taste Scores, and finally their Tenderness Scores.  

Sometimes, the team will find that the scores are not consistent.  This may indicate that the judges at that particular judging table were not consistent -- which could mean that some are too easy or that others are too strict.  KCBS Certified Barbecue Judge school is helping contests improve the quality of the judging by improving the expertise of the judges.  In California, most contests are judges by a majority of KCBS Certified judges, and the KCBS Representatives and Table Captains work to ensure that all judges apply consistent standards.  The occasional "rogue" judge -- who refuses to follow the rules -- is not invited to judge again.

However, too many teams seem to blame a poor showing on the judges, but poor or inconsistent scores may mean not the judging was substandard but that the entry itself was not consistent.  For example, in the Ribs Category, inconsistent scores in Tenderness are common when ribs are barbecued in an un-tuned offset smoker (which has hot spots).  The ribs near the end of the rack may finish faster than those in the middle.  As a result, some will be juicier or tougher, depending on where they are cut from the rack.

Teams should look for patterns like this and then try to figure out ways to continue the high scores while minimizing the low ones.