Lab Manual: CS Accuracy Measurements Overview Procedures for making Cued Speech Transliterator accuracy measurements are documented in this manual. This section of the manual provides definitions, background information, and scoring tips. The other four sections of the manual describe the detailed procedures used in each of the four components of the scoring process: 1. Decide Whether to Score 2. Score Categorization 3. Placement Identification 4. Handshape Identification In each of the remaining sections of the manual, procedures used for evaluating a particular component of the cue production are first summarized in a flow chart. Hyperlinks from each element in the flow chart lead to more detailed explanations. All four flow charts are posted in the lab and should be used as a reference for all measurement. Definitions A. CST = Cued Speech Transliterator B. Target Cue = The cue that the CST is expected to produce. It is written in cue notation on the Excel spreadsheet. C. Cue Under Review = The actual cue that the CST produces that is presently under consideration for scoring. It appears on the computer screen. o Finger Under Review = The finger of the cue under review that is presently under consideration for “up”/“down” decisions used in determining handshape. It appears on the computer screen. D. Response Cue = The cue that is actually scored (not every cue under review is actually scored). It is written in cue notation and entered in the “CST response” column on the Excel spreadsheet. E. Correct = This occurs when the cue under review matches the handshape and placement of the target cue. Scoring = A correct cue is indicated with a “C” that is recorded in the CST’s response column as follows: Target cue OK Target CST Word CV pair sequence insertions cue Response Movements /mu/ 5c C /v/ 2s C F. Substitution = This is an error that occurs when the cue under review is produced in place of the target cue. The correct handshape and/or correct placement of the target cue is replaced by a different Cued Speech handshape and/or placement. Scoring = The substituted cue is recorded in cue notation in the CST’s response column as follows: Overview - Page 1 of 10 Lab Manual: CS Accuracy Measurements Target CV cue OK Target CST Word pair sequence insertions cue Response Shrink /Ѐ/ 6s 3s On /Ϫ/, /ѐ/ 5sf, 5c C /n/ 4s 4c G. Insertion = This is an error that occurs when the cue under review is an extra cue that is not produced in place of any target. Scoring = The inserted cue is recorded in cue notation after an “I” in the CST’s response column (“I-inserted cue”) by inserting a row in the Excel spreadsheet at the point where the cue was inserted in the script. Target CV cue OK Target CST Word pair sequence insertions cue Response females /fi/ 5m C I-4m 5c + C /گme/ 5t C /l/ 6s C /z/ 2s C H. Omission = This is an error that occurs when none of the cues under review can be scored for the target cue. This occurs when the cue is not produced by the CST or when the attempted cue under review is not scoreable. Scoring = An omitted cue is indicated with a “X” that is recorded in the CST’s response column as follows: Target CV cue OK Target CST Word pair sequence insertions cue Response On /Ϫ/, /ѐ/ 5sf, 5c C /n/ 4s X I. Transition = Transitions are any movements or cue productions that occur between cues and are not scoreable. Examples of transitions include a persisting handshape that is moving away from the last placement (but does not remain after arrival at the new placement) or a constantly changing handshape (that does not last at least three frames) produced on the way to the new placement. Overview - Page 2 of 10 Lab Manual: CS Accuracy Measurements Measuring Tips A. Making the Measurement Tool Many scoring judgments require the grader to measure the distance between a reference point on the hand (usually the contact finger) and a reference point on the CST’s body (e.g. face midline). Distance is measured in “finger-width” units. Therefore, a “measurement tool” that is the same width as the CST’s finger must be made prior to scoring. Because the CST may be in different positions/placements in each video and the video zoom may have changed, one measurement tool for each CST at each separate speed needs to be made according to the following steps: 1. Step through the video, and select a frame that has a clear handshape 5 that is flat and facing the camera. Be sure there is no wrist/arm rotation and that the fingernail and edges of the middle finger are clearly visible. 2. Measure the CST’s middle finger from the left side of the fullest (or middle) part of the nail to the right side of the nail at the same point. 3. Use a piece of cardboard with a straight edge (an edge that hasn’t been cut ensures the full length of the finger will be straight). A long straight piece of cardboard of equal width all the way through needs to be used. 4. Mark the measured distance from the straight edge in two places, then draw a line connecting the two measurements. 5. Cut down the center of the line carefully. 6. Hold the measurement tool up on the computer screen to the finger that was used to make the measurement tool. You should still be able to see the edges of the middle finger but no part of the fingernail. B. Reference points Consistency is a key factor when measuring to/from any reference point (e.g. midline). Once each of the reference points (throat midline, face midline, chin crease, etc.) is determined for a particular CST the same reference points must be used every time when determining placement. For each reference point, find a facial feature, body marking, element of an article of clothing or piece of jewelry that can be used consistently. Reference points CANNOT be changed for each cue under review. Scoring Tips A. Benefit of the Doubt A rule that is used to give the CST positive credit for the cue they are producing. The benefit of the doubt rule is used to help the CST’s score in borderline situations when absolute answers are not easily formulated, as a result of a number of issues affecting the CST’s production. These issues fall into two broad categories 1. Visibility issues: A visibility issue that is not the transliterator’s fault (i.e. not poor form). Most often due to equipment limitations of the camera such as blurry frames, poor lighting etc. Examples of how the benefit of the doubt rule would apply when there is a visibility issue: Overview - Page 3 of 10 Lab Manual: CS Accuracy Measurements o Handshape = If the handshape of the cue under review could match the target cue (within reason), give credit for the handshape in order to minimize errors. o Finger (equipment issues) = If the finger of the cue under review could be determined to be “up” or “down” (within reason), then make the decision that would minimize errors. 2. Measurement issues (tie goes to the CST): Any measurement that results in a tie between two placements or reference points. o Placement (measurement tool) = : If the edge of the finger and measurement tool just meet (abutting) and the placement of the cue under review could match the target cue, give credit for the placement in order to minimize errors. o Placement (chin crease) = If the CST touches directly on the chin crease, and the placement of the cue under review could match the target cue, give credit for the placement in order to minimize errors. o Placement (closer to mouth vs. closer to bottom of chin) = If the CST produces a cue directly even with the chin crease, and the placement of the cue under review could match the target cue, give credit for the placement in order to minimize errors. o Placement (determining reference points) = Depending on the CST’s face anatomy, using the chin crease as a reference point may not produce the same result as using the mouth corner and bottom of chin as reference points. In these situations, use the reference point that gives credit for the placement in order to minimize errors. B. Error minimization In deciding what cues to score on which row of the Excel sheet, the goal is to minimize the number of total errors (with partial credit) but also to align things with what the transliterator is mouthing. When deciding what cues to omit/substitute, stay as close to the production visible on the CST’s mouth as possible. 1. General: If cues scored can be aligned with target cues in more than one way, count errors for each possible scoring alignment using the values below to determine the sequence with the least errors. i. Correct: -0 ii. Insertions: -1 iii. Omissions: -1 iv. Substitutions 1. wrong handshape / wrong placement (if scoreable): - 1 2. right handshape /wrong placement (or wrong/right): -0.5 3. movement errors: -0 2. Liaisons: If cues can be aligned so that an omission occurs for a target cue immediately above or below a liaison opportunity (in the Excel spreadsheet), re-align the response cues to minimize errors. Overview - Page 4 of 10 Lab Manual: CS Accuracy Measurements i. Example 1: Score one “X” for the liaison rather than scoring two “X’s” immediately above and below the target cue ( - 1 instead of – 2).