Attribute Agreement and Misclassification

Why it matters

Not every quality decision produces a continuous measurement value. Finding a surface acceptable, counting a mark as legible or accepting a joint are classification decisions. In such a system, reliability does not consist only of the same observer repeating the same decision. Observers can agree with each other and still make the same error against the reference. For this reason two separate questions are asked: how consistent are the decisions, and how well do they match the accepted reference status? The data that answers the first question does not automatically answer the second. If it is not known how the reference was set up, accuracy cannot be declared from the number of shared decisions alone. A classification system covers the instrument, the description, the sample and the person making the decision together.

In a two-class decision there are four possibilities. An object that is nonconforming according to the reference can be found nonconforming, or it can be missed by being called conforming. An object that is conforming according to the reference can be found conforming, or it can be wrongly rejected. The share of correct decisions in the total gives the overall agreement. The denominator of the miss rate is the objects that are nonconforming according to the reference. The denominator of the false rejection rate is the objects that are conforming according to the reference. When the denominators change, the meaning of the question changes. How many nonconforming items escape among all decisions is not the same information as what share of the nonconforming items is caught. This distinction matters especially in processes where the defect is rare. Constantly giving the same answer for the majority class can produce a high overall agreement.

The reference decision must be set independently and with justification. Taking the studied observer's own decision as the reference is circular verification. A clear definition, suitable inspection conditions and, where necessary, a more reliable verification method should be used. For borderline samples the reference may also have uncertainty. Instead of silently pushing these objects into the conforming or nonconforming column, the uncertainty is flagged and the reference is clarified. A joint decision of several experts can be useful; but a majority vote alone does not constitute absolute truth. It should be explained which finding it rests on. The purpose of the study is not to blame people but to find where the description and the decision conditions diverge. A faulty shared description can produce the same error in all observers.

Diagram: [Reference class] × [observer decision] → [correct acceptance / correct rejection / false acceptance / false rejection] → [its own denominator for each rate]. The cross-tabulation makes a shared error visible; observers merely giving the same answer does not show accuracy against the reference. The choice of samples determines the scope of the result. If defects are rare in real production, enough nonconforming items may not be found in a small random sample. To test the error types, nonconforming and borderline samples can be deliberately increased. However, the overall agreement in this enriched group or the share of errors among the rejected cannot be presented as if it were the natural distribution of production. Even conditional miss and false rejection rates depend on the defect types and difficulty levels selected. For this reason the variety of the samples is explained as well as their number. Read the decision diagram as reference status → observer decision → four cells → separate rates. The total percentage must not erase the detail in the last boxes; acceptance should rest on the results for the especially important error types. All numerical scenarios in this lesson's short quiz are also covered by the following notice: This dataset was generated synthetically for training; it does not represent a real business, employee or customer study.

Formula