Confirmation Run and the Limits of an Experiment

Why it matters

The combination that looks best in the experiment table may also have carried a favorable result by chance. Using the same data both to select the candidate and to prove that the selection is certainly right leaves this risk invisible. A confirmation run tests, with separate runs planned after the selection, whether the expected gain appears again. New specimens, recorded conditions and an acceptance rule set in advance are needed. Copying a few rows from the old table into a new file does not create a separate series. The purpose of confirmation is not to try to find the best-looking number again; it is to test honestly a prediction written in advance about the selected candidate.

The expected gain is not meaningful without stating which reference it was calculated against. Comparing the response of the candidate on a new day with a reference from the past can add a day or material difference to the gain. Where possible, the candidate and the reference are run in the same blocks, under common measurement conditions and in an order randomized in advance. In this way the new series gets its own comparison. The reference method must be described clearly and left unchanged. Neglecting the maintenance of the reference while preparing the candidate carefully also does not give an unbiased comparison. The two conditions are run with the same discipline in terms of safe and defined application; the actual settings and deviations are kept visible.

The acceptance rule is written before the results arrive. The rule may include the smallest technically meaningful gain, the quality and safety conditions to be preserved and, if needed, an acceptable distance from the prediction. These limits do not replace a statistical confidence interval. A small confirmation series can provide support only within a limited scope. If the risk of the decision is higher, an appropriate uncertainty assessment and a more comprehensive plan are needed. When the result misses the target by a small margin, lowering the threshold afterwards is not confirmation. A failure is also useful information: the run may have differed, the measurement may have changed, or the first gain may have looked larger than it was. Every possibility is investigated through the record.

The limit of the experiment is the scope of product, material, machine, run and measurement together with the selected factor levels. Getting a good result at two temperatures does not make certain the behavior at every temperature in between or at a higher temperature. The response curve may not be linear; the safety limit also needs a separate assessment. It cannot be assumed that another product that looks similar will give the same response to the same setting. The confirmation result is stated only within the scope that was actually tested. When moving into production, the monitoring and re-examination conditions are determined. This record does not reduce the value of the experiment; it makes clear how much basis it gives to which decision. Staying within the range tried also does not by itself ensure a certain transfer. Even if similar responses are seen at the two ends, different behavior may exist in between. A decision that requires the suitability of an intermediate setting needs new information. Moreover, while the temperature range stays the same, a change in product thickness can create a different physical situation. The scope of validity is made up not only of the numerical settings but of the process conditions tested together.

Formula