Comparing the observed performance of two machine learning algorithms requires statistical hypothesis testing rather than direct comparison of point estimates, because a difference in measured scores may arise from the variability of the underlying data distribution rather than from a genuine difference between the methods. Hypothesis testing formalises this by fixing a confidence level in advance (conventionally 95% or 99%) and rejecting the assumption that the systems are equivalent only when the evidence exceeds that threshold. Valid comparison further depends on design of experiments: exploratory analysis to identify which hypotheses are worth testing, explicit classification of dependent, independent, and extraneous variables with the latter held constant, and active avoidance of the spurious effects — ceiling and floor effects, order effects, and sampling bias — that would otherwise invalidate the inference.
Comparing the observed performance of two machine learning algorithms requires statistical hypothesis testing rather than direct comparison of point estimates, because a difference in measured scores…