Perturbation analysis for an orthogonal least-squares estimator

Coordinate measuring machines (CMMs) are used to check the geometric integrity of component parts. The geometric constraints to which a part must conform, as defined e.g., by The American National Standards Institute, assume the use of some type of gauging system when inspecting the part. Statistical issues arise in interpretting CMM data in the inspection of part tolerances. We consider a set of n planar regions on the surface of a part. The unit vector normal to each plane is estimated by orthogonal least squares. The small-sample density of this estimator (on the unit sphere S²) is determined asymptotically as the variance of the CMM error approaches 0. To a first-degree approximation, this density is Fisher-von Mises. Diagnostics are reviewed to test the geometric constraint that the n planar regions are oriented correctly with respect to one another, and to test the flatness of planar regions.