Lateral Shift Error due to Graduation Anomalies and Line-Detection Algorithm in Line Scale Measurement

The measurement error resulting from graduation anomalies and the signal processing algorithm used for determining the positions of graduations on line scales was investigated by simulation and experiment. Optical image-forming simulations were carried out on models of 6-µm-wide graduations with three sizes of defects (0.5, 1.0 and 1.5 µm) at one edge. A digital filter was used in signal processing to obtain the first differential to determine the positions of the graduations. The minimum values of the lateral shift of the determined graduation positions were observed for the three defect sizes when using a 9-µm-wide differential filter. An experiment was also carried out on an ordinary line scale with 6-µm-wide graduations using a high-precision laser-interferometric line scale calibration system by measuring seven positions on the scale in the direction perpendicular to the measurement axis. The root mean square of the standard deviations from the linear fitting lines constructed using the measured positions over a 300-mm-long line scale was 2.8 nmwhen the differential filter width was 9 µm. It was demonstrated that a differential filter was effective in reducing the lateral error due to graduation anomalies.