Tutorial Contents

Robust fitting and smoothing

Fitting

Smoothing

Contents

Robust fitting and smoothing

Robust Polynomial Fitting

Robust polynomial fitting is implemented as follows. First, a standard least-squares polynomial fit to all the data is performed. The median average deviation (MAD) is then calculated- i.e. the median of the absolute values of the residuals, where a residual is the differences between an actual data Y value and its fitted Y value. On the next iteration, a second fit is performed, but now the contribution of each data value to the fit depends on the size of its residual (to the previous fit) relative to the MAD. The weighting is set so that data whose residuals are greater than a certain threshold have zero weight and are ignored. This threshold is a multiple of the MAD, and 6 is a typical threshold value. After this fit, a new value of MAD is calculated. This process is then repeated for further iterations as desired. Each iteration reduces the influence of outliers, but in practice, two iterations are usually sufficient to produce a good fit.

The user sets:

Robust Smoothing

Robust fitting and smoothing is carried out using a modified LOWESS technique. The term LOWESS stands for “locally-weighted scatterplot smoothing”, and the robust version of this is a technique developed by William S. Cleveland. LOWESS is an extension of local polynomial (LP) smoothing, which is itself an extension of the moving average (MA) smoothing. The principles of robust LOWESS are well described in a paper by Hen et al. Quotes in the following sections are taken directly from that paper.

LP [local polynomial smoothing] uses a time window centered on each time point, but instead of using the average location [value] over the window [which is the MA moving average technique], it fits a low-order polynomial (usually a straight line or a parabola) ... to the data at the window. Once such a polynomial is fitted (using simple or weighted least squares) the smoothed location at time t is the value of the polynomial at that point.

Simple polynomial fitting means that all data points within the window contribute equally to the fit. Weighted polynomial fitting means that data points close to the point which is actually being fitted, i.e. the point at the centre of the window, carry more weight in the fitting procedure than data points nearer to the edge of the window. The usual weighting function is a tricubic function. The LOWESS technique uses this weighted method. “The strength of LP smoothing is controlled mainly by choosing the window’s width (as in MA) but also by the degree of the polynomial.” The advantage of LP fitting over MA fitting is that it does not distort peaks in the data so much as the latter technique.

Neither MA nor LP are robust in handling data outliers. All data points contribute equally to the LP fit as they pass through the centre of the window, and thus pull the fit in their direction. A few large outliers can quite dramatically distort the fit from what is the obvious ‘true’ path of the data. The LOWESS technique introduces an iterative reduction in the contribution of outliers to the fit.

As in the weighted LP, the first iteration of LOWESS fits a polynomial to the data at a time-window centred at t. The resulting polynomial, however, is used only as a first estimation. Each original data point is then assigned a weight according to its difference from its first estimation (residual). A larger residual (indicating a poorer fit) results in a smaller weight for the corresponding data point, implying it will be less relevant for computing the next fitted polynomial. At the extreme, a very large residual indicates that the point is an outlier, and it is assigned a zero weight, implying it will have no effect at all on the next iteration. In the second iteration of LOWESS, the raw data in the window is fitted again with weighted LP, but this time using also the weights according to the residuals [as well as weights according to the distance from the centre of the window].”

After a suitable number of iterations “the fitted polynomial is used to derive the LOWESS smoothed location  [value] ... at time t.”

The user sets: