Guide

Despike data and remove outliers from a time series (Hampel and friends)

Spikes from EMI, loose connectors or step transitions wreck averages, FFTs and thresholds. But the danger in learning to despike data and remove outliers is deleting a real event by mistake. Here is how to do it safely.

Spike vs outlier vs real event

A spike is a one- or two-sample excursion with no physical support. A real event — a load step, an arc, a fault — can look identical for a sample and then persist. The rule that keeps you honest: a genuine spike is isolated and unsupported by its neighbours; a real event has continuity. Never despike blind.

1. Z-score threshold (crude baseline)

These are limited offline examples, not the NLSYS engine. Download the tested example functions

import numpy as np
x = np.asarray(x, dtype=float)  # one confirmed contiguous measurement segment
valid = np.isfinite(x)
mask = np.zeros(x.shape, dtype=bool)
if valid.sum() >= 2:
    sd = x[valid].std()
    if sd > 0:
        mask[valid] = np.abs((x[valid] - x[valid].mean()) / sd) > 4
# Review candidates; a global score may miss local changes and real events.

Weak because mean and standard deviation are themselves wrecked by the spikes you are trying to find, and a global threshold ignores local context.

2. Hampel filter (the workhorse)

The Hampel filter compares each point to a rolling median and a robust scale (median absolute deviation). Median and MAD are not dragged around by outliers, so it flags spikes reliably and replaces them with the local median.

from engineering_examples import hampel_flags
flags, window_median, window_mad = hampel_flags(x, window=7, n_sigma=3.0)
# Each MAD is measured around the median of that same window.
# Full finite windows only; no replacement occurs.
# Call separately on verified contiguous segments; investigate each flag.

Tune window to your sample rate (wide enough to hold several clean samples around a spike) and n_sigma to how aggressive you want to be (3 is a sensible start). Return the mask, not just the cleaned data — you want to see what was removed.

3. Rolling median

# A median-filter output is a candidate, not a classification of noise.
# Avoid zero padding and crossing physical boundaries; protect short events.
# Keep the original measurement and validate any proposed replacement.

A median filter can suppress genuine short pulses and round corners. It is not a safe default first pass before deciding which events are measurement artifacts.

4. Residual-based (model-aware)

Predict each sample from its recent history (or a model) and flag points whose residual is large. This distinguishes a spike (large residual, then back to normal) from a real regime change (large residual that persists) — the safest approach when you must not delete events.

How to choose window and threshold

• Window: a few times wider than the widest spike, but narrower than the fastest real feature you must keep.
• Threshold (n_sigma): lower removes more (and risks real events); higher is conservative. Sweep it and watch the mask.
• Always: flag first, remove second, and keep an audit trail so a reviewer can reverse any deletion.

FAQ

What is the best filter to remove spikes from data?

A Hampel filter (rolling median plus MAD) is the standard workhorse because the median and MAD are not distorted by the spikes themselves, unlike a mean/standard-deviation z-score.

How do I avoid deleting real events when despiking?

Only remove excursions that are isolated and return to the local trend within a sample or two; a real event persists. Flag candidates first, inspect the mask, and keep an audit trail.

Z-score or Hampel for outliers?

Prefer Hampel. A z-score uses the mean and standard deviation, which the outliers themselves corrupt, so it both misses and over-flags on spiky data.