nlsys

Guide

Despike data and remove outliers from a time series (Hampel and friends)

Spikes from EMI, loose connectors or step transitions wreck averages, FFTs and thresholds. But the danger in learning to despike data and remove outliers is deleting a real event by mistake. Here is how to do it safely.

Spike vs outlier vs real event

A spike is a one- or two-sample excursion with no physical support. A real event — a load step, an arc, a fault — can look identical for a sample and then persist. The rule that keeps you honest: a genuine spike is isolated and unsupported by its neighbours; a real event has continuity. Never despike blind.

1. Z-score threshold (crude baseline)

import numpy as np
z = (x - x.mean()) / x.std()
mask = np.abs(z) > 4          # flag, then decide

Weak because mean and standard deviation are themselves wrecked by the spikes you are trying to find, and a global threshold ignores local context.

2. Hampel filter (the workhorse)

The Hampel filter compares each point to a rolling median and a robust scale (median absolute deviation). Median and MAD are not dragged around by outliers, so it flags spikes reliably and replaces them with the local median.

import numpy as np, pandas as pd

def hampel(x, window=7, n_sigma=3.0):
    x = pd.Series(x)
    med = x.rolling(window, center=True).median()
    mad = (x - med).abs().rolling(window, center=True).median()
    thresh = n_sigma * 1.4826 * mad          # 1.4826 * MAD approx std
    spikes = (x - med).abs() > thresh
    out = x.copy(); out[spikes] = med[spikes]
    return out.values, spikes.values

Tune window to your sample rate (wide enough to hold several clean samples around a spike) and n_sigma to how aggressive you want to be (3 is a sensible start). Return the mask, not just the cleaned data — you want to see what was removed.

3. Rolling median

from scipy.signal import medfilt
clean = medfilt(x, kernel_size=5)   # odd kernel

Simple and spike-robust, but a wide kernel rounds real corners. Good as a first pass, not as a final smoother.

4. Residual-based (model-aware)

Predict each sample from its recent history (or a model) and flag points whose residual is large. This distinguishes a spike (large residual, then back to normal) from a real regime change (large residual that persists) — the safest approach when you must not delete events.

How to choose window and threshold

FAQ

What is the best filter to remove spikes from data?

A Hampel filter (rolling median plus MAD) is the standard workhorse because the median and MAD are not distorted by the spikes themselves, unlike a mean/standard-deviation z-score.

How do I avoid deleting real events when despiking?

Only remove excursions that are isolated and return to the local trend within a sample or two; a real event persists. Flag candidates first, inspect the mask, and keep an audit trail.

Z-score or Hampel for outliers?

Prefer Hampel. A z-score uses the mean and standard deviation, which the outliers themselves corrupt, so it both misses and over-flags on spiky data.

Skip the code. Drop your CSV into the free browser demo — no sign-up — and get a denoised, gap-filled, de-spiked file back in a minute. Every column, priced by volume when you need more. See the tool →

Related guides