Guide

Remove noise from sensor data: which filter, and when

If you record from a DAQ, a load cell, a thermocouple or an IMU, you have noisy channels. Here are the methods engineers actually reach for to remove noise from sensor data in Python, with runnable code and — more usefully — the exact failure mode of each, so you pick the right one instead of the first one.

Moving average and low-pass (Butterworth)

The cheapest denoiser is a moving average; its principled cousin is a Butterworth low-pass. You choose a cutoff frequency and everything faster is attenuated.

These are limited offline examples, not the NLSYS engine. Download the tested example functions

import numpy as np
from scipy.signal import butter, sosfiltfilt

x = np.asarray(x, dtype=float)
fs, cutoff = 1000.0, 20.0  # example settings, not a general prescription
if x.ndim != 1 or not np.isfinite(x).all():
    raise ValueError("Use one finite, uniformly sampled contiguous segment")
if not 0 < cutoff < fs / 2:
    raise ValueError("Cutoff must be below Nyquist")
sos = butter(4, cutoff, fs=fs, output="sos")
# The function rejects inputs too short for its padding requirement.
candidate = sosfiltfilt(sos, x)
# Offline only. Zero phase does not mean unchanged waveform or peak height.

Where it fails. A single cutoff is only right for one part of the record. Pick it for the quiet stretch and you smear the fast transients; pick it for the transients and you keep the noise. On non-stationary signals (a chirp, a load step, a vibration whose character changes) a low-pass can actually add error at high SNR — you would have been better off leaving the signal alone. A causal moving average also adds lag; filtfilt cancels the lag but only works offline on a full record.

Savitzky-Golay

Savitzky–Golay fits a local polynomial. Depending on window, degree and feature width it can attenuate, broaden or shift features; it does not guarantee peak preservation. Check the features and uncertainties relevant to the experiment.

from engineering_examples import savgol_candidate
candidate = savgol_candidate(t_seconds, x, window=51, degree=3)
# Requires at least 51 finite, uniformly sampled, contiguous measurements.
# Compare peak height, width, position and relevant integrals; no guarantee.

Where it fails. You still hand-pick window length and polynomial order, and the right pair depends on the local signal. Too wide a window flattens real features; too narrow keeps the noise. The moment the signal’s character changes mid-record, one setting is wrong somewhere.

Wavelet denoising

Wavelet shrinkage (e.g. VisuShrink) transforms the signal, thresholds small coefficients as noise, and inverts. It handles sharp features and some non-stationarity better than a fixed low-pass.

import numpy as np
import pywt
x = np.asarray(x, dtype=float)
if x.ndim != 1 or len(x) < 2 or not np.isfinite(x).all():
    raise ValueError("Use one finite contiguous segment")
wavelet = pywt.Wavelet("db4")
level = min(4, pywt.dwt_max_level(len(x), wavelet.dec_len))
if level < 1:
    raise ValueError("Segment too short for this wavelet example")
coeffs = pywt.wavedec(x, wavelet, level=level)
sigma = np.median(np.abs(coeffs[-1] - np.median(coeffs[-1]))) / 0.67448975
threshold = sigma * np.sqrt(2 * np.log(len(x)))
filtered = [coeffs[0]] + [pywt.threshold(c, threshold, "soft") for c in coeffs[1:]]
candidate = pywt.waverec(filtered, wavelet)[:len(x)]
# Noise estimate and threshold are assumptions; inspect edge and feature effects.

Where it fails. Wavelet and threshold choice matter, artefacts (ringing near discontinuities) appear if you over-threshold, and it is easy to get subtly wrong without checking the residual.

Kalman filter

A Kalman filter is optimal — if you have a state-space model of the system and its noise covariances. It is excellent for tracking and sensor fusion (an IMU on a moving platform, a battery equivalent circuit).

Where it fails. For most bench data you do not have a model, and building one per channel is a project. A poorly specified model quietly biases the estimate. Kalman is the right tool for known dynamics and the wrong tool for a generic noisy trace.

Quick reference

The common thread

Classical methods require assumptions or tuning choices. A fixed choice can work well within a stable regime and fail elsewhere. Compare methods within the same protocol and test the features that matter; visual smoothness alone is not an accuracy criterion.

FAQ

How do I remove noise from sensor data in Python?

Choose a method for the signal assumptions and latency budget. Butterworth, Savitzky–Golay and other methods can change real features; validate parameters on suitable data rather than choosing solely by filter name.

Which is better, Savitzky-Golay or a moving average?

Savitzky-Golay preserves peak height and width far better because it fits a local polynomial, so prefer it for spectra and peaks. A moving average is simpler but flattens features.

Why does my filter make the signal worse?

A fixed cutoff or window may not fit changing regimes. Compare candidate outputs with relevant reference measurements or held-out tests and inspect changes to task-specific features.