Back to Blog
IoT Engineering

IoT System Testing: Unit, Integration, Hardware-in-the-Loop, and End-to-End

Testing IoT systems is harder than testing web apps. The hardware is physical, the protocols are asynchronous, and bugs in production firmware require field visits to fix. Here is the full testing strategy.

July 20, 2024
14 min read
IoT TestingFirmware TestingHardware-in-the-LoopCI/CD

IoT System Testing: Unit, Integration, Hardware-in-the-Loop, and End-to-End

IoT system testing strategies differ from web application testing in ways that catch new teams off guard. You can't mock a temperature sensor reading that's coming from a faulty ADC. You can't replay a network partition at will when the failure mode is a physical cable. And a firmware bug that makes it to production requires OTA updates to every deployed device — if you're lucky — or field visits if you're not.

The consequence: IoT testing must be more rigorous, more automated, and planned earlier than most teams expect. This guide covers the full pyramid from firmware unit tests to chaos testing in production.

The IoT Testing Pyramid

                    ┌─────────────┐
                    │  Chaos/     │  (production, controlled)
                    │  Resilience │
                  ┌─┴─────────────┴─┐
                  │   End-to-End    │  (full stack, real devices)
                ┌─┴─────────────────┴─┐
                │ Hardware-in-the-    │  (real hardware, simulated cloud)
                │ Loop (HIL)          │
              ┌─┴─────────────────────┴─┐
              │   Integration           │  (MQTT mock, backend + DB)
            ┌─┴─────────────────────────┴─┐
            │   Unit (firmware + backend)   │  (fast, no hardware)
            └───────────────────────────────┘

Run everything below HIL in CI on every commit. HIL on every pull request merge. End-to-end nightly. Chaos testing weekly.

Layer 1: Firmware Unit Testing with Unity

Unity is the industry-standard C unit testing framework for embedded firmware. It runs on the target hardware or on a Linux host (using hardware abstraction layers to mock peripherals).

Structure your firmware so business logic is separated from hardware drivers — this is the essential prerequisite for unit testing:

// sensor_processor.h — pure business logic, no hardware calls
typedef struct {
  float temperature;
  float humidity;
  uint32_t timestamp;
} SensorReading;

typedef struct { float tempMin; float tempMax; float humidityMin; float humidityMax; } AlertThresholds;

// Returns true if reading is within expected bounds bool sensor_reading_is_valid(const SensorReading* reading);

// Returns alert bitmask (bit 0 = temp high, bit 1 = temp low, etc.) uint8_t sensor_check_alerts(const SensorReading* reading, const AlertThresholds* thresholds);

// Pack reading into MQTT payload — returns bytes written int sensor_reading_to_json(const SensorReading* reading, char* buf, size_t buf_size);

// test_sensor_processor.c — Unity tests, no hardware
#include "unity.h"
#include "sensor_processor.h"

void setUp(void) {} void tearDown(void) {}

void test_reading_validation_rejects_impossible_humidity(void) { SensorReading r = {.temperature = 25.0f, .humidity = 150.0f, .timestamp = 1000}; TEST_ASSERT_FALSE(sensor_reading_is_valid(&r)); }

void test_alert_fires_on_high_temperature(void) { SensorReading r = {.temperature = 85.0f, .humidity = 50.0f, .timestamp = 1000}; AlertThresholds t = {.tempMin = 0.0f, .tempMax = 80.0f, .humidityMin = 10.0f, .humidityMax = 90.0f}; uint8_t alerts = sensor_check_alerts(&r, &t); TEST_ASSERT_BIT_HIGH(0, alerts); // bit 0 = temp high }

void test_json_serialization_produces_valid_output(void) { SensorReading r = {.temperature = 22.5f, .humidity = 48.3f, .timestamp = 12345}; char buf[256]; int len = sensor_reading_to_json(&r, buf, sizeof(buf));

TEST_ASSERT_GREATER_THAN(0, len); TEST_ASSERT_NOT_NULL(strstr(buf, ""t":22.50")); TEST_ASSERT_NOT_NULL(strstr(buf, ""h":48.30")); }

int main(void) { UNITY_BEGIN(); RUN_TEST(test_reading_validation_rejects_impossible_humidity); RUN_TEST(test_alert_fires_on_high_temperature); RUN_TEST(test_json_serialization_produces_valid_output); return UNITY_END(); }

Run Unity tests in CI using a Linux build target — no hardware needed. We use CMake + GitHub Actions for this:

.github/workflows/firmware-test.yml

name: Firmware Unit Tests on: [push, pull_request]

jobs: firmware-unit-tests: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4

- name: Install build tools run: sudo apt-get install -y cmake gcc

- name: Build and run Unity tests run: | cmake -B build/test -DTARGET=host -DBUILD_TESTS=ON cmake --build build/test cd build/test && ctest --output-on-failure

Layer 2: Integration Testing with MQTT Mock Broker

Test the full message flow — firmware logic → MQTT → backend processing — without real hardware or a live cloud broker.

Mosquitto with a test config or the mqtt-mock library works well. We use Aedes (JavaScript MQTT broker) for integration tests because it runs in the same Node.js process as the backend:

// integration/test-helpers/mqtt-broker.ts
import Aedes from 'aedes'
import { createServer } from 'net'

export async function startTestBroker(port = 1883): Promise<{ broker: Aedes stop: () => Promise }> { const broker = new Aedes() const server = createServer(broker.handle)

await new Promise((resolve) => server.listen(port, resolve))

return { broker, stop: () => new Promise((resolve) => { broker.close(() => server.close(() => resolve())) }), } }

// integration/telemetry-pipeline.test.ts import { startTestBroker } from './test-helpers/mqtt-broker' import { startTelemetryProcessor } from '../src/telemetry-processor' import mqtt from 'mqtt'

describe('Telemetry pipeline integration', () => { let broker: Awaited> let processor: Awaited>

beforeAll(async () => { broker = await startTestBroker(1883) processor = await startTelemetryProcessor({ brokerUrl: 'mqtt://localhost:1883' }) })

afterAll(async () => { await processor.stop() await broker.stop() })

it('processes batched telemetry and stores all readings', async () => { const client = mqtt.connect('mqtt://localhost:1883') await new Promise((r) => client.on('connect', r))

const batch = { readings: [ { t: 22.5, h: 48.0, ts: Date.now() - 60000 }, { t: 22.8, h: 47.5, ts: Date.now() }, ], }

client.publish('devices/test-device-01/telemetry/batch', JSON.stringify(batch))

// Wait for async processing await new Promise((r) => setTimeout(r, 500))

const stored = await getStoredReadings('test-device-01') expect(stored).toHaveLength(2) expect(stored[0].temperature).toBeCloseTo(22.5)

client.end() }) })

Layer 3: Hardware-in-the-Loop (HIL)

HIL testing uses real hardware but simulated or instrumented signals. A test jig applies known inputs (simulated sensor voltages, injected CAN messages, scripted GPIO events) and verifies the device's MQTT output.

Key components of a HIL rig:

  • Signal injector: DAC or signal generator producing known voltages for ADC inputs
  • Network traffic monitor: Wireshark or tcpdump capturing MQTT traffic
  • Test controller: Raspberry Pi or similar running the test orchestration script
  • DUT (Device Under Test): Production hardware connected to the jig
  • hil/test_temperature_alert.py — runs on test controller (Raspberry Pi)

    import paho.mqtt.client as mqtt import RPi.GPIO as GPIO import time import pytest

    BROKER_HOST = "192.168.1.100" DEVICE_MQTT_ID = "device-under-test-001" DAC_CHANNEL = 0 # MCP4725 DAC for injecting voltage

    class HILTestSession: def __init__(self): self.received_alerts = [] self.client = mqtt.Client() self.client.on_message = self._on_message self.client.connect(BROKER_HOST, 1883) self.client.subscribe(f"devices/{DEVICE_MQTT_ID}/#") self.client.loop_start()

    def _on_message(self, client, userdata, msg): self.received_alerts.append({ 'topic': msg.topic, 'payload': msg.payload.decode(), 'timestamp': time.time() })

    def inject_temperature_voltage(self, celsius: float): # Convert celsius to DAC voltage for thermistor simulation voltage = temp_to_voltage(celsius) set_dac_voltage(DAC_CHANNEL, voltage)

    def test_high_temperature_alert_fires_within_30_seconds(): session = HILTestSession()

    # Inject normal temperature first session.inject_temperature_voltage(22.0) time.sleep(15)

    # Inject critically high temperature session.inject_temperature_voltage(92.0) start = time.time()

    # Wait for alert topic to appear timeout = 30 while time.time() - start < timeout: alerts = [m for m in session.received_alerts if 'alert' in m['topic']] if alerts: latency = time.time() - start assert latency < 30, f"Alert fired too late: {latency:.1f}s" return time.sleep(1)

    pytest.fail("No alert received within 30 seconds of temperature injection")

    HIL tests are slower to write but catch hardware-software integration bugs that no amount of unit testing finds: wrong ADC gain settings, off-by-one in ADC bit depth conversion, timer drift in sampling intervals.

    Layer 4: End-to-End and Chaos Testing

    End-to-end tests use production firmware, a staging cloud environment, and real devices. Run nightly with a test device fleet (we keep 5–10 devices per project in a permanent staging rack).

    Chaos testing verifies resilience. Deliberately induce failures and verify recovery:

    // chaos/network-partition-test.ts
    // Verify device reconnects and delivers queued messages after network outage

    async function testNetworkPartitionRecovery() { const deviceId = 'chaos-test-device-01' const messagesBefore: number[] = []

    // Collect baseline message count over 2 minutes await collectMessages(deviceId, 120_000, (msg) => messagesBefore.push(msg.ts))

    // Cut network to device for 5 minutes (via managed switch API or iptables on gateway) await setNetworkPartition(deviceId, 'isolated') console.log('Network partitioned — waiting 5 minutes') await sleep(5 * 60 * 1000)

    await setNetworkPartition(deviceId, 'connected') console.log('Network restored — waiting for recovery')

    // Device should reconnect within 60 seconds and deliver queued messages const messagesAfter: number[] = [] await collectMessages(deviceId, 120_000, (msg) => messagesAfter.push(msg.ts))

    // Verify: no gap in timestamps larger than the partition duration const sortedTs = [...messagesBefore, ...messagesAfter].sort() const maxGap = Math.max(...sortedTs.slice(1).map((ts, i) => ts - sortedTs[i]))

    console.assert(maxGap < 8 * 60 * 1000, Message gap too large: ${maxGap / 1000}s) console.log(Test passed. Max message gap: ${maxGap / 1000}s) }

    Other chaos scenarios worth automating:

  • MQTT broker restart (verify device reconnects with exponential backoff)
  • OTA update failure mid-flash (verify rollback to previous firmware)
  • Clock drift (verify NTP sync keeps timestamps valid)
  • Full flash write (verify graceful degradation when NVS is full)
  • CI/CD Pipeline for IoT

    Simplified CI pipeline

    stages: - firmware-unit-test # 2 min — runs on every commit - backend-unit-test # 2 min — runs on every commit - backend-integration # 5 min — runs on every commit - firmware-build # 8 min — cross-compile for ESP32 - hil-tests # 20 min — PR merge only, requires jig - staging-deploy # 5 min — deploy to staging environment - e2e-tests # 30 min — nightly, on real device fleet - chaos-tests # 90 min — weekly

    The HIL and E2E stages require physical lab infrastructure — self-hosted GitHub Actions runners connected to the test rig. Budget this into your project plan from the start; retrofitting a test rig after you've shipped firmware to 5,000 devices in the field is painful.

    IoT system testing strategies require more investment than web testing, but the asymmetry of failure costs makes it non-negotiable. A firmware bug caught in HIL takes 20 minutes to fix. The same bug in production takes an OTA release cycle, field team coordination, and customer communication.

    Need help? [Contact Code Caracal](/contact) — we've shipped these systems for clients across 15+ countries.

    Written by CodeCaracal Engineering

    We write from production experience — every technique in our articles has been deployed to real clients. No academic theory.

    More Articles

    Business · 12 min read

    IoT Device Compliance: FCC, CE, and Product Certification Guide for Hardware Startups

    Business · 11 min read

    What to Look for When Hiring an IoT Development Partner: 8 Critical Criteria

    Business · 11 min read

    IoT MVP to Production: Realistic Timeline and Budget for Hardware Startups

    Business · 11 min read

    IoT Development Agency vs Building In-House: A Decision Framework for Founders

    IoT Dashboard · 13 min read

    Next.js IoT Analytics Dashboard: From Sensor Data to Production App

    Business · 11 min read

    How Much Does It Cost to Build an IoT Product in 2024? A Realistic Breakdown

    IoT Dashboard · 11 min read

    IoT Dashboard UX: Design Principles for Industrial Monitoring Interfaces

    IoT Dashboard · 12 min read

    Node.js WebSocket Server: The Real-Time Backend for IoT Dashboards

    Cloud & DevOps · 12 min read

    Containerizing IoT Backend Services with Docker: From Dev to Production

    IoT Dashboard · 14 min read

    Grafana + InfluxDB IoT Monitoring: Complete Production Setup Guide

    IoT Dashboard · 12 min read

    Building Real-Time IoT Dashboards with React and Recharts

    Cloud & DevOps · 13 min read

    CI/CD for Embedded Firmware: Automated Build, Test, and OTA Release Pipeline

    Mobile Development · 12 min read

    Flutter Offline-First IoT Apps: Hive + Sync Architecture That Works in the Field

    Cloud & DevOps · 14 min read

    Terraform for IoT Infrastructure: Provisioning AWS IoT Core, Lambda, and InfluxDB as Code

    Mobile Development · 10 min read

    Flutter IoT Alerts: Firebase Push Notifications for Device Events

    Cloud & DevOps · 12 min read

    Deploying IoT Backends on AWS: ECS Fargate vs Lambda vs EC2 Decision Guide

    Mobile Development · 11 min read

    Flutter + MQTT: Building Production IoT Mobile Apps That Scale

    Mobile Development · 13 min read

    Flutter BLE: Building a Bluetooth IoT Controller App from Scratch

    Cloud & DevOps · 13 min read

    AWS IoT Core vs Azure IoT Hub vs Google Cloud IoT: 2024 Honest Comparison

    IoT Engineering · 13 min read

    Kafka vs RabbitMQ for IoT: Choosing the Right Message Queue for High-Volume Telemetry

    IoT Engineering · 14 min read

    Predictive Maintenance with IoT Sensor Data: From Threshold to Machine Learning

    Embedded Systems · 14 min read

    IoT Bootloader Design: Secure Boot, A/B Partitions, and Reliable OTA Recovery

    IoT Engineering · 14 min read

    Multi-Tenant IoT Platform Architecture: Isolation, Scaling, and Data Partitioning

    Embedded Systems · 14 min read

    Memory Management in Embedded Firmware: Avoiding Heap Fragmentation and Stack Overflows

    IoT Engineering · 13 min read

    IoT Cost Optimization: How We Cut AWS IoT Bills by 60% Without Sacrificing Reliability

    IoT Engineering · 12 min read

    Edge Computing in IoT: When to Process On-Device vs In the Cloud

    IoT Engineering · 13 min read

    Digital Twins for IoT: Building a Virtual Mirror of Your Physical Devices

    Embedded Systems · 14 min read

    ESP32 Deep Sleep Mastery: Cutting Power Consumption from 240mA to 10µA

    IoT Engineering · 10 min read

    MQTT QoS 0, 1, and 2 Explained: Choosing the Right Level for IoT

    IoT Engineering · 14 min read

    IoT Monitoring and Observability: Metrics, Logs, and Distributed Tracing

    Embedded Systems · 14 min read

    Debugging Embedded Firmware: JTAG, GDB, Logic Analyzers, and Serial Tracing

    IoT Engineering · 12 min read

    WebSocket vs MQTT vs Server-Sent Events: Real-Time IoT Protocol Deep Dive

    Embedded Systems · 13 min read

    STM32 HAL vs Low-Level Drivers: When the Abstraction Costs You Too Much

    IoT Engineering · 13 min read

    IoT Data Pipeline: From Raw Sensor Reading to Live Dashboard in Under 100ms

    IoT Engineering · 13 min read

    Zero-Touch IoT Device Provisioning: Scaling from 10 to 100,000 Devices

    Embedded Systems · 13 min read

    UART vs SPI vs I2C: Choosing the Right Protocol for Sensor Integration

    IoT Engineering · 12 min read

    Real-Time IoT Alerting: From Simple Thresholds to ML Anomaly Detection

    Embedded Systems · 12 min read

    ESP32 Partition Table: Designing Flash Layout for Production Firmware

    IoT Engineering · 12 min read

    IoT Architecture Patterns: Hub-and-Spoke, Mesh, and Edge-Cloud Hybrid

    Embedded Systems · 13 min read

    IoT Battery Life Optimization: Engineering Devices That Last Years on a Single Charge

    IoT Engineering · 13 min read

    Time-Series Databases for IoT: InfluxDB vs TimescaleDB vs AWS Timestream

    Security · 14 min read

    Zero-Trust Security for Embedded IoT: Why Your Devices Are Probably Vulnerable

    Embedded Systems · 14 min read

    FreeRTOS on ESP32: Task Scheduling, Queues, and Resource Management for IoT

    IoT Engineering · 12 min read

    Building a Production IoT Gateway with Raspberry Pi and Node.js

    Embedded Systems · 13 min read

    ESP32 vs STM32: Choosing the Right Microcontroller for Your IoT Project

    Mobile Development · 10 min read

    Flutter + WebSocket: Building Real-Time IoT Dashboards That Don't Stutter

    IoT Engineering · 13 min read

    IoT Fleet Management at Scale: AWS IoT Core Device Registry and Provisioning

    IoT Engineering · 11 min read

    MQTT vs HTTP for IoT: Which Protocol Wins in Production?

    IoT Engineering · 12 min read

    ESP32 → MQTT → AWS IoT Core: The Production-Grade Architecture Guide

    Let's Build Together

    Got an IoT challenge?
    We've shipped it.

    Whether you need a fleet to track, a factory to monitor, or a farm to automate — our team has done it before and we'd love to build it with you. Typical response time: under 24 hours.

    No upfront commitment99.9% uptime SLANDA on requestFixed-price options