Autonomous vehicle perception is not a single model. It is a layered stack of models  each responsible for a different stage of converting raw sensor data into the structured understanding of the environment that planning and control systems use to make driving decisions. Each layer in the stack has different training data requirements, and optimizing the training data for one layer without considering what the downstream layers need produces a system that performs well on individual model benchmarks but fails when the layers are integrated.

Understanding what each perception layer needs from its training data is the prerequisite for designing an AV perception model training data program that supports the full stack, not just its most visible components.

Layer 1: Sensor Preprocessing and Calibration Data

Before any perception model sees raw sensor data, that data passes through preprocessing stages: camera de-noising and exposure normalization, LiDAR intensity calibration and echo filtering, radar clutter suppression. These preprocessing stages are often implemented as learned models rather than hand-crafted algorithms  which means they also need training data.

Sensor preprocessing training data requires matched pairs: raw, unprocessed sensor readings paired with the “ideal” processed output the preprocessing should produce. For camera de-noising, this means pairs of noisy images (captured in adverse lighting) paired with clean reference images of the same scene captured under favorable conditions. For LiDAR calibration, it means measurements of known reference targets compared against the LiDAR’s raw returns from those targets.

The quality of preprocessing training data determines the signal-to-noise ratio that downstream perception models receive. Perception models trained on clean, well-preprocessed sensor data that then encounter poorly preprocessed data in deployment will produce degraded outputs not because the perception model is wrong but because the preprocessing layer delivered inputs outside the distribution the perception model trained on.

Layer 2: Object Detection and Classification

The most annotation-intensive layer of the stack. Object detection training data needs labeled examples of every object class the system needs to detect, across every sensor modality the detector will consume, across the full range of environmental conditions and viewing angles the deployed system will encounter.

What well-designed detection training data specifically requires:

• Class completeness: Every object class in the detection taxonomy needs annotated examples  not just the frequent classes (passenger vehicles, pedestrians) but the rare classes (agricultural vehicles, animals, unusual road users) that detection models trained on imbalanced data miss at unacceptable rates.
• Viewpoint coverage: The same object class appears differently from different viewing angles. A vehicle viewed head-on looks different from the same vehicle viewed obliquely from behind. Training data needs to cover the full range of relative viewing positions that the sensor placement generates.
• Scale coverage: Objects appear at different scales depending on their distance from the sensor. Near-field objects are large in the image and easy to detect; far-field objects are small and harder. Detection training data needs examples of each object class across its full operational distance range, with specific attention to the distant, small-scale instances where detection failure is most likely.
• Occlusion coverage: Objects in real traffic are frequently partially occluded  by other vehicles, by infrastructure, by vegetation. Training data with partially occluded objects, annotated with the visible portion only (or with the full estimated extent where appropriate for the annotation convention), teaches detection models to detect objects from partial evidence.

Layer 3: Multi-Object Tracking

Object detection answers “what is in this frame?” Object tracking answers “how do the objects in this frame relate to the objects in the previous frames?” Tracking connects detections across time, assigning consistent identities to objects and estimating their state trajectory.

Tracking model training data requires labeled detection sequences  not individual frames but temporally ordered sequences of frames with consistent object identities maintained across the sequence. The specific training data requirements that distinguish tracking from detection:

Track ID assignment: Every object in the sequence needs a consistent unique identifier from its first appearance to its last. Tracking models learn how to maintain object identity from these annotated examples  including through occlusion events where the object temporarily disappears and must be re-associated on reappearance.

State estimation: Beyond the detection bounding box, tracking training data needs velocity and acceleration labels that let the model learn to estimate object dynamics from the sequence of positions rather than only current position. Some tracking training data programs include IMU-derived ground truth velocity for instrumented reference vehicles that appear in the training data, providing physically accurate velocity labels that improve the model’s velocity estimation precision.

Association failure cases: Training data that includes scenarios where data association is genuinely ambiguous  where multiple objects could be the match for a reappearing detection, where objects cross paths and their identities become temporarily indistinguishable  teaches the tracking model how to handle the hard association cases rather than only the easy ones where identity is unambiguous.

Layer 4: Trajectory Prediction

Trajectory prediction models estimate where objects will be in the near future  typically 1–5 seconds ahead  to support collision avoidance and path planning. This is arguably the hardest AV Perception Model Training Data in autonomous driving because it requires the model to go beyond “what is the object doing now?” to “what is the object likely to do next?”

Trajectory prediction training data differs qualitatively from detection and tracking training data because it requires outcome labels that cannot be annotated from the current frame; the label is what actually happened in the subsequent seconds.

What trajectory prediction training data requires:

• Complete trajectory sequences: Annotated sequences that include both the historical trajectory (the frames leading up to the prediction point) and the future trajectory (the frames following the prediction point) for each tracked object. The future trajectory is the ground truth the model trains against.
• Interaction context: Trajectory prediction accuracy depends on modeling how objects interact  a pedestrian’s trajectory at a crosswalk depends on whether vehicles are stopping or continuing. Training data needs to include the full spatial context of interactions, with all relevant objects annotated, not just the object whose trajectory is being predicted.
• Intent labels: For pedestrians and cyclists, crossing intent labels (is this person about to cross the road?) provide a high-level behavioral label that trajectory prediction models can use as an intermediate representation. These intent labels require annotators to make behavioral judgments  assessing from the sequence of poses and positions whether the pedestrian is committed to crossing or is still ambiguous  that are more demanding than simple positional annotation.
• Outcome diversity: Trajectory prediction models need training data that covers the full diversity of outcomes that can follow from similar initial conditions. A pedestrian hesitating at the curb might cross, might turn back, or might continue waiting. Training data that only includes one outcome type for each initial condition teaches the model to make deterministic predictions rather than probabilistic ones.

Layer 5: Sensor Fusion

Sensor fusion models combine the outputs of camera, LiDAR, and radar detectors into a unified environmental model that is more reliable than any single sensor’s output. Training fusion models requires training data where all sensor modalities are annotated consistently with the same object labeled with the same class, the same ID, and spatially consistent extent across all sensor streams.

Cross-modal consistency is the defining training data requirement for fusion models. When the camera detector sees “pedestrian” and the LiDAR detector sees the same physical person labeled as “unknown person” with a different object ID, the fusion model receives contradictory inputs that it cannot reconcile into a coherent unified detection. Training data where cross-modal annotation consistency was enforced  through unified taxonomy, synchronized timestamps, and cross-modal quality checks  teaches the fusion model correct multi-sensor associations.

Sensor degradation scenarios: Fusion models provide their most significant safety value in the scenarios where one or more sensors degrade  heavy rain that reduces camera clarity, precipitation that attenuates LiDAR range, radar clutter from bridges. Training data that includes these sensor degradation scenarios, with ground truth annotations that the fusion model should produce from the degraded sensor inputs, teaches the model to rely appropriately on the sensors that are performing well when others are degraded.

Final Thought

AV perception model training data is not a single dataset, it is a hierarchy of datasets, each designed for the specific learning task of one layer in the perception stack. Programs that design training data for the stack as a whole  ensuring that each layer’s training data supports the interface between that layer and the downstream layers  produce perception systems with coherent integrated behavior. Programs that optimize each layer’s training data in isolation produce perception systems that perform well on individual model benchmarks and fail in integration.