Toby Smith | Machine Learning Engineer, Mindtech Global
In Part 1, we explored how collecting and then clustering false positive detections of a detector model could reveal critical gaps in training datasets for object detection. By analyzing misclassifications on real-world scenes, we identified specific object types missing from our synthetic dataset, which led to recurring false positives. These false positives occur because the model has not learned to ignore certain visual patterns that appear in the real-world. In this second part, we apply those insights by enhancing the synthetic dataset with the missing objects and measuring the impact on model performance. This targeted approach shows how refining synthetic datasets can help models perform more reliably in real-world scenarios.
From Clustering Insights to Dataset Enhancement
The initial dataset, which we’ll call Baseline, was formed from synthetic data generated by the Mindtech Chameleon simulator. The task here was to detect only vehicles (1 class) in the real-world Audi A2D2 dataset, using synthetic data generated from the chameleon platform. Removing false positives in this case for example, would help a self-driving car to avoid making unsafe decisions. However, when the model trained on the baseline dataset was tested on real-world data, it struggled with certain misclassifications, often mistaking objects like bushes, bicycles, and road signs with vehicles. Clustering analysis of these false positives confirmed that the absence of these objects in the Baseline synthetic dataset was a primary factor contributing to detection errors.
Creating the Enhanced Dataset
In response to the clustering insights, we developed an Enhanced Dataset by adding the missing object types — bicycles, foliage, road signs, and cones — into the synthetic scenes. This adjustment aimed to make the dataset more representative of real-world driving environments, giving the model a chance to learn the appearance of these frequently misidentified objects. With the Enhanced Dataset we hoped the model would better differentiate between objects that it previously struggled with, reducing false positives when encountering similar scenes in real-world data.
Results After Training on the Enhanced Dataset
After training the model on the Enhanced Dataset, we observed significant improvements in detection precision. Specifically, the model exhibited fewer false positives, particularly in images that had previously contained misclassification.
In the first example, while the false positive detection of the bike improved, the model merged the detections of the vehicles in the background into a single detection. In the second example, performance was better overall: the bush in the centre, previously detected as a false positive, was correctly ignored, and the vehicles on the right were detected more accurately.
This example shows cases where the added objects didn’t lead to improvements. In the first image, a sign is still mistakenly detected as a vehicle, likely because this is a billboard that wasn’t represented in the synthetic data, making it an edge case. This is another why it is important to have detailed clustering of false positives so that these cases can be spotted early on. In the second image, two bush planters are misclassified as vehicles, with only one corrected after the dataset enhancement. This represents another edge case, as this specific type of bush wasn’t included in the synthetic data, highlighting an area for future improvement by adding similar objects to prevent such false positives.
Quantitative Results: Precision Improvements
The Enhanced Dataset led to a substantial increase in model precision, a measure of how accurately a model identifies true positives, with higher precision indicating fewer false positives among the detected objects. This indicated a 11.7% gain in precision over the Baseline dataset. Importantly, this increase in precision was achieved without any reduction in recall, leading to higher mean Average Precision (mAP) scores overall. Recall measures a model’s ability to detect all relevant objects, focusing on minimizing missed detections. It contributes to mAP, which averages precision across recall levels, providing a balanced view of the model’s accuracy in detecting objects consistently.
Comparison of precision, recall, and mAP, showing an 11.7% precision gain with the Enhanced Dataset
These quantitative gains underscore the importance of strategically enhancing synthetic datasets by adding missing objects. With the Enhanced Dataset, the model was better equipped to navigate complex real-world scenes, yielding improved accuracy across varied scenarios. Future improvements to this dataset involves simply importing new assets into our Chameleon platform, adding them to the scenario, and re-running the simulation to get a new dataset.
Conclusion
This study highlights the potential of synthetic data refinement for improving object detection models. By analyzing clustering results and addressing key data gaps, we created a more precise, real-world-ready model. The Enhanced Dataset, enriched with challenging objects, illustrates how even a single targeted adjustment to synthetic data can substantially improve model performance in the field.
For machine learning practitioners and teams, this case study highlights the value of data-driven synthetic enhancements in building reliable AI solutions. Having an on-demand tool like Chameleon to create new data, rather than relying on a fixed dataset and needing to request updates from a supplier, empowers teams to quickly address model weaknesses. This approach not only streamlines the analysis of false positives but also enables rapid, targeted dataset improvements, leading to more robust real-world performance.
