Anomaly Detection in AR Environments

Project Overview

Our project, Anomaly Detection in Augmented Reality Environments, explores the intersection of machine learning and synthetic data generation. We developed a real-time anomaly detection system capable of identifying irregularities in physical environments through augmented reality interfaces. By combining Unity3D's powerful simulation tools with TensorFlow's efficient modeling capabilities, we demonstrate how lightweight deep learning models trained on synthetic datasets can be deployed for real-world anomaly detection across industries like manufacturing, logistics, and quality control.

Dataset Generation in Unity 3D

To build a robust anomaly detection model, we first created a fully synthetic dataset using Unity3D. Leveraging Unity’s Perception package, we generated thousands of labeled images across a wide range of scenarios. We introduced randomized object placements, dynamic lighting conditions, different surface textures, and background variations to simulate diverse real-world environments. By automating the dataset generation process, we avoided the time-consuming effort of manual data collection and ensured that our model would be trained on a highly diverse and scalable synthetic dataset, greatly improving its ability to generalize to unseen environments.

TensorFlow Training Model

For our anomaly detection pipeline, we designed a hybrid model architecture that combines a lightweight EfficientNetB0 feature extractor with a custom-trained autoencoder. The EfficientNetB0 backbone allowed us to maintain high-quality feature representations while keeping the model size optimized for real-time performance. We chose TensorFlow due to its mature ecosystem, deployment flexibility, and excellent support for real-time inference scenarios. By integrating a feature extraction layer before the autoencoder, we ensured that the model could focus on learning meaningful patterns in the environment, resulting in significantly faster convergence during training and improved generalization to unseen environments. This strategic combination allowed us to achieve robust anomaly detection capabilities without relying on excessively large or complex architectures.

How We Trained the Model

Our training strategy was carefully designed to maximize the model's ability to recognize anomalies without relying on labeled defect data. We trained exclusively on normal samples, allowing the autoencoder to learn a compressed latent representation of expected environments. Instead of using traditional loss functions like Mean Squared Error (MSE), we implemented Structural Similarity Index (SSIM) loss, which prioritizes the preservation of structural details and visual consistency, leading to more reliable anomaly detection. To further improve model robustness, we employed extensive data augmentation techniques, including random rotations, noise injection, brightness shifts, and geometric transformations. These augmentations simulated real-world variability and made the model significantly more resilient to changes in lighting, perspective, and scene composition. This methodology enabled our model to achieve high precision in identifying subtle deviations without ever seeing defective examples during training.

User Interface Design

To deliver real-time anomaly feedback in an intuitive and user-friendly way, we developed a custom interface using OpenCV in Python. The interface captures a live camera feed and overlays an anomaly heatmap generated by the model, highlighting deviations directly on screen. Our priority was to ensure minimal latency between camera input and model output, making the system feel immediate and responsive. We also implemented a dynamic thresholding system to control sensitivity, allowing users to fine-tune the detection based on environmental noise. The UI was intentionally designed with simplicity and performance in mind, avoiding unnecessary complexity while maintaining a smooth, real-time experience even on mid-range hardware. This user-centric approach made the system not only functional for technical users, but also accessible to non-experts needing fast, visual anomaly insights.

Live Results

Live testing demonstrated the system's ability to detect anomalies reliably across both synthetic and real-world environments. The model consistently identified unexpected objects, lighting changes, and scene deviations that differed from its learned normal patterns. We observed detection precision rates exceeding 90% on synthetic test scenes, and similarly strong performance in real-world applications, even under variable lighting conditions and background noise. The real-time heatmaps provided clear, interpretable feedback to users, highlighting regions of concern without overwhelming the interface. System latency remained consistently low, preserving the feeling of immediate responsiveness that was crucial for real-time applications. These results validated both our model architecture and our dataset generation strategy, confirming the effectiveness of combining synthetic data with lightweight deep learning models for practical anomaly detection use cases.

Final Evaluation

Our final evaluation confirmed that combining synthetic dataset generation with a lightweight deep learning architecture offers a highly effective approach to real-time anomaly detection. The system achieved over 90% precision across diverse test scenarios, including varying lighting conditions, occlusions, and environmental changes. Key lessons included the importance of diverse and realistic synthetic data, the benefits of using SSIM loss for visual tasks, and the advantages of compact models like EfficientNetB0 for maintaining real-time performance. Our results demonstrated that sophisticated anomaly detection does not require complex or heavy architectures when data quality and training methodology are carefully considered. This project highlights the practical potential of applying machine learning and augmented reality together to build accessible, responsive, and impactful real-world AI systems.

Anomaly Detection in Augmented Reality

Project Overview

Dataset Generation in Unity 3D

TensorFlow Training Model

How We Trained the Model

User Interface Design

Live Results

Final Evaluation

Collaborators

Syed Ammar Ali

Usba Gohir

Abdul Aziz Ibrahim