- State change detection
- Anomaly detection
Unknown data detection technology for monitoring deep learning models
Accurately and memory-efficiently detects unknown data without reference to training data.
- Can be used in monitoring deterioration of model performance based on properties of data input into deep learning models.
- Enables memory-efficient unknown data detection by using only weight parameters in deep learning models without reference to training data.
- Reduced memory requirement to about 1/70th, while maintaining detection performance equal to or greater than conventional methods in MNIST, CIFAR-100, and SVHN datasets.
Applications
- Detects deterioration of model performance on MLOps platforms
- Detects deterioration of model performance on edge devices due to memory efficiency
Benchmarks, strengths, and track record
Detection performance of unknown data evaluated on various open image datasets, and additional memory volumes required for evaluation
(Evaluated detection performance of MNIST, SVHN, and CIFAR-100 using several models trained on CIFAR-10; performance values in the table are average values for VGG-13, VGG-16, and ResNet-18 models)
| Unknown data detection using prediction probabilities (conventional technology) |
Unknown data detection using Mahalanobis distance (conventional technology) | Proposed method | ||
| Performance (AUROC) | MNIST | 93.3 | 96.5 | 99.1 |
| SVHN | 94.9 | 96.2 | 97.0 | |
| CIFAR-100 | 88.8 | 91.6 | 91.8 | |
| Additional memory requirement | 0.00GB | 3.50GB | 0.05GB | |
Inquiries
Contact the Toshiba Corporate Research & Development Center
Please include the title “Toshiba AI Technology Catalog: Unknown data detection technology for monitoring deep learning models” or the URL in the inquiry text.
Please note that because this technology is currently the subject of R&D activities, immediate responses to inquiries may not be possible.
