Edge to Edge: Enabling Comparative Benchmarking for on-Device Training with a TensorFlow Lite Baseline

Adapting neural networks to compensate for real-world data drift is a common practice: networks or their sub components are typically trained on new samples, either by receiving updated weights externally or by retraining locally on the device. Multiple training frameworks that aim to enable training on such highly resource-limited devices exist, among them TensorFlow Lite. Evaluating these existing frameworks in terms of on-device training capabilities is a challenge, as they are all evaluated for different purposes against different baselines, making a concise comparison difficult. In this paper, we propose a benchmarking process that enables the gathering of comparable metrics for on-device training frameworks, specifically for computer vision tasks, and demonstrate this process on the TensorFlow Lite framework. The metrics are the memory and performance requirements during training, an approximation of the maximum achievable accuracy, and an accuracy-training-time curve. All metrics of this process are designed to be hardware-agnostic and give insight into the real-world impact of these frameworks. Our baseline evaluation on a Raspberry Pi 4B with VGG16 and MobileNetV2 in two transfer scenarios illustrates the viability of our metrics by establishing a baseline using the TensorFlow Lite framework. In the evaluation we showcased the differences of our chosen architectures, with MobileNetV2 requiring more than 17 times fewer instructions and up to 2.6 times less training memory than VGG16, while achieving comparable maximum accuracy. We release benchmark tooling and pretrained weights to enable a reproducible and comparative evaluation of other on-device training methods.