Embedded Deep Learning
(2017-Present)

Deep learning technologies are at the core of the current AI revolution. The affordable GPU hardware and large annotated datasets jointly allow the training of deep learning models with hundreds of layers and millions of parameters. The deep learning-enabled breakthroughs result in great successes in numerous AI research fields. However, deep learning models typically have severe demands on the computing resources of devices, which makes leveraging the power of deep learning on resource-constrained platforms or in real-time applications infeasible. The goal of this research direction is to allow embedded deep neural networks to be easily deployed and efficiently executed on low-end devices without powerful GPUs and vendor-provided libraries so that AI researchers can focus merely on deep algorithm design rather than computation parallelization.

Representative Publications

Self-Powered Intermittent IoT Systems
(2016-Present)

The development of technologies for use in IoT devices has recently accelerated significantly. In particular, limited battery life requires frequent recharging, raising interest in energy harvesting as a promising power source and means of improving usability. There are various energy sources, including the use of body motion, vibration, piezoelectric materials, and thermal, solar and wireless sources, but they are all inherently unstable, resulting in performance and reliability challenges to traditional processors. Non-volatile processors have emerged as a promising technology for self-powered IoT devices. To realize the benefits of this technology, several circuit-level configurations and architectural-level designs have been evaluated and explored to maximize forward progress of running applications. However, there remains comparatively little research on system-level software supports. The goal of this research direction is to develop a lightweight operating system intended for self-powered intermittent IoT systems that can survive unstable power environments.

Representative Publications

User-Centric Mobile Systems
(2013-2018)

The personal computing landscape is undergoing a massive transition from stationary desktops to mobile devices. Emerging hardware technologies, such as OLED displays to replace traditional LCD displays and big.LITTLE processors that allows mobile devices to combine computing performance and energy efficiency, have been developed to reflect the paradigm shift. However, existing “hasty software solutions” that borrow legacy designs directly from conventional operating systems cannot be applied seamlessly to mobile systems. For example, resource management in Android, which is built upon Linux, often overlooks a key factor – the user’s perception. We advocate that mobile systems should move toward user-centric designs that consider human factors. To this end, we tried to understand the interplay among end users, mobile applications, and system resources. Understanding the implications has facilitated better system resource management.

In particular, we exploited application sensitivity, which reflects the user’s perception, to redesign the CPU scheduler so that user experience and energy efficiency can be improved simultaneously. We have incorporated human visual attention and acuity into the quality-retaining power-saving design for mobile OLED displays. Besides, we introduced the concept of alarm similarity into wakeup management for mobile systems in connected standby, with the objective of reducing energy waste while maintaining the quality of the user experience. We have also developed an augmented-sensing framework for mobile systems so that smartphones can easily mount sensors on wearables to create multi-device applications.

Representative Publications

Cloud-Based Energy Saving Services
(2009-2014)

To meet the growing demand for more creative applications, next-generation mobile devices will provide a significant improvement in terms of performance; yet, ironically, the trend will raise a further challenge in energy consumption. Low-power system designs from only the perspective of devices may not be sufficient for the emerging genre of mobile applications. To address the challenge, we proposed the concept of cloud-based energy-saving services. A possible way to realize the concept would be to develop energy saving as a value-added service offered by Internet service providers. Then, mobile users could apply for the energy-saving service in the same way that they apply for other value-added services. The service should be presented in a way that is easy for end users to understand. They do not need to know how the service is provided and where the system that delivers the service is located. The service provider helps reduce the energy consumption of mobile devices when they access Internet applications.

To prove the concept, we deploy an exemplary service called the dynamic backlight scaling service on Chunghwa Telecom’s hicloud. The service minimizes the energy consumption of the screen backlight when displaying a video stream without adversely impacting the user’s visual experience. With the service subscribed, HTC Desire smartphones and Apple iPad tablets can achieve an energy reduction of 10 to 40% when browsing video streams on YouTube. This work was published in IEEE Trans. on Computers 2014 and received the 2015 Y. Z. Hsu Scientific Paper Award (up to two papers per year in the ICT field of Taiwan). Another exemplary service deployed is called the data fetch scheduling service, which can save the communication energy by 35 to 70% by scheduling the fetching locations of data based on signal strength without changing a local-based application’s semantics. We also performed postoptimal analysis to explore how the service responds to signal strength fluctuations. This work, which was published in IEEE/ACM Trans. on Networking 2015, received the 2017 Young Scholar Best Paper Award from IEEE ITSOC and IEEE COMSOC Taipei/Tainan Chapters (one paper selected by the two societies per year in Taiwan).

Representative Publications

Intermittent-Aware Neural Network Pruning

Deep neural network inference on energy harvesting tiny de- vices has emerged as a solution for sustainable edge intelligence. However, compact models optimized for continuously-powered systems may become suboptimal when deployed on intermittently-powered systems. This paper presents the pruning criterion, pruning strategy, and prototype imple- mentation of iPrune, the first framework which introduces intermittency into neural network pruning to produce compact models adaptable to intermittent systems. The pruned models are deployed and evaluated on a Texas Instruments device with various power strengths and TinyML applications. Compared to an energy-aware pruning framework, iPrune can speed up intermittent inference by 1.1 to 2 times while achieving comparable model accuracy.

• [Conference] [Demo]

Intermittent-aware Distributed Protocol

Intermittent-aware Distributed Concurrency Control

Internet-of-Things (IoT) devices are gradually adopting battery-less, energy harvesting solutions, thereby driving the development of an intermittent computing paradigm to accumulate computation progress across multiple power cycles. While many attempts have been made to enable standalone intermittent systems, little attention has focused on IoT networks formed by intermittent devices. We observe that the computation progress improved by distributed task concurrency in an intermittent network can be significantly offset by data unavailability due to frequent system failures. This work presents an intermittent-aware distributed concurrency control protocol which leverages existing data copies inherently created in the network to improve the computation progress of concurrently executed tasks. In particular, we propose a borrowing-based data management method to increase data availability and an intermittent two-phase commit procedure incorporated with distributed backward validation to ensure data consistency in the network. The proposed protocol was integrated into a FreeRTOS-extended intermittent operating system running on Texas Instruments devices. Experimental results show that the computation progress can be significantly improved, and this improvement is more apparent under weaker power, where more devices will remain offline for longer duration.

• [Journal] [Demo]

Stateful Neural Network

Stateful Neural Networks for Intermittent Systems

Deep neural network (DNN) inference on intermittently-powered battery-less devices has the potential to unlock new possibilities for sustainable and intelligent edge applications. Existing intermittent inference approaches preserve progress information separate from the computed output features during inference. However, we observe that even in highly specialized approaches, the additional overhead incurred for inference progress preservation still accounts for a significant portion of the inference latency. This work proposes the concept of stateful neural networks, which enables a DNN to indicate the inference progress itself. Our runtime middleware embeds state information into the DNN such that the computed and preserved output features intrinsically contain progress indicators, avoiding the need to preserve them separately. The specific position and representation of the embedded states jointly ensure both output features and states are not corrupted while maintaining model accuracy, and the embedded states allow the latest output feature to be determined, enabling correct inference recovery upon power resumption. Evaluations were conducted on different Texas Instruments devices under varied intermittent power strengths and network models. Compared to the state of the art, our approach can speed up intermittent inference by 1.3 to 5 times, achieving higher performance when executing modern convolutional networks with weaker power

• [Journal] [Demo]

JAPARI

More is Less: Model Augmentation for Intermittent Deep Inference

Energy harvesting creates an emerging intermittent computing paradigm, but poses new challenges for sophisticated applications such as intermittent deep neural network (DNN) inference. Although model compression has adapted DNNs to resource constrained devices, under intermittent power, compressed models will still experience multiple power failures during a single inference. Footprint-based approaches enable hardware accelerated intermittent DNN inference by tracking footprints, independent of model computations, to indicate accelerator progress across power cycles. However, we observe that the extra overhead required to preserve progress indicators can severely offset the computation progress accumulated by intermittent DNN inference. This work proposes the concept of model augmentation to adapt DNNs to intermittent devices. Our middleware stack, JAPARI, appends extra neural network components into a given DNN, to enable the accelerator to intrinsically integrate progress indicators into the inference process, without affecting model accuracy. Their specific positions allow progress indicator preservation to be piggybacked onto output feature preservation to amortize the extra overhead, and their assigned values ensure uniquely distinguishable progress indicators for correct inference recovery upon power resumption. Evaluations on a Texas Instruments device under various DNN models, capacitor sizes, and progress preservation granularities, show that JAPARI can speed up intermittent DNN inference by 3x over the state of the art, for common convolutional neural architectures that require heavy acceleration.

• [Journal] [Demo]

iNAS

Intermittent-Aware Neural Architecture Search

The increasing paradigm shift towards intermittent computing has made it possible to intermittently execute deep neural network (DNN) inference on edge devices powered by ambient energy. Recently, neural architecture search (NAS) techniques have achieved great success in automatically finding DNNs with high accuracy and low inference latency on the deployed hardware. We make a key observation, where NAS attempts to improve inference latency by primarily maximizing data reuse, but the derived solutions when deployed on intermittently-powered systems may be inefficient, such that the inference may not satisfy an end-to-end latency requirement and, more seriously, they may be unsafe given an insufficient energy budget. This work proposes iNAS, which introduces intermittent execution behavior into NAS to find accurate network architectures with corresponding execution designs, which can safely and efficiently execute under intermittent power. An intermittent-aware execution design explorer is presented, which finds the right balance between data reuse and the costs related to intermittent inference, and incorporates a preservation design search space into NAS, while ensuring the power-cycle energy budget is not exceeded. To assess an intermittent execution design, an intermittent-aware abstract performance model is presented, which formulates the key costs related to progress preservation and recovery during intermittent inference. We implement iNAS on top of an existing NAS framework and evaluate their respective solutions found for various datasets, energy budgets and latency requirements, on a Texas Instruments device. Compared to those NAS solutions that can safely complete the inference, the iNAS solutions reduce the intermittent inference latency by 60% on average while achieving comparable accuracy, with an average 7% increase in search overhead.

• [Journal] [Demo]

Intermittent Multicore Synchronization

Heterogeneity-aware Multicore Synchronization for Intermittent Systems

Intermittent systems enable batteryless devices to operate through energy harvesting by leveraging the complementary characteristics of volatile (VM) and non-volatile memory (NVM). Unfortunately, alternate and frequent accesses to heterogeneous memories for accumulative execution across power cycles can significantly hinder computation progress. The progress impediment is mainly due to more CPU time being wasted for slow NVM accesses than for fast VM accesses. This work explores how to leverage heterogeneous cores to mitigate the progress impediment caused by heterogeneous memories. In particular, a delegable and adaptive synchronization protocol is proposed to allow memory accesses to be delegated between cores and to dynamically adapt to diverse memory access latency. Moreover, our design guarantees task serializability across multiple cores and maintains data consistency despite frequent power failures. We integrated our design into FreeRTOS running on a Cypress device featuring heterogeneous dual cores and hybrid memories. Experimental results show that, compared to recent approaches that assume single-core intermittent systems, our design can improve computation progress at least 1.8x and even up to 33.9x by leveraging core heterogeneity

• [Journal] [Demo]

HAWAII

Everything Leaves Footprints: Hardware Accelerated Intermittent Deep Inference

Current peripheral execution approaches for intermittently-powered systems require full access to the internal hardware state for checkpointing or rely on application-level energy estimation for task partitioning to make correct forward progress. Both requirements present significant practical challenges for energy-harvesting, intelligent edge IoT devices, which perform hardware accelerated DNN inference. Sophisticated compute peripherals may have inaccessible internal state, and the complexity of DNN models makes it difficult for programmers to partition the application into suitably sized tasks that fit within an estimated energy budget. This paper presents the concept of inference footprinting for intermittent DNN inference, where accelerator progress is accumulatively preserved across power cycles. Our middleware stack, HAWAII, tracks and restores inference footprints efficiently and transparently to make inference forward progress, without requiring access to the accelerator internal state and application-level energy estimation. Evaluations were carried out on a Texas Instruments device, under varied energy budgets and network workloads. Compared to a variety of task-based intermittent approaches, HAWAII improves the inference throughput by 5.7% to 95.7%, particularly achieving higher performance on heavily accelerated DNNs

• [Journal] [Demo]

Intermittent OS

Enabling Failure-resilient Intermittent Systems Without Runtime Checkpointing

This project develops an intermittent operating system (OS) to preserve computation progress across power cycles without checkpointing for energy-harvesting systems. The intermittent OS is endowed with several capabilities as follows: 1) Run multiple tasks concurrently to improve computation progress. 2) Achieve consistency between data and computing progress. 3) Recover the system instantly from power failures. 4) Accumulatively preserve computation progress across power cycles to avoid progress stagnation. The intermittent OS is built upon FreeRTOS, a real-time operating system supporting many kinds of commercial microcontrollers, running on MSP-EXP430FR5994 LaunchPad, a Texas Instruments platform featuring 256KB FRAM and 8KB on-chip SRAM. We add a data manager and a recovery handler in FreeRTOS, so that the system runtime can cope with intermittence and exempts application developers from this responsibility. Due to the limitation of the memory size, the current implementation supports up to 10 user tasks and 16 data objects.

• [Journal] [Demo]

Accumulative Display Updating

Accumulative Display Updating for Intermittent Systems

Electrophoretic displays (EPDS) are ideal for self-powered systems, but currently require an uninterrupted power supply to carry out the full display update cycle. Conventional self-powered EPDs will either re-execute the update process if it failed in the previous power cycle or incur significant HW/SW/programmer overheads and energy buffering latency to perform an uninterrupted display update, and hence is not suitable for intermittently-powered systems. Our Accumulative Display Updating (ADU) technique allows the EPD update to be completed across power cycles, without loosing update progress state due to power failure and removes the need for sufficient energy for the entire display update. We also propose a context-aware updating policy (denoted as ADU+F) to handle new data that becomes available to the system during an ongoing display update. Experimental results on a Texas Instruments device (MSP430FR5994) with an integrated 1.44 inch EPD by Pervasive Displays Inc., show that, compared to re-execution based atomic display updating (denoted as DU), our design can increase accurate forward progress of display updating by up to 31% and using the data freshness handling updating policy, further average response time reductions of up to 47.4% can be achieved.

• [Journal] [Demo]

Oasis

A Mobile Cyber-Physical System for Accessible Location Exploration

Users roaming cellular signal coverage with their mobile devices essentially form a mobile cyber-physical system (CPS). By modeling cyber human mentality and physical signal coverage, as well as their interplay, user mobility can be leveraged to improve users' mobile experience with limited wireless bandwidth. Through a real-world case study, we observed that numerous "null zones" and "hot zones" exist in cellular signal coverage areas, where mobile devices cannot obtain sufficiently high data rates for delay-sensitive applications. Over one third of the locations in a crowded area could have weak signal coverage and low bandwidth shares, resulting in poor mobile connectivity experience. This paper considers the practicality of a mobile CPS called Oasis, which guides users to leave those zones and move to nearby locations with better mobile experience. To realize the system, we model and maximize a user's willingness to travel to another location, where the willingness involves the compound impact of the travel distance and the improved perceptual quality. We also develop a prototype system that creates a feedback control loop to allow self-adaptation to users' needs. To evaluate the efficacy, we conducted a series of experiments based on the real data collected in downtown Taipei, Taiwan. The results demonstrate that our mobile CPS can further reduce the average distance per unit of quality improvement achieved with OpenSignalMaps by about 80%, and motivate further research.

• [Journal] [Demo]

HomeRun

HW/SW Co-Design for Program Atomicity on Self-Powered Intermittent Systems

Self-powered intermittent systems featuring nonvolatile proces- sors (NVPs) allow for accumulative execution in unstable power environments. However, frequent power failures may cause incor- rect NVP execution results due to invalid data generated intermit- tently. This paper presents a HW/SW co-design, called HomeRun, to guarantee atomicity by ensuring that an uninterruptible pro- gram section can be run through at one execution. We design a HW module to ensure that a power pulse is sufficient for an atomic section, and develop a SW mechanism for programmers to protect atomic sections. The proposed design is validated through the development of a prototype pattern locking system. Experimental results demonstrate that the proposed design can completely guar- antee atomicity and significantly improve the energy utilization of self-powered intermittent systems.

• [Conference] [Demo]

Adaptive Hidden Layers

Learning Adaptive Hidden Layers for Mobile Gesture Recognition

This paper addresses two obstacles hindering advances in accurate gesture recognition on mobile devices. First, gesture recognition performance is highly dependant on feature selection, but optimal features typically vary from gesture to gesture. Second, diverse user behaviors and mobile environments result in extremely large intra-class variations. We tackle these issues by introducing a new network layer, called an adaptive hidden layer (AHL), to generalize a hidden layer in deep neural networks and dynamically generate an activation map conditioned on the input. To this end, an AHL is composed of multiple neuron groups and an extra selector. The former compiles multi-modal features captured by mobile sensors, while the latter adaptively picks a plausible group for each input sample. The AHL is end-to-end trainable and can generalize an arbitrary subset of hidden layers. Through a series of AHLs, the great expressive power from exponentially many forward paths allows us to choose proper multi-modal features in a sample-specific fashion and resolve the problems caused by the unfavorable variations in mobile gesture recognition.The proposed approach is evaluated on a benchmark for gesture recognition and a newly collected dataset. Superior performance demonstrates its effectiveness.

• [Conference] [Poster] [Demo] [Web]

Shift Mask

Dynamic OLED Power Shifting Based on Visual Acuity for Interactive Mobile Applications

OLED power management on mobile devices is very challenging due to the dynamic nature of human-screen interaction. This paper presents the design, algorithms, and implementation of a lightweight mobile app called ShiftMask, which allows the user to dynamically shift OLED power to the portion of interest, while dimming the remainder of the screen based on visual acuity. To adapt to the user’s focus of attention, we propose efficient algorithms that consider visual fixation in static scenes, as well as changes in focus and screen scrolling. The results of experiments conducted on a commercial smartphone with popular interactive apps demonstrate that ShiftMask can achieve substantial energy savings, while preserving acceptable readability.

• [Conference] [Demo]

Win-Win Camera

A Win-Win Camera: Quality-Enhanced Power-Saving Images on Mobile OLED Displays.

Mobile systems will increasingly feature emerging OLED displays, whose power consumption is highly dependent on the image content. Existing OLED power-saving techniques change users’ visual experience or degrade images’ visual quality in exchange for power reduction. This paper posits the possibility of a win-win scheme that enhances image quality and reduces power consumption simultaneously. We define metrics to assess the profit and the cost for potential image enhancement and power reduction. Then, we propose fundamental algorithms that ensure the transformation of images into their quality- enhanced power-saving versions. Finally, the proposed scheme is realized as a practical camera application on mobile devices. The results of experiments conducted on a commercial tablet with a popular image database are very encouraging and provide valuable insights for future research and practices.

• [Conference] [Poster] [Demo] [App Demo]

Augmented Sensing

A Semantics-Aware Design for Mounting Remote Sensors on Mobile Systems.

Application paradigms will increasingly exceed a mobile device’s physical boundaries. This paper presents a system solution for a mobile device to mount remote sensors on other devices. Our design is generic to mobile senor stacks, thus supporting unmodified apps and commodity sensors. Furthermore, it uses an asynchronous access model to facilitate semantics passing and data reporting in between. Such semantic information allows the development of an energy-efficient reporting policy for remote sensing applications. The results of experiments conducted on commercial Android smartphones with popular apps demonstrate that our design is very efficient in terms of energy consumption and completion time.

• [Conference] [Demo]

SIMTY Wakeup Management

Similarity-Based Wakeup Management for Mobile Systems in Connected Standby

Resident applications, which autonomously awaken mobile devices, can gradually and imperceptibly drain device batteries. This paper introduces the concept of alarm similarity into wakeup management for mobile systems in connected standby. First, we define hardware similarity to reflect the degree of energy savings and time similarity to reflect the impact on user experience. We then propose a policy that aligns alarms based on their similarity to save standby energy while maintaining the quality of the user experience. Finally, we integrate our design into Android and conduct extensive experiments on a commercial smartphone running popular mobile apps. The results demonstrate that our design can further extend the standby time achieved with Android’s native policy by up to one-third.

• [Conference] [Demo]

User-Centric Scheduling

User-Centric Energy-Efficient Scheduling on Multi-Core Mobile Devices

Mobile devices will provide improved computing resources to sustain progressively more complicated applications. However, the design concept of fair scheduling and governing borrowed from legacy operating systems cannot be applied seamlessly in mobile systems, thereby degrading user experience or reducing energy efficiency. In this paper, we posit that mobile applications should be treated unfairly. To this end, we exploit the concept of application sensitivity and devise a user-centric scheduler and governor that allocate computing resources to applications according to their sensitivity. Furthermore, we integrate our design into the Android operating system. The results of extensive experiments on a commercial smartphone with real-world mobile apps demonstrate that the proposed design can achieve significant energy efficiency gains while improving the quality of user experience.

• [Conference] [Journal] [Demo]

Catch Your Attetion

CURA: A Framework for Quality-retaining Power Saving on Mobile OLED Displays.

Organic light-emitting diode (OLED) technology is regarded as a promising alternative to mobile displays. In this paper, we introduce the design, algorithm, and implementation of a novel framework called CURA for quality-retaining power saving on mobile OLED displays. First, we link human visual attention to OLED power saving and model the OLED image scaling optimization problem. The objective is to minimize the power required to display an image without adversely impacting the user’s visual experience. Then, we present the algorithm used to solve the fundamental problem, and prove its optimality even without an accurate power model. Finally, based on the framework, we implement two practical applications on a commercial OLED mobile tablet. The results of experiments conducted on the tablet with real images demonstrate that CURA can reduce significant OLED power consumption while retaining the visual quality of images.

• [Conference] [Journal] [Poster] [Demo]

Extend Your Journey

Extend Your Journey: Considering Signal Strength and Fluctuation in Location-based Applications

Reducing the communication energy is essential to facilitate the growth of emerging mobile applications. In this paper, we introduce signal strength into location-based applications to reduce the energy consumption of mobile devices for data reception. First, we model the problem of data fetch scheduling, with the objective of minimizing the energy required to fetch location-based information without impacting the application’s semantics adversely. To solve the fundamental problem, we propose a dynamicprogramming algorithm and prove its optimality in terms of energy savings. Then, we perform postoptimal analysis to explore the tolerance of the algorithm to signal strength fluctuations. Finally, based on the algorithm, we consider implementation issues. We have also developed a virtual tour system integrated with existing web applications to validate the practicability of the proposed concept. The results of experiments conducted based on real-world case studies are very encouraging and demonstrate the applicability of the proposed algorithm towards signal strength fluctuations.

• [Conference] [Journal] [Demo]

Dynamic Backlight Scaling

Dynamic Backlight Scaling Optimization: A Cloud-Based Energy-Saving Service for Mobile Streaming Applications

With the increasing variety of mobile applications, reducing the energy consumption of mobile devices is a major challenge in sustaining multimedia streaming applications. This paper explores how to minimize the energy consumption of the backlight when displaying a video stream without adversely impacting the user’s visual experience. First, we model the problem as a dynamic backlight scaling optimization problem. Then, we propose algorithms to solve the fundamental problem and prove the optimality in terms of energy savings. Finally, based on the algorithms, we present a cloud-based energy-saving service. We have also developed a prototype implementation integrated with existing video streaming applications to validate the practicability of the approach. The results of experiments conducted to evaluate the ef¿cacy of the proposed approach are very encouraging and show energy savings of 15-49% on off-the-shelf mobile devices.

• [Conference] [Journal] [Demo] [Android Demo] [iOS Demo]