diff --git a/docs/contents/ai/socratiq.html b/docs/contents/ai/socratiq.html index c20e2327..73b009a7 100644 --- a/docs/contents/ai/socratiq.html +++ b/docs/contents/ai/socratiq.html @@ -674,9 +674,9 @@

Quick Start Guide

Button Overview

The top nav bar provices quick access to the following features:

    -
  1. Adjust your settings at any time.
    2. -
  2. Track your progress by viewing the dashboard.
    3. -
  3. Start new or save your conversations with SocratiQ.
    4. +
  4. Adjust your settings at any time.
    5. +
  5. Track your progress by viewing the dashboard.
    6. +
  6. Start new or save your conversations with SocratiQ.
diff --git a/docs/contents/core/frameworks/frameworks.html b/docs/contents/core/frameworks/frameworks.html index 3bfef71e..dad5cfa7 100644 --- a/docs/contents/core/frameworks/frameworks.html +++ b/docs/contents/core/frameworks/frameworks.html @@ -1610,7 +1610,7 @@

6.9 Choosing the Right Framework

Choosing the right machine learning framework for a given application requires carefully evaluating models, hardware, and software considerations. Figure 6.13 provides a comparison of different TensorFlow frameworks, which we’ll discuss in more detail:

-
+
@@ -1630,7 +1630,7 @@

6.9.2 Software

As shown in Figure 6.14, TensorFlow Lite Micro does not have OS support, while TensorFlow and TensorFlow Lite do. This design choice for TensorFlow Lite Micro helps reduce memory overhead, make startup times faster, and consume less energy. Instead, TensorFlow Lite Micro can be used in conjunction with real-time operating systems (RTOS) like FreeRTOS, Zephyr, and Mbed OS.

The figure also highlights an important memory management feature: TensorFlow Lite and TensorFlow Lite Micro support model memory mapping, allowing models to be directly accessed from flash storage rather than loaded into RAM. In contrast, TensorFlow does not offer this capability.

-
+
@@ -1646,7 +1646,7 @@

6.9.3 Hardware

TensorFlow Lite and TensorFlow Lite Micro have significantly smaller base binary sizes and memory footprints than TensorFlow (see Figure 6.15). For example, a typical TensorFlow Lite Micro binary is less than 200KB, whereas TensorFlow is much larger. This is due to the resource-constrained environments of embedded systems. TensorFlow supports x86, TPUs, and GPUs like NVIDIA, AMD, and Intel.

-
+
@@ -1693,7 +1693,7 @@

6.10.1 Decomposition

Currently, the ML system stack consists of four abstractions as shown in Figure 6.16, namely (1) computational graphs, (2) tensor programs, (3) libraries and runtimes, and (4) hardware primitives.

-
+
diff --git a/docs/contents/core/ml_systems/images/png/hybrid.png b/docs/contents/core/ml_systems/images/png/hybrid.png new file mode 100644 index 00000000..98dc681a Binary files /dev/null and b/docs/contents/core/ml_systems/images/png/hybrid.png differ diff --git a/docs/contents/core/ml_systems/images/png/venndiagram.png b/docs/contents/core/ml_systems/images/png/venndiagram.png deleted file mode 100644 index 5754133c..00000000 Binary files a/docs/contents/core/ml_systems/images/png/venndiagram.png and /dev/null differ diff --git a/docs/contents/core/ml_systems/ml_systems.html b/docs/contents/core/ml_systems/ml_systems.html index 63c8eb57..0b69e679 100644 --- a/docs/contents/core/ml_systems/ml_systems.html +++ b/docs/contents/core/ml_systems/ml_systems.html @@ -518,7 +518,6 @@

Table of contents

@@ -653,9 +686,9 @@

    -

  • Understand the key characteristics and differences between Cloud ML, Edge ML, and TinyML systems.

    • +

  • Understand the key characteristics and differences between Cloud ML, Edge ML, Mobile ML, and TinyML systems.

  • Analyze the benefits and challenges associated with each ML paradigm.

    • -

  • Explore real-world applications and use cases for Cloud ML, Edge ML, and TinyML.

    • +

  • Explore real-world applications and use cases for Cloud ML, Edge ML, Mobile ML, and TinyML.

  • Compare the performance aspects of each ML approach, including latency, privacy, and resource utilization.

  • Examine the evolving landscape of ML systems and potential future developments.

@@ -665,7 +698,113 @@

2.1 Overview

ML is rapidly evolving, with new paradigms reshaping how models are developed, trained, and deployed. The field is experiencing significant innovation driven by advancements in hardware, software, and algorithmic techniques. These developments are enabling machine learning to be applied in diverse settings, from large-scale cloud infrastructures to edge devices and even tiny, resource-constrained environments.

Modern machine learning systems span a spectrum of deployment options, each with its own set of characteristics and use cases. At one end, we have cloud-based ML, which leverages powerful centralized computing resources for complex, data-intensive tasks. Moving along the spectrum, we encounter edge ML, which brings computation closer to the data source for reduced latency and improved privacy. At the far end, we find TinyML, which enables machine learning on extremely low-power devices with severe memory and processing constraints.

-

This chapter explores the landscape of contemporary machine learning systems, covering three key approaches: Cloud ML, Edge ML, and TinyML. Figure 2.1 illustrates the spectrum of distributed intelligence across these approaches, providing a visual comparison of their characteristics. We will examine the unique characteristics, advantages, and challenges of each approach, as depicted in the figure. Additionally, we will discuss the emerging trends and technologies that are shaping the future of machine learning deployment, considering how they might influence the balance between these three paradigms.

+

To better understand the dramatic differences between these ML deployment options, Table 2.1 provides examples of representative hardware platforms for each category. These examples illustrate the vast range of computational resources, power requirements, and cost considerations across the ML systems spectrum. As we explore each paradigm in detail, you can refer back to these concrete examples to better understand the practical implications of each approach.

+
+
+
+Table 2.1: Representative hardware platforms across the ML systems spectrum, showing typical specifications and capabilities for each category. +
+
+ ++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CategoryExample DeviceProcessorMemoryStoragePowerPrice RangeExample Models/Tasks
Cloud MLNVIDIA DGX A1008x NVIDIA A100 GPUs (40GB/80GB)1TB System RAM15TB NVMe SSD6.5kW$200K+Large language models (GPT-3), real-time video processing
Google TPU v4 Pod4096 TPU v4 chips128TB+Networked storage~MWPay-per-useTraining foundation models, large-scale ML research
Edge MLNVIDIA Jetson AGX Orin12-core Arm® Cortex®-A78AE, NVIDIA Ampere GPU32GB LPDDR564GB eMMC15-60W$899Computer vision, robotics, autonomous systems
Intel NUC 12 ProIntel Core i7-1260P, Intel Iris Xe32GB DDR41TB SSD28W$750Edge AI servers, industrial automation
Mobile MLiPhone 15 ProA17 Pro (6-core CPU, 6-core GPU)8GB RAM128GB-1TB3-5W$999+Face ID, computational photography, voice recognition
TinyMLArduino Nano 33 BLE SenseArm Cortex-M4 @ 64MHz256KB RAM1MB Flash0.02-0.04W$35Gesture recognition, voice detection
ESP32-CAMDual-core @ 240MHz520KB RAM4MB Flash0.05-0.25W$10Image classification, motion detection
+
+
+
+

This chapter explores the landscape of contemporary machine learning systems, covering four key approaches: Cloud ML, Edge ML, and TinyML. Figure 2.1 illustrates the spectrum of distributed intelligence across these approaches, providing a visual comparison of their characteristics. We will examine the unique characteristics, advantages, and challenges of each approach, as depicted in the figure. Additionally, we will discuss the emerging trends and technologies that are shaping the future of machine learning deployment, considering how they might influence the balance between these three paradigms.

@@ -676,20 +815,20 @@

-

  • Cloud ML: Initially, ML was predominantly cloud-based. Powerful servers in data centers were used to train and run large ML models. This approach leverages vast computational resources and storage capacities, enabling the development of complex models trained on massive datasets. Cloud ML excels at tasks requiring extensive processing power and is ideal for applications where real-time responsiveness isn’t critical.

    • -

  • Edge ML: As the need for real-time, low-latency processing grew, Edge ML emerged. This paradigm brings inference capabilities closer to the data source, typically on edge devices such as smartphones, smart cameras, or IoT gateways. Edge ML reduces latency, enhances privacy by keeping data local, and can operate with intermittent cloud connectivity. It’s particularly useful for applications requiring quick responses or handling sensitive data.

    • -

  • TinyML: The latest development in this progression is TinyML, which enables ML models to run on extremely resource-constrained microcontrollers and small embedded systems. TinyML allows for on-device inference without relying on connectivity to the cloud or edge, opening up new possibilities for intelligent, battery-operated devices. This approach is crucial for applications where size, power consumption, and cost are critical factors.

    • - +

    The evolution of machine learning systems can be seen as a progression from centralized to increasingly distributed and specialized computing paradigms:

    +

    Cloud ML: Initially, ML was predominantly cloud-based. Powerful, scalable servers in data centers are used to train and run large ML models. This approach leverages vast computational resources and storage capacities, enabling the development of complex models trained on massive datasets. Cloud ML excels at tasks requiring extensive processing power, distributed training of large models, and is ideal for applications where real-time responsiveness isn’t critical. Popular platforms like AWS SageMaker, Google Cloud AI, and Azure ML offer flexible, scalable solutions for model development, training, and deployment. Cloud ML can handle models with billions of parameters, training on petabytes of data, but may incur latencies of 100-500ms for online inference due to network delays.

    +

    Edge ML: As the need for real-time, low-latency processing grew, Edge ML emerged. This paradigm brings inference capabilities closer to the data source, typically on edge devices such as industrial gateways, smart cameras, autonomous vehicles, or IoT hubs. Edge ML reduces latency (often to less than 50ms), enhances privacy by keeping data local, and can operate with intermittent cloud connectivity. It’s particularly useful for applications requiring quick responses or handling sensitive data in industrial or enterprise settings. Frameworks like NVIDIA Jetson or Google’s Edge TPU enable powerful ML capabilities on edge devices. Edge ML plays a crucial role in IoT ecosystems, enabling real-time decision making and reducing bandwidth usage by processing data locally.

    +

    Mobile ML: Building on edge computing concepts, Mobile ML focuses on leveraging the computational capabilities of smartphones and tablets. This approach enables personalized, responsive applications while reducing reliance on constant network connectivity. Mobile ML offers a balance between the power of edge computing and the ubiquity of personal devices. It utilizes on-device sensors (e.g., cameras, GPS, accelerometers) for unique ML applications. Frameworks like TensorFlow Lite and Core ML allow developers to deploy optimized models on mobile devices, with inference times often under 30ms for common tasks. Mobile ML enhances privacy by keeping personal data on the device and can operate offline, but must balance model performance with device resource constraints (typically 4-8GB RAM, 100-200GB storage).

    +

    TinyML: The latest development in this progression is TinyML, which enables ML models to run on extremely resource-constrained microcontrollers and small embedded systems. TinyML allows for on-device inference without relying on connectivity to the cloud, edge, or even the processing power of mobile devices. This approach is crucial for applications where size, power consumption, and cost are critical factors. TinyML devices typically operate with less than 1MB of RAM and flash memory, consuming only milliwatts of power, enabling battery life of months or years. Applications include wake word detection, gesture recognition, and predictive maintenance in industrial settings. Platforms like Arduino Nano 33 BLE Sense and STM32 microcontrollers, coupled with frameworks like TensorFlow Lite for Microcontrollers, enable ML on these tiny devices. However, TinyML requires significant model optimization and quantization to fit within these constraints.

    Each of these paradigms has its own strengths and is suited to different use cases:

    • Cloud ML remains essential for tasks requiring massive computational power or large-scale data analysis.
      • -
    • Edge ML is ideal for applications needing low-latency responses or local data processing.
      • +
    • Edge ML is ideal for applications needing low-latency responses or local data processing in industrial or enterprise environments.
      • +
    • Mobile ML is suited for personalized, responsive applications on smartphones and tablets.
    • TinyML enables AI capabilities in small, power-efficient devices, expanding the reach of ML to new domains.
    -

    The progression from Cloud to Edge to TinyML reflects a broader trend in computing towards more distributed, localized processing. This evolution is driven by the need for faster response times, improved privacy, reduced bandwidth usage, and the ability to operate in environments with limited or no connectivity.

    -

    Figure 2.2 illustrates the key differences between Cloud ML, Edge ML, and TinyML in terms of hardware, latency, connectivity, power requirements, and model complexity. As we move from Cloud to Edge to TinyML, we see a dramatic reduction in available resources, which presents significant challenges for deploying sophisticated machine learning models. This resource disparity becomes particularly apparent when attempting to deploy deep learning models on microcontrollers, the primary hardware platform for TinyML. These tiny devices have severely constrained memory and storage capacities, which are often insufficient for conventional deep learning models. We will learn to put these things into perspective in this chapter.

    +

    This progression reflects a broader trend in computing towards more distributed, localized, and specialized processing. The evolution is driven by the need for faster response times, improved privacy, reduced bandwidth usage, and the ability to operate in environments with limited or no connectivity, while also catering to the specific capabilities and constraints of different types of devices.

    +

    Figure 2.2 illustrates the key differences between Cloud ML, Edge ML, Mobile ML, and TinyML in terms of hardware, latency, connectivity, power requirements, and model complexity. As we move from Cloud to Edge to TinyML, we see a dramatic reduction in available resources, which presents significant challenges for deploying sophisticated machine learning models. This resource disparity becomes particularly apparent when attempting to deploy deep learning models on microcontrollers, the primary hardware platform for TinyML. These tiny devices have severely constrained memory and storage capacities, which are often insufficient for conventional deep learning models. We will learn to put these things into perspective in this chapter.

    @@ -705,7 +844,7 @@

    2.2 Cloud ML

    -

    Cloud ML leverages powerful servers in the cloud for training and running large, complex ML models and relies on internet connectivity. Figure 2.3 provides an overview of Cloud ML’s capabilities which we will discuss in greater detail throughout this section.

    +

    Cloud Machine Learning (Cloud ML) is a subfield of machine learning that leverages the power and scalability of cloud computing infrastructure to develop, train, and deploy machine learning models. By utilizing the vast computational resources available in the cloud, Cloud ML enables the efficient handling of large-scale datasets and complex machine learning algorithms. Figure 2.3 provides an overview of Cloud ML’s capabilities which we will discuss in greater detail throughout this section.

    @@ -718,10 +857,6 @@

    2.2.1 Characteristics

    -
    -

    Definition of Cloud ML

    -

    Cloud Machine Learning (Cloud ML) is a subfield of machine learning that leverages the power and scalability of cloud computing infrastructure to develop, train, and deploy machine learning models. By utilizing the vast computational resources available in the cloud, Cloud ML enables the efficient handling of large-scale datasets and complex machine learning algorithms.

    -

    Centralized Infrastructure

    One of the key characteristics of Cloud ML is its centralized infrastructure. Figure 2.4 illustrates this concept with an example from Google’s Cloud TPU data center. Cloud service providers offer a virtual platform that consists of high-capacity servers, expansive storage solutions, and robust networking architectures, all housed in data centers distributed across the globe. As shown in the figure, these centralized facilities can be massive in scale, housing rows upon rows of specialized hardware. This centralized setup allows for the pooling and efficient management of computational resources, making it easier to scale machine learning projects as needed.

    @@ -831,10 +966,6 @@

    Security an

    2.3 Edge ML

    -
    -

    2.3.1 Characteristics

    -
    -

    Definition of Edge ML

    Edge Machine Learning (Edge ML) runs machine learning algorithms directly on endpoint devices or closer to where the data is generated rather than relying on centralized cloud servers. This approach brings computation closer to the data source, reducing the need to send large volumes of data over networks, often resulting in lower latency and improved data privacy. Figure 2.5 provides an overview of this section.

    @@ -846,7 +977,8 @@

    Definition of Edge M

    -
    +
    +

    2.3.1 Characteristics

    Decentralized Data Processing

    In Edge ML, data processing happens in a decentralized fashion, as illustrated in Figure 2.6. Instead of sending data to remote servers, the data is processed locally on devices like smartphones, tablets, or Internet of Things (IoT) devices. The figure showcases various examples of these edge devices, including wearables, industrial sensors, and smart home appliances. This local processing allows devices to make quick decisions based on the data they collect without relying heavily on a central server’s resources.

    @@ -914,12 +1046,73 @@

    Industrial IoT

    -
    -

    2.4 Tiny ML

    +
    +

    2.4 Mobile ML

    +

    Mobile Machine Learning (Mobile ML) represents a specialized branch of Edge ML that focuses on deploying and running machine learning models directly on mobile devices like smartphones and tablets. This approach leverages the computational capabilities of modern mobile processors to perform ML tasks locally, offering a balance between the power of edge computing and the ubiquity of personal devices.

    2.4.1 Characteristics

    -
    -

    Definition of TinyML

    +
    +

    On-Device Processing

    +

    Mobile ML utilizes the processing power of mobile devices’ System-on-Chip (SoC) architectures, including specialized Neural Processing Units (NPUs) and AI accelerators. This enables efficient execution of ML models directly on the device, allowing for real-time processing of data from device sensors like cameras, microphones, and motion sensors without constant cloud connectivity.

    +
    +
    +

    Optimized Frameworks

    +

    Mobile ML is supported by specialized frameworks and tools designed specifically for mobile deployment, such as TensorFlow Lite for Android devices and Core ML for iOS devices. These frameworks are optimized for mobile hardware and provide efficient model compression and quantization techniques to ensure smooth performance within mobile resource constraints.

    +
    +
    +
    +

    2.4.2 Benefits

    +
    +

    Real-Time Processing

    +

    Mobile ML enables real-time processing of data directly on mobile devices, eliminating the need for constant server communication. This results in faster response times for applications requiring immediate feedback, such as real-time translation, face detection, or gesture recognition.

    +
    +
    +

    Privacy Preservation

    +

    By processing data locally on the device, Mobile ML helps maintain user privacy. Sensitive information doesn’t need to leave the device, reducing the risk of data breaches and addressing privacy concerns, particularly important for applications handling personal data.

    +
    +
    +

    Offline Functionality

    +

    Mobile ML applications can function without constant internet connectivity, making them reliable in areas with poor network coverage or when users are offline. This ensures consistent performance and user experience regardless of network conditions.

    +
    +
    +
    +

    2.4.3 Challenges

    +
    +

    Resource Constraints

    +

    Despite modern mobile devices being powerful, they still face resource constraints compared to cloud servers. Mobile ML must operate within limited RAM, storage, and processing power, requiring careful optimization of models and efficient resource management.

    +
    +
    +

    Battery Life Impact

    +

    ML operations can be computationally intensive, potentially impacting device battery life. Developers must balance model complexity and performance with power consumption to ensure reasonable battery life for users.

    +
    +
    +

    Model Size Limitations

    +

    Mobile devices have limited storage space, necessitating careful consideration of model size. This often requires model compression and quantization techniques, which can affect model accuracy and performance.

    +
    +
    +
    +

    2.4.4 Example Use Cases

    +
    +

    Computer Vision Applications

    +

    Mobile ML has revolutionized how we use cameras on mobile devices, enabling sophisticated computer vision applications that process visual data in real-time. Modern smartphone cameras now incorporate ML models that can detect faces, analyze scenes, and apply complex filters instantaneously. These models work directly on the camera feed to enable features like portrait mode photography, where ML algorithms separate foreground subjects from backgrounds. Document scanning applications use ML to detect paper edges, correct perspective, and enhance text readability, while augmented reality applications use ML-powered object detection to accurately place virtual objects in the real world.

    +
    +
    +

    Natural Language Processing

    +

    Natural language processing on mobile devices has transformed how we interact with our phones and communicate with others. Speech recognition models run directly on device, enabling voice assistants to respond quickly to commands even without internet connectivity. Real-time translation applications can now translate conversations and text without sending data to the cloud, preserving privacy and working reliably regardless of network conditions. Mobile keyboards have become increasingly intelligent, using ML to predict not just the next word but entire phrases based on the user’s writing style and context, while maintaining all learning and personalization locally on the device.

    +
    +
    +

    Health and Fitness Monitoring

    +

    Mobile ML has enabled smartphones and tablets to become sophisticated health monitoring devices. Through clever use of existing sensors combined with ML models, mobile devices can now track physical activity, analyze sleep patterns, and monitor vital signs. For example, cameras can measure heart rate by detecting subtle color changes in the user’s skin, while accelerometers and ML models work together to recognize specific exercises and analyze workout form. These applications process sensitive health data directly on the device, ensuring privacy while providing users with real-time feedback and personalized health insights.

    +
    +
    +

    Personalization and User Experience

    +

    Perhaps the most pervasive but least visible application of Mobile ML lies in how it personalizes and enhances the overall user experience. ML models continuously analyze how users interact with their devices to optimize everything from battery usage to interface layouts. These models learn individual usage patterns to predict which apps users are likely to open next, preload content they might want to see, and adjust system settings like screen brightness and audio levels based on environmental conditions and user preferences. This creates a deeply personalized experience that adapts to each user’s needs while maintaining privacy by keeping all learning and adaptation on the device itself.

    +

    These applications demonstrate how Mobile ML bridges the gap between cloud-based solutions and edge computing, providing efficient, privacy-conscious, and user-friendly machine learning capabilities on personal mobile devices. The continuous advancement in mobile hardware capabilities and optimization techniques continues to expand the possibilities for Mobile ML applications.

    +
    +
    +
    +
    +

    2.5 Tiny ML

    TinyML sits at the crossroads of embedded systems and machine learning, representing a burgeoning field that brings smart algorithms directly to tiny microcontrollers and sensors. These microcontrollers operate under severe resource constraints, particularly regarding memory, storage, and computational power. Figure 2.7 encapsulates the key aspects of TinyML discussed in this section.

    @@ -931,10 +1124,11 @@

    Definition of TinyML<

    -
    +
    +

    2.5.1 Characteristics

    On-Device Machine Learning

    -

    In TinyML, the focus is on on-device machine learning. This means that machine learning models are deployed and trained on the device, eliminating the need for external servers or cloud infrastructures. This allows TinyML to enable intelligent decision-making right where the data is generated, making real-time insights and actions possible, even in settings where connectivity is limited or unavailable.

    +

    In TinyML, the focus, much like in Mobile ML, is on on-device machine learning. This means that machine learning models are deployed and trained on the device, eliminating the need for external servers or cloud infrastructures. This allows TinyML to enable intelligent decision-making right where the data is generated, making real-time insights and actions possible, even in settings where connectivity is limited or unavailable.

    Low Power and Resource-Constrained Environments

    @@ -968,8 +1162,8 @@

    -

    2.4.2 Benefits

    +
    +

    2.5.2 Benefits

    Extremely Low Latency

    One of the standout benefits of TinyML is its ability to offer ultra-low latency. Since computation occurs directly on the device, the time required to send data to external servers and receive a response is eliminated. This is crucial in applications requiring immediate decision-making, enabling quick responses to changing conditions.

    @@ -983,8 +1177,8 @@

    Energy Efficiency

    TinyML operates within an energy-efficient framework, a necessity given its resource-constrained environments. By employing lean algorithms and optimized computational methods, TinyML ensures that devices can execute complex tasks without rapidly depleting battery life, making it a sustainable option for long-term deployments.

    -
    -

    2.4.3 Challenges

    +
    +

    2.5.3 Challenges

    Limited Computational Capabilities

    However, the shift to TinyML comes with its set of hurdles. The primary limitation is the devices’ constrained computational capabilities. The need to operate within such limits means that deployed models must be simplified, which could affect the accuracy and sophistication of the solutions.

    @@ -998,8 +1192,8 @@

    Model O

    A central challenge in TinyML is model optimization and compression. Creating machine learning models that can operate effectively within the limited memory and computational power of microcontrollers requires innovative approaches to model design. Developers often face the challenge of striking a delicate balance and optimizing models to maintain effectiveness while fitting within stringent resource constraints.

    -
    -

    2.4.4 Example Use Cases

    +
    +

    2.5.4 Example Use Cases

    Wearable Devices

    In wearables, TinyML opens the door to smarter, more responsive gadgets. From fitness trackers offering real-time workout feedback to smart glasses processing visual data on the fly, TinyML transforms how we engage with wearable tech, delivering personalized experiences directly from the device.

    @@ -1019,107 +1213,208 @@

    Environmental Mon

    -
    -

    2.5 Comparison

    -

    Let’s bring together the different ML variants we’ve explored individually for a comprehensive view. Figure 2.9 illustrates the relationships and overlaps between Cloud ML, Edge ML, and TinyML using a Venn diagram. This visual representation effectively highlights the unique characteristics of each approach while also showing areas of commonality. Each ML paradigm has its own distinct features, but there are also intersections where these approaches share certain attributes or capabilities. This diagram helps us understand how these variants relate to each other in the broader landscape of machine learning implementations.

    -
    +
    +

    2.6 Hybrid ML

    +

    While we’ve examined Cloud ML, Edge ML, Mobile ML, and TinyML as distinct approaches, the reality of modern ML deployments is more nuanced. Systems architects often combine these paradigms to create solutions that leverage the strengths of each approach while mitigating their individual limitations. Understanding how these systems can work together opens up new possibilities for building more efficient and effective ML applications.

    +
    +

    2.6.1 Train-Serve Split

    +

    One of the most common hybrid patterns is the train-serve split, where model training occurs in the cloud but inference happens on edge, mobile, or tiny devices. This pattern takes advantage of the cloud’s vast computational resources for the training phase while benefiting from the low latency and privacy advantages of on-device inference. For example, smart home devices often use models trained on large datasets in the cloud but run inference locally to ensure quick response times and protect user privacy. In practice, this might involve training models on powerful systems like the NVIDIA DGX A100, leveraging its 8 A100 GPUs and terabyte-scale memory, before deploying optimized versions to edge devices like the NVIDIA Jetson AGX Orin for efficient inference. Similarly, mobile vision models for computational photography are typically trained on powerful cloud infrastructure but deployed to run efficiently on phone hardware.

    +
    +
    +

    2.6.2 Hierarchical Processing

    +

    Hierarchical processing creates a multi-tier system where data and intelligence flow between different levels of the ML stack. In industrial IoT applications, tiny sensors might perform basic anomaly detection, edge devices aggregate and analyze data from multiple sensors, and cloud systems handle complex analytics and model updates. For instance, we might see ESP32-CAM devices performing basic image classification at the sensor level with their minimal 520KB RAM, feeding data up to Jetson AGX Orin devices for more sophisticated computer vision tasks, and ultimately connecting to cloud infrastructure for complex analytics and model updates.

    +

    This hierarchy allows each tier to handle tasks appropriate to its capabilities—TinyML devices handle immediate, simple decisions; edge devices manage local coordination; and cloud systems tackle complex analytics and learning tasks. Smart city installations often use this pattern, with street-level sensors feeding data to neighborhood-level edge processors, which in turn connect to city-wide cloud analytics.

    +
    +
    +

    2.6.3 Federated Learning

    +

    Federated learning represents a sophisticated hybrid approach where model training is distributed across many edge or mobile devices while maintaining privacy. Devices learn from local data and share model updates, rather than raw data, with cloud servers that aggregate these updates into an improved global model. This pattern is particularly powerful for applications like keyboard prediction on mobile devices or healthcare analytics, where privacy is paramount but benefits from collective learning are valuable. The cloud coordinates the learning process without directly accessing sensitive data, while devices benefit from the collective intelligence of the network.

    +
    +
    +

    2.6.4 Progressive Deployment

    +

    Progressive deployment strategies adapt models for different computational tiers, creating a cascade of increasingly lightweight versions. A model might start as a large, complex version in the cloud, then be progressively compressed and optimized for edge servers, mobile devices, and finally tiny sensors. Voice assistant systems often employ this pattern—full natural language processing runs in the cloud, while simplified wake-word detection runs on-device. This allows the system to balance capability and resource constraints across the ML stack.

    +
    +
    +

    2.6.5 Collaborative Learning

    +

    Collaborative learning enables peer-to-peer learning between devices at the same tier, often complementing hierarchical structures. Autonomous vehicle fleets, for example, might share learning about road conditions or traffic patterns directly between vehicles while also communicating with cloud infrastructure. This horizontal collaboration allows systems to share time-sensitive information and learn from each other’s experiences without always routing through central servers.

    +

    These hybrid patterns demonstrate how modern ML systems are evolving beyond simple client-server architectures into rich, multi-tier systems that combine the strengths of different approaches. By understanding these patterns, system architects can design solutions that effectively balance competing demands for computation, latency, privacy, and power efficiency. The future of ML systems likely lies not in choosing between cloud, edge, mobile, or tiny approaches, but in creatively combining them to build more capable and efficient systems.

    +
    +
    +

    2.6.6 Real-World Integration Patterns

    +

    In practice, ML systems rarely operate in isolation. Instead, they form interconnected networks where each paradigm—Cloud, Edge, Mobile, and TinyML—plays a specific role while communicating with other parts of the system. These interactions follow distinct patterns that emerge from the inherent strengths and limitations of each approach.

    +

    Cloud systems excel at training and analytics but require significant infrastructure. Edge systems provide local processing power and reduced latency. Mobile devices offer personal computing capabilities and user interaction. TinyML enables intelligence in the smallest devices and sensors.

    +
    -
    - +
    +
    -
    -Figure 2.9: ML Venn diagram. Source: arXiv +
    +Figure 2.9: Example interaction patterns between ML paradigms, showing data flows, model deployment, and processing relationships across Cloud, Edge, Mobile, and TinyML systems.
    -

    For a more detailed comparison of these ML variants, we can refer to Table 2.1. This table offers a comprehensive analysis of Cloud ML, Edge ML, and TinyML based on various features and aspects. By examining these different characteristics side by side, we gain a clearer perspective on the unique advantages and distinguishing factors of each approach. This detailed comparison, combined with the visual overview provided by the Venn diagram, aids in making informed decisions based on the specific needs and constraints of a given application or project.

    +

    Figure 2.9 illustrates the key interactions between these different ML paradigms. Notice how data flows upward from sensors through processing layers to cloud analytics, while model deployments flow downward from cloud training to various inference points. The interactions aren’t strictly hierarchical—mobile devices might communicate directly with both cloud services and tiny sensors, while edge systems can assist mobile devices with complex processing tasks. To understand how these interactions manifest in real applications, let’s explore several common scenarios using Figure 2.9:

    +
      +

    • Model Deployment Scenario: A company develops a computer vision model for defect detection. After training in the cloud, optimized versions are deployed to edge servers in factories, quality control tablets on the production floor, and tiny cameras embedded in the production line. This showcases how a single ML solution can be distributed across different computational tiers for optimal performance.

      • +

    • Data Flow and Analysis Scenario: In a smart agriculture system, soil sensors (TinyML) collect moisture and nutrient data, sending results to edge processors in local stations. These process the data and forward insights to the cloud for farm-wide analytics, while also sharing results with farmers’ mobile apps. This demonstrates the hierarchical flow of data from sensors to cloud analytics.

      • +

    • Edge-Mobile Assistance Scenario: When a mobile app needs to perform complex image processing that exceeds the phone’s capabilities, it connects to a nearby edge server. The edge system helps process the heavier computational tasks, sending back results to enhance the mobile app’s performance. This shows how different ML tiers can cooperate to handle demanding tasks.

      • +

    • TinyML-Mobile Integration Scenario: A fitness tracker uses TinyML to continuously monitor activity patterns and vital signs. It synchronizes this processed data with the user’s smartphone, which combines it with other health data before sending consolidated updates to the cloud for long-term health analysis. This illustrates the common pattern of tiny devices using mobile devices as gateways to larger networks.

      • +

    • Multi-Layer Processing Scenario: In a smart retail environment, tiny sensors monitor inventory levels, sending inference results to both edge systems for immediate stock management and mobile devices for staff notifications. The edge systems process this data alongside other store metrics, while the cloud analyzes trends across all store locations. This shows how multiple ML tiers can work together in a complete solution.

      • +
    +

    These real-world patterns demonstrate how different ML paradigms naturally complement each other in practice. While each approach has its own strengths, their true power emerges when they work together as an integrated system. By understanding these patterns, system architects can better design solutions that effectively leverage the capabilities of each ML tier while managing their respective constraints.

    +
    +
    +
    +

    2.7 Comparison

    +

    Let’s bring together the different ML variants we’ve explored individually for a comprehensive view. For a detailed comparison of these ML variants, we can refer to Table 2.2. This table offers a comprehensive analysis of Cloud ML, Edge ML, and TinyML based on various features and aspects. By examining these different characteristics side by side, we gain a clearer perspective on the unique advantages and distinguishing factors of each approach. This detailed comparison, combined with the visual overview provided by the Venn diagram, aids in making informed decisions based on the specific needs and constraints of a given application or project.

    -Table 2.1: Comparison of feature aspects across Cloud ML, Edge ML, and TinyML. +Table 2.2: Comparison of feature aspects across Cloud ML, Edge ML, and TinyML.
    ----+++++ + - - - + + + + - - - + + + + - - - + + + + - - - + + + + - - - + + + + - - - + + + + - - - + + + + - - - - + + + + + - - - + + + + - - - - + + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Aspect Cloud ML Edge MLMobile ML TinyML
    Processing LocationCentralized servers (Data Centers)Local devices (closer to data sources)On-device (microcontrollers, embedded systems)Centralized cloud servers (Data Centers)Local edge devices (gateways, servers)Smartphones and tabletsUltra-low-power microcontrollers and embedded systems
    LatencyHigh (Depends on internet connectivity)Moderate (Reduced latency compared to Cloud ML)Low (Immediate processing without network delay)High (100ms-1000ms+)Moderate (10-100ms)Low-Moderate (5-50ms)Very Low (1-10ms)
    Data PrivacyModerate (Data transmitted over networks)High (Data remains on local networks)Very High (Data processed on-device, not transmitted)Basic-Moderate (Data leaves device)High (Data stays in local network)High (Data stays on phone)Very High (Data never leaves sensor)
    Computational PowerHigh (Utilizes powerful data center infrastructure)Moderate (Utilizes local device capabilities)Low (Limited to the power of the embedded system)Very High (Multiple GPUs/TPUs)High (Edge GPUs)Moderate (Mobile NPUs/GPUs)Very Low (MCU/tiny processors)
    Energy ConsumptionHigh (Data centers consume significant energy)Moderate (Less than data centers, more than TinyML)Low (Highly energy-efficient, designed for low power)Very High (kW-MW range)High (100s W)Moderate (1-10W)Very Low (mW range)
    ScalabilityHigh (Easy to scale with additional server resources)Moderate (Depends on local device capabilities)Low (Limited by the hardware resources of the device)Excellent (virtually unlimited)Good (limited by edge hardware)Moderate (per-device scaling)Limited (fixed hardware)
    CostHigh (Recurring costs for server usage, maintenance)Variable (Depends on the complexity of local setup)Low (Primarily upfront costs for hardware components)High ($1000s+/month)Moderate ($100s-1000s)Low ($0-10s)Very Low ($1-10s)
    ConnectivityHigh (Requires stable internet connectivity)Low (Can operate with intermittent connectivity)Very Low (Can operate without any network connectivity)Connectivity RequiredConstant high-bandwidthIntermittentOptionalNone
    Real-time ProcessingModerate (Can be affected by network latency)High (Capable of real-time processing locally)Very High (Immediate processing with minimal latency)Dependent on networkGoodVery GoodExcellent
    Application ExamplesBig Data Analysis, Virtual AssistantsAutonomous Vehicles, Smart HomesWearables, Sensor NetworksStorage CapacityUnlimited (petabytes+)Large (terabytes)Moderate (gigabytes)Very Limited (kilobytes-megabytes)
    ComplexityModerate to High (Requires knowledge in cloud computing)Moderate (Requires knowledge in local network setup)Moderate to High (Requires expertise in embedded systems)Primary Use CasesBig Data Analytics, Training, Complex AI ModelsSmart Manufacturing, Video Analytics, IoT HubsAR/VR Apps, Mobile Gaming, Photo/Video ProcessingSensor Processing, Gesture Detection, Keyword Spotting
    Development ComplexityHigh (cloud expertise needed)Moderate-High (edge+networking)Moderate (mobile SDKs)High (embedded expertise)
    Deployment SpeedFastModerateFastSlow
    Hardware RequirementsCloud infrastructureEdge servers/gatewaysModern smartphonesMCUs/embedded systems
    Framework SupportAll ML frameworksMost frameworksMobile-optimized (TFLite, CoreML)TinyML frameworks
    Model Size LimitsNoneSeveral GB10s-100s MBBytes-KB range
    Battery ImpactN/AN/AModerateMinimal
    Offline CapabilityNoneGoodExcellentComplete
    @@ -1127,14 +1422,14 @@

    -

    2.6 Conclusion

    +
    +

    2.8 Conclusion

    In this chapter, we’ve offered a panoramic view of the evolving landscape of machine learning, covering cloud, edge, and tiny ML paradigms. Cloud-based machine learning leverages the immense computational resources of cloud platforms to enable powerful and accurate models but comes with limitations, including latency and privacy concerns. Edge ML mitigates these limitations by bringing inference directly to edge devices, offering lower latency and reduced connectivity needs. TinyML takes this further by miniaturizing ML models to run directly on highly resource-constrained devices, opening up a new category of intelligent applications.

    Each approach has its tradeoffs, including model complexity, latency, privacy, and hardware costs. Over time, we anticipate converging these embedded ML approaches, with cloud pre-training facilitating more sophisticated edge and tiny ML implementations. Advances like federated learning and on-device learning will enable embedded devices to refine their models by learning from real-world data.

    The embedded ML landscape is rapidly evolving and poised to enable intelligent applications across a broad spectrum of devices and use cases. This chapter serves as a snapshot of the current state of embedded ML. As algorithms, hardware, and connectivity continue to improve, we can expect embedded devices of all sizes to become increasingly capable, unlocking transformative new applications for artificial intelligence.

    -
    -

    2.7 Resources

    +
    +

    2.9 Resources

    Here is a curated list of resources to support students and instructors in their learning and teaching journeys. We are continuously working on expanding this collection and will be adding new exercises soon.

    diff --git a/docs/index.html b/docs/index.html index 87aac7c6..58a1bd33 100644 --- a/docs/index.html +++ b/docs/index.html @@ -7,7 +7,7 @@ - + Machine Learning Systems