[Feat] Prometheus metric export

docs/measure/index.md

@@ -116,55 +116,17 @@ if __name__ == "__main__":
 ```
 ## Metric Monitoring
 
-To monitor energy and power consumption effectively using Zeus, Prometheus and the Prometheus Push Gateway must be properly set up. This section outlines the assumptions and provides a guide to configure Prometheus and the Push Gateway. 
+Zeus allows for efficient monitoring of energy and power consumption for GPUs, CPUs, and DRAM using Prometheus. It tracks key metrics such as energy usage, power draw, and cumulative consumption. Users can define measurement windows to track energy usage for specific operations, enabling granular analysis and optimization.
 
-### **Metric Name Construction**
-Zeus organizes metrics using **static metric names** and **dynamic labels** to ensure flexibility and ease of querying in Prometheus. Below, we document how metric names are constructed and how users can query them effectively.
+!!! Assumption
+    A Prometheus Push Gateway must be deployed and accessible at the URL provided in your Zeus configuration. **This ensures that metrics collected by Zeus can be pushed to Prometheus.**
 
-Currently, metric names (e.g., `energy_monitor_gpu_{gpu_index}_energy_joules`) are static and cannot be overridden. However, users can customize the name of the window to define the context of the metrics.
+### Local Setup Guide
 
-- **Metric Name**: `energy_monitor_gpu_{gpu_index}_energy_joules`
-- **Labels**:
-  - `window`: The user-defined window name provided during `begin_window()` and `end_window()` (e.g. energy_histogram.begin_window(f"epoch_{epoch}_energy")).
-  - `index`: The GPU index (e.g., `0` for GPU 0).
-
-
-## How to Query Metrics in Prometheus
-
-### 1. Query Metrics for a Specific Window
-Retrieve energy metrics for a GPU during a specific window:
-```promql
-energy_monitor_gpu_0_energy_joules{window="epoch_1_step_0"}
-```
-```promql
-sum(energy_monitor_gpu_0_energy_joules) by (window)
-```
----
-
-## Assumptions
-
-1. **Prometheus Push Gateway Deployment**  
-   A Prometheus Push Gateway must be deployed and accessible at the URL provided in your Zeus configuration. This ensures that metrics collected by Zeus can be pushed to Prometheus.
-
-2. **Prometheus Configuration**  
-   Prometheus is configured to scrape data from the Push Gateway. This involves adding the Push Gateway URL to the Prometheus `prometheus.yml` configuration file.
-
-3. **Network Accessibility**  
-   If the Push Gateway and Prometheus are hosted on a remote server, ensure that firewall settings allow traffic on the required ports:
-   - **Push Gateway**: Port `9091`
-   - **Prometheus**: Port `9090`
-
-4. **Optional Visualization Tools**  
-   Tools like Grafana can be integrated with Prometheus to create detailed visualizations of the metrics collected.
-
----
-
-## Setup Guide
-
-### Step 1: Install and Start the Prometheus Push Gateway
+#### Step 1: Install and Start the Prometheus Push Gateway
 Choose either Option 1 (Binary) or Option 2 (Docker).
 
-#### Option 1: Download Binary
+##### Option 1: Download Binary
 1. Visit the [Prometheus Push Gateway Download Page](https://prometheus.io/download/#pushgateway).
 2. Download the appropriate binary for your operating system.
 3. Extract the binary:
@@ -178,7 +140,7 @@ Choose either Option 1 (Binary) or Option 2 (Docker).
 ```
 5. Verify the Push Gateway is running by visiting http://localhost:9091 in your browser.
 
-### Option 2: Using Docker
+##### Option 2: Using Docker
 1. Pull the official Prometheus Push Gateway Docker image:
 ```sh
 docker pull prom/pushgateway
@@ -189,7 +151,7 @@ docker run -d -p 9091:9091 prom/pushgateway
 ```
 3. Verify it is running by visiting http://localhost:9091 in your browser.
 
-### Step 2: Install and Configure Prometheus
+#### Step 2: Install and Configure Prometheus
 1. Visit the Prometheus [Prometheus Download Page](https://prometheus.io/download/#prometheus).
 2. Download the appropriate binary for your operating system.
 3. Extract the binary:
@@ -212,16 +174,25 @@ scrape_configs:
 6. Visit http://localhost:9090 in your browser, or use curl http://localhost:9090/api/v1/targets
 7. Verify Prometheus is running by visiting http://localhost:9090 in your browser.
 
-Zeus allows you to monitor energy and power consumption through different metrics, such as Histograms, Counters, and Gauges, which can be pushed to a Prometheus Push Gateway for further analysis. 
+### Metric Name Construction
 
----
+Zeus organizes metrics using **static metric names** and **dynamic labels** for flexibility and ease of querying in Prometheus. Metric names are static and cannot be overridden, but users can customize the context of the metrics by naming the window when using `begin_window()` and `end_window()`.
 
-[`EnergyHistogram`][zeus.metric.EnergyHistogram] records energy consumption data for GPUs, CPUs, and DRAM in Prometheus Histograms. This is ideal for observing how often energy usage falls within specific ranges.
+#### Metric Name
+- For Histogram: `energy_monitor_{component}_energy_joules`
+- For Counter: `energy_monitor_{component}_energy_joules`
+- For Gauge: `power_monitor_gpu_power_watts`
 
-You can customize the bucket ranges for each component (GPU, CPU, and DRAM), or let Zeus use default ranges.
+component: gpu, cpu, or dram
 
-```python hl_lines="2 5-15"
+#### Labels
+- window: The user-defined window name provided during `begin_window()` and `end_window()` (e.g., `energy_histogram.begin_window(f"epoch_energy")`).
+- index: The GPU index (e.g., `0` for GPU 0).
 
+### Usage and Initialization
+[`EnergyHistogram`][zeus.metric.EnergyHistogram] records energy consumption data for GPUs, CPUs, and DRAM in Prometheus Histograms. This is ideal for observing how often energy usage falls within specific ranges.
+
+```python hl_lines="2 5-15"
 from zeus.metric import EnergyHistogram
 
 if __name__ == "__main__":
@@ -235,46 +206,37 @@ if __name__ == "__main__":
 
     for epoch in range(100):
         # Start monitoring energy for the entire epoch
-        energy_histogram.begin_window(f"epoch_{epoch}_energy")
-
-        # Step-level monitoring
-        for step_idx, (x, y) in enumerate(train_loader):
-            energy_histogram.begin_window(f"epoch_{epoch}_step_{step_idx}_energy")
-            train_one_step(x, y)
-            energy_histogram.end_window(f"epoch_{epoch}_step_{step_idx}_energy")
-
-        # Perform epoch-level operations (e.g., aggregation)
+        energy_histogram.begin_window("epoch_energy")
+        # Perform epoch-level operations 
         train_one_epoch(train_loader, model, optimizer, criterion, epoch, args)
         acc1 = validate(val_loader, model, criterion, args)
-
         # End monitoring energy for the epoch
-        energy_histogram.end_window(f"epoch_{epoch}_energy")
-
+        energy_histogram.end_window("epoch_energy")
         print(f"Epoch {epoch} completed. Validation Accuracy: {acc1}%")
 
 ```
 You can use the `begin_window` and `end_window` methods to define a measurement window, similar to other ZeusMonitor operations. Energy consumption data will be recorded for the entire duration of the window.
 
 !!! Tip 
     You can customize the bucket ranges for GPUs, CPUs, and DRAM during initialization to tailor the granularity of energy monitoring. For example:
-
-```python hl_lines="2 5-15"
-energy_histogram = EnergyHistogram(
-    cpu_indices=[0], 
-    gpu_indices=[0], 
-    prometheus_url='http://localhost:9091', 
-    job='training_energy_histogram',
-    gpu_bucket_range = [10.0, 25.0, 50.0, 100.0],
-    cpu_bucket_range = [5.0, 15.0, 30.0, 50.0],
-    dram_bucket_range = [2.0, 8.0, 20.0, 40.0],
-)
-```
+    ```python hl_lines="2 5-15"
+    energy_histogram = EnergyHistogram(
+        cpu_indices=[0], 
+        gpu_indices=[0], 
+        prometheus_url='http://localhost:9091', 
+        job='training_energy_histogram',
+        gpu_bucket_range = [10.0, 25.0, 50.0, 100.0],
+        cpu_bucket_range = [5.0, 15.0, 30.0, 50.0],
+        dram_bucket_range = [2.0, 8.0, 20.0, 40.0],
+    )
+    ```
 
 If no custom `bucket ranges` are specified, Zeus uses these default ranges:
-- **GPU**: `[50.0, 100.0, 200.0, 500.0, 1000.0]`
-- **CPU**: `[10.0, 20.0, 50.0, 100.0, 200.0]`
-- **DRAM**: `[5.0, 10.0, 20.0, 50.0, 150.0]`
-
+```
+- GPU: [50.0, 100.0, 200.0, 500.0, 1000.0]
+- CPU: [10.0, 20.0, 50.0, 100.0, 200.0]
+- DRAM: [5.0, 10.0, 20.0, 50.0, 150.0]
+```
 !!! Warning
     Empty bucket ranges (e.g., []) are not allowed and will raise an error. Ensure you provide a valid range for each device or use the defaults.
 
@@ -345,6 +307,22 @@ if __name__ == "__main__":
 ```
 The `update_period` parameter defines how often the power datas are updated and pushed to Prometheus.
 
+
+### How to Query Metrics in Prometheus
+
+#### Query to View Energy for a Specific Window
+```promql
+energy_monitor_gpu_energy_joules{window="epoch_energy"}
+```
+#### Query to Sum Energy for a Specific Window
+```promql
+sum(energy_monitor_gpu_energy_joules) by (window)
+```
+#### Query to Sum Energy for Specific GPU Across All Windows
+```promql
+sum(energy_monitor_gpu_energy_joules{index="0"})
+```
+
 ## CLI power and energy monitor
 
 The energy monitor measures the total energy consumed by the GPU during the lifetime of the monitor process.

examples/prometheus/README.md

@@ -19,13 +19,13 @@ You just need to download and extract the ImageNet data and mount it to the Dock
     ```
 1. Install `prometheus_client`:
     ```sh
-    pip install prometheus-client
+    pip install zeus-ml[prometheus]
     ```
 
 ## EnergyHistogram, PowerGauge, and EnergyCumulativeCounter
-- [`EnergyHistogram`][zeus.metric.EnergyHistogram]: Records energy consumption data for GPUs, CPUs, and DRAM and pushes the data to Prometheus as histogram metrics. This is useful for tracking energy usage distribution over time.
-- [`PowerGauge`][zeus.metric.PowerGauge]: Monitors real-time GPU power usage and pushes the data to Prometheus as gauge metrics, which are updated at regular intervals.
-- [`EnergyCumulativeCounter`][zeus.metric.EnergyCumulativeCounter]: Tracks cumulative energy consumption over time for CPUs and GPUs and pushes the results to Prometheus as counter metrics.
+- [`EnergyHistogram`](https://ml.energy/zeus/reference/metric/#zeus.metric.EnergyHistogram): Records energy consumption data for GPUs, CPUs, and DRAM and pushes the data to Prometheus as histogram metrics. This is useful for tracking energy usage distribution over time.
+- [`PowerGauge`](https://ml.energy/zeus/reference/metric/#zeus.metric.PowerGauge): Monitors real-time GPU power usage and pushes the data to Prometheus as gauge metrics, which are updated at regular intervals.
+- [`EnergyCumulativeCounter`](https://ml.energy/zeus/reference/metric/#zeus.metric.EnergyCumulativeCounter): Tracks cumulative energy consumption over time for CPUs and GPUs and pushes the results to Prometheus as counter metrics.
 
 ## `ZeusMonitor` and `GlobalPowerLimitOptimizer`
 

examples/prometheus/train_single.py

@@ -19,6 +19,8 @@
 import torchvision.datasets as datasets
 import torchvision.models as models
 from multiprocessing import set_start_method
+from PIL import Image, ImageFile, UnidentifiedImageError
+#set_start_method("fork", force=True)
 
 # ZEUS
 from zeus.monitor import ZeusMonitor
@@ -110,6 +112,20 @@ def parse_args() -> argparse.Namespace:
 
     return parser.parse_args()
 
+ImageFile.LOAD_TRUNCATED_IMAGES = True  # Optionally allow truncated images
+
+def remove_corrupted_images(dataset_dir):
+    """Remove corrupted or truncated image files from the dataset directory."""
+    for root, _, files in os.walk(dataset_dir):
+        for file in files:
+            img_path = os.path.join(root, file)
+            try:
+                with Image.open(img_path) as img:
+                    img.verify()  # Verify if the image is valid
+                    img.convert("RGB")  # Ensure it's in a proper format
+            except (UnidentifiedImageError, OSError):
+                print(f"Removing corrupted or truncated file: {img_path}")
+                os.remove(img_path)
 
 def main():
     """Main function that prepares values and spawns/calls the worker function."""
@@ -136,7 +152,11 @@ def main():
     scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
 
     traindir = os.path.join(args.data, "train")
+    #remove_corrupted_images(traindir)
+
     valdir = os.path.join(args.data, "val")
+    #remove_corrupted_images(valdir)
+
     normalize = transforms.Normalize(
         mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]
     )
@@ -180,64 +200,32 @@ def main():
     val_dataset = Subset(val_dataset, range(2))
 
     ################################## The important part #####################################
-    # ZeusMonitor is used to profile the time and energy consumption of the GPU.
-
-    # EnergyHistogram: Records the energy consumption during specific phases of the program execution
-    # and pushes it to the Prometheus Push Gateway as histogram metrics.
-    energy_histogram = EnergyHistogram(
-        cpu_indices=[0], 
-        gpu_indices=[0], 
-        prometheus_url='http://localhost:9091', 
-        job='training_energy_histogram'
-    )
-
-    # PowerGauge: Monitors real-time power usage of the GPUs and pushes the data to the Prometheus 
-    # Push Gateway as gauge metrics, updated at regular intervals.
-    power_gauge = PowerGauge(
-        gpu_indices=[0], 
-        update_period=2, 
-        prometheus_url='http://localhost:9091', 
-        job='training_power_gauge'
-    )
+    # Histogram to track energy consumption over time
+    energy_histogram = EnergyHistogram(cpu_indices=[0,1], gpu_indices=[0], prometheus_url='http://localhost:9091', job='training_energy_histogram')
+    # Gauge to track power consumption over time
+    power_gauge = PowerGauge(gpu_indices=[0], update_period=2, prometheus_url='http://localhost:9091', job='training_power_gauge')
+    # Counter to track energy consumption over time
+    energy_counter = EnergyCumulativeCounter(cpu_indices=[0,1], gpu_indices=[0], update_period=2, prometheus_url='http://localhost:9091', job='training_energy_counter')
 
-    # EnergyCumulativeCounter: Tracks cumulative energy consumption over time for CPUs and GPUs
-    # and pushes the results to the Prometheus Push Gateway as counter metrics.
-    energy_counter = EnergyCumulativeCounter(
-        cpu_indices=[0], 
-        gpu_indices=[0], 
-        update_period=2, 
-        prometheus_url='http://localhost:9091', 
-        job='training_energy_counter'
-    )
-
-    # Start monitoring real-time power usage.
     power_gauge.begin_window("epoch_power")
-
-    # Start tracking cumulative energy consumption.
     energy_counter.begin_window("epoch_energy")
 
-    # Loop through training epochs while measuring energy and power metrics.
     for epoch in range(args.epochs):
-        # Validate the model and compute accuracy.
         acc1 = validate(val_loader, model, criterion, args)
-
-        # Begin and end energy monitoring for the current epoch.
-        energy_histogram.begin_window(f"epoch_{epoch}_energy")
-        energy_histogram.end_window(f"epoch_{epoch}_energy")
-
+        energy_histogram.begin_window("training_energy")
+        energy_histogram.end_window("training_energy")
+        train(train_loader, model, criterion, optimizer, epoch, args)
         print(f"Top-1 accuracy: {acc1}")
 
-    # Allow metrics to capture remaining data before shutting down monitoring.
     time.sleep(10)
 
-    # End the cumulative energy and power monitoring windows.
     energy_counter.end_window("epoch_energy")
     power_gauge.end_window("epoch_power")
     ################################## The important part #####################################
 
 
 def train(
-    train_loader, model, criterion, optimizer, epoch, args, power_limit_optimizer
+    train_loader, model, criterion, optimizer, epoch, args
 ):
     batch_time = AverageMeter("Time", ":6.3f")
     data_time = AverageMeter("Data", ":6.3f")
@@ -256,7 +244,7 @@ def train(
 
     end = time.time()
     for i, (images, target) in enumerate(train_loader):
-        power_limit_optimizer.on_step_begin()  # Mark the beginning of one training step.
+        #power_limit_optimizer.on_step_begin()  # Mark the beginning of one training step.
 
         # Load data to GPU
         images = images.cuda(args.gpu, non_blocking=True)
@@ -280,7 +268,7 @@ def train(
         loss.backward()
         optimizer.step()
 
-        power_limit_optimizer.on_step_end()  # Mark the end of one training step.
+        #power_limit_optimizer.on_step_end()  # Mark the end of one training step.
 
         # measure elapsed time
         batch_time.update(time.time() - end)
@@ -430,3 +418,4 @@ def accuracy(output, target, topk=(1,)):
 
 if __name__ == "__main__":
     main()
+

pyproject.toml

@@ -46,10 +46,10 @@ pfo-server = ["fastapi[standard]", "pydantic<2", "lowtime", "aiofiles", "torch"]
 bso = ["pydantic<2"]
 bso-server = ["fastapi[standard]", "sqlalchemy", "pydantic<2", "python-dotenv"]
 migration = ["alembic", "sqlalchemy", "pydantic<2", "python-dotenv"]
+prometheus = ["prometheus-client"]
 lint = ["ruff", "black==22.6.0", "pyright", "pandas-stubs", "transformers"]
 test = ["fastapi[standard]", "sqlalchemy", "pydantic<2", "pytest==7.3.2", "pytest-mock==3.10.0", "pytest-xdist==3.3.1", "anyio==3.7.1", "aiosqlite==0.20.0", "numpy<2"]
 docs = ["mkdocs-material[imaging]==9.5.19", "mkdocstrings[python]==0.25.0", "mkdocs-gen-files==0.5.0", "mkdocs-literate-nav==0.6.1", "mkdocs-section-index==0.3.9", "mkdocs-redirects==1.2.1", "urllib3<2", "black"]
-prometheus = ["prometheus-client"]
 # greenlet is for supporting apple mac silicon for sqlalchemy(https://docs.sqlalchemy.org/en/20/faq/installation.html)
 dev = ["zeus-ml[pfo-server,bso,bso-server,migration,prometheus,lint,test]", "greenlet"]