Skip to content

Commit

Permalink
refactor(sentinel): restructure program and add Sentinel-5P API integ…
Browse files Browse the repository at this point in the history
…ration

BREAKING CHANGE: complete program architecture rewrite for better maintainability and extensibility

feat:
- Implement Sentinel-5P API data retrieval
- Add token authentication with auto-refresh mechanism
- Implement data download with retry logic
- Add HDF5 data format support

refactor:
- Modularize program structure, separate data fetching and processing
- Enhance error handling and logging system
- Optimize file reading and data processing workflow
- Add code documentation and comments

technical:
- Use requests Session for connection efficiency
- Implement streaming data download
- Add progress bar visualization
- Improve data quality filtering mechanism

improvements:
- Add auto-installation of dependencies
- Enhance NO2 data visualization
- Add statistical analysis functionality
- Optimize memory usage
  • Loading branch information
Alex870521 committed Nov 10, 2024
1 parent 2d2b61d commit 4430c71
Show file tree
Hide file tree
Showing 89 changed files with 1,405 additions and 172 deletions.
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.txt text
*.png binary
103 changes: 103 additions & 0 deletions .github/workflows/cleanup.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# .gitHub/workflows/cleanup.yml
name: Repository Cleanup

on:
workflow_dispatch:
inputs:
action_type:
description: '選擇要執行的操作'
required: true
type: choice
options:
- 'Cleanup Workflow'
- 'Cleanup Deployments'
workflow_status:
description: '要清理的工作流程狀態 (僅在選擇 Cleanup Workflow 時需要)'
required: false
type: choice
options:
- 'disabled' # 已停用的工作流程
- 'active' # 活躍的工作流程
- 'all' # 所有工作流程
environment:
description: '要清理的部署環境 (僅在選擇 Cleanup Deployments 時需要)'
required: false
type: choice
options:
- 'all'
- 'github-pages'
- 'pypi'

jobs:
cleanup-workflows:
if: ${{ github.event.inputs.action_type == 'Cleanup Workflow' }}
runs-on: ubuntu-latest
permissions:
actions: write
steps:
- name: Cleanup workflows
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const status = '${{ github.event.inputs.workflow_status }}';
console.log(`Cleaning up workflows with status: ${status}`);
// 獲取所有工作流程
const workflows = await github.rest.actions.listRepoWorkflows({
owner: context.repo.owner,
repo: context.repo.repo
});
for (const workflow of workflows.data.workflows) {
// 根據選擇的狀態過濾工作流程
if (status === 'all' ||
(status === 'disabled' && !workflow.state === 'active') ||
(status === 'active' && workflow.state === 'active')) {
console.log(`Processing workflow: ${workflow.name} (${workflow.state})`);
// 獲取此工作流程的所有運行
const runs = await github.rest.actions.listWorkflowRuns({
owner: context.repo.owner,
repo: context.repo.repo,
workflow_id: workflow.id,
});
// 刪除運行
console.log(`Found ${runs.data.total_count} runs to delete`);
for (const run of runs.data.workflow_runs) {
console.log(`Deleting run #${run.run_number} of ${workflow.name}`);
await github.rest.actions.deleteWorkflowRun({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: run.id
});
}
}
}
console.log('Cleanup completed');
cleanup-deployments:
if: ${{ github.event.inputs.action_type == 'Cleanup Deployments' }}
runs-on: ubuntu-latest
permissions:
deployments: write
actions: write
contents: write
steps:
- name: Delete github-pages deployments
if: ${{ github.event.inputs.environment == 'github-pages' || github.event.inputs.environment == 'all' }}
uses: strumwolf/delete-deployment-environment@v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
environment: github-pages
onlyRemoveDeployments: true

- name: Delete pypi deployments
if: ${{ github.event.inputs.environment == 'pypi' || github.event.inputs.environment == 'all' }}
uses: strumwolf/delete-deployment-environment@v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
environment: pypi
onlyRemoveDeployments: true
49 changes: 49 additions & 0 deletions .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: Python Tests

on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]

jobs:
test:
strategy:
matrix:
python-version: [ "3.11", "3.12" ]
os: [ ubuntu-latest ]

fail-fast: false

runs-on: ${{ matrix.os }}

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Python 3.XX
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip' # 啟用 pip 緩存加速安裝

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e .
pip install -e ".[test]"
- name: Run tests with coverage
run: |
pytest tests/ -m "not requires_data" \
--cov=AeroViz \
--cov-report=term-missing \
--cov-report=xml \
-v
- name: Upload coverage reports
uses: actions/upload-artifact@v4
with:
name: coverage-report-${{ matrix.python-version }}-${{ github.sha }}
path: coverage.xml
if-no-files-found: error
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@

# MacOX product
.DS_store
temp/

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
Expand Down
Binary file removed Figure/NO2_2021.png
Binary file not shown.
Binary file removed Figure/NO2_202101.png
Binary file not shown.
Binary file removed Figure/NO2_202102.png
Binary file not shown.
Binary file removed Figure/NO2_202103.png
Binary file not shown.
Binary file removed Figure/NO2_202104.png
Binary file not shown.
Binary file removed Figure/NO2_202105.png
Binary file not shown.
Binary file removed Figure/NO2_202106.png
Binary file not shown.
Binary file removed Figure/NO2_202107.png
Binary file not shown.
Binary file removed Figure/NO2_202108.png
Binary file not shown.
Binary file removed Figure/NO2_202109.png
Binary file not shown.
Binary file removed Figure/NO2_202110.png
Binary file not shown.
Binary file removed Figure/NO2_202111.png
Binary file not shown.
Binary file removed Figure/NO2_202112.png
Binary file not shown.
Binary file removed Figure/NO2_2022.png
Binary file not shown.
Binary file removed Figure/NO2_202201.png
Binary file not shown.
Binary file removed Figure/NO2_202202.png
Binary file not shown.
Binary file removed Figure/NO2_202203.png
Binary file not shown.
Binary file removed Figure/NO2_202204.png
Binary file not shown.
Binary file removed Figure/NO2_202205.png
Binary file not shown.
Binary file removed Figure/NO2_202206.png
Binary file not shown.
Binary file removed Figure/NO2_202207.png
Binary file not shown.
Binary file removed Figure/NO2_202208.png
Binary file not shown.
Binary file removed Figure/NO2_202209.png
Binary file not shown.
Binary file removed Figure/NO2_202210.png
Binary file not shown.
Binary file removed Figure/NO2_202211.png
Binary file not shown.
Binary file removed Figure/NO2_202212.png
Diff not rendered.
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2024 Chih-Yu Chan

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Binary file added docs/Satellite.png
Binary file removed ncData/NO2_202101.nc
Binary file not shown.
Binary file removed ncData/NO2_202102.nc
Binary file not shown.
Binary file removed ncData/NO2_202103.nc
Binary file not shown.
Binary file removed ncData/NO2_202104.nc
Binary file not shown.
Binary file removed ncData/NO2_202105.nc
Binary file not shown.
Binary file removed ncData/NO2_202106.nc
Binary file not shown.
Binary file removed ncData/NO2_202107.nc
Binary file not shown.
Binary file removed ncData/NO2_202108.nc
Binary file not shown.
Binary file removed ncData/NO2_202109.nc
Binary file not shown.
Binary file removed ncData/NO2_202110.nc
Binary file not shown.
Binary file removed ncData/NO2_202111.nc
Binary file not shown.
Binary file removed ncData/NO2_202112.nc
Binary file not shown.
Binary file removed ncData/NO2_202201.nc
Binary file not shown.
Binary file removed ncData/NO2_202202.nc
Binary file not shown.
Binary file removed ncData/NO2_202203.nc
Binary file not shown.
Binary file removed ncData/NO2_202204.nc
Binary file not shown.
Binary file removed ncData/NO2_202205.nc
Binary file not shown.
Binary file removed ncData/NO2_202206.nc
Binary file not shown.
Binary file removed ncData/NO2_202207.nc
Binary file not shown.
Binary file removed ncData/NO2_202208.nc
Binary file not shown.
Binary file removed ncData/NO2_202209.nc
Binary file not shown.
Binary file removed ncData/NO2_202210.nc
Binary file not shown.
Binary file removed ncData/NO2_202211.nc
Binary file not shown.
Binary file removed ncData/NO2_202212.nc
Binary file not shown.
153 changes: 0 additions & 153 deletions read_nc.py

This file was deleted.

9 changes: 9 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Cartopy~=0.24.1
numpy~=2.1.2
xarray~=2024.9.0
matplotlib~=3.9.2
geopandas~=1.0.1
scipy~=1.14.1
netCDF4~=1.7.1
pandas~=2.2.3
shapely~=2.0.6
Empty file added src/__init__.py
Empty file.
Empty file added src/api/__init__.py
Empty file.
Loading

0 comments on commit 4430c71

Please sign in to comment.