Create Solar Panel Detection (#22)

* Create README.md * add files * Rename notebooks * Update content --------- Co-authored-by: Gabriel Stefanini Vicente <[email protected]>
datapartnership · Feb 2, 2024 · 52a6147 · 52a6147
1 parent 923bf6f
commit 52a6147
Show file tree

Hide file tree

Showing 9 changed files with 2,331 additions and 27 deletions.
diff --git a/docs/_toc.yml b/docs/_toc.yml
@@ -5,8 +5,6 @@ parts:
   - caption: Introduction to Data Goods
     chapters:
       - file: docs/introduction_to_data_goods
-  - caption: Datasets
-    chapters:
       - file: docs/data
   - caption: Data Products
     chapters:
@@ -18,6 +16,9 @@ parts:
         sections:
           - file: notebooks/nighttime-lights/nighttime-lights.ipynb
           - file: notebooks/nighttime-lights/nighttime-lights-in-jordan.ipynb
+      - file: notebooks/solar-panel-segmentation/README
+        sections:
+          - file: notebooks/solar-panel-segmentation/panel-detection-gaza.ipynb
   - caption: Acknowledgements
     chapters:
       - file: docs/team
diff --git a/docs/team.md b/docs/team.md
@@ -8,5 +8,6 @@ The Data Lab would like to express our sincere gratutude and appreciation for th
 | [Jose Manuel Delgado Blasco](mailto:[email protected]) | Data Scientist | DECSC              |
 | [Gabriel Stefanini Vicente](mailto:[email protected])        | Data Scientist | WB Data Lab, DECDG |
 | [Sahiti Sarva](mailto:[email protected])                       | Data Scientist | WB Data Lab, DECDG |
+| [Xiaoqi Tang](mailto:[email protected])                        | Data Scientist | ITSES |
 
 The Data Lab would also like to express our appreciation for Gianluca Mele, Lead Country Economist for Palestine, and Luan Zhao, Senior Economist, who have entrusted our team with using alternative data sources to understand a rapidly changing, complex problem.
diff --git a/notebooks/damage-assessment/README.md b/notebooks/damage-assessment/README.md
@@ -1,46 +1,46 @@
-# Estimating Damage to Buildings and Infrastructure
+# Damage to Buildings and Infrastructure
 
-The situation in Gaza-Israel has escalated since October 2023. To allow World Bank Staff to estimate the damage undertaken to buildings and points of interest, the Data Lab team conducted a damage assessment using satellite Interferometric Synthetic Aperture Radar (InSAR) imagery. 
+The situation in Gaza-Israel has escalated since October 2023. To allow World Bank Staff to estimate the damage undertaken to buildings and points of interest, the Data Lab team conducted a damage assessment using satellite Interferometric Synthetic Aperture Radar (InSAR) imagery.
 
 ## Data
 
-The baseline data used for this analysis is as follows. These data act as an input to damage assessment, or add another layer of information to estimate the impact of damage to infrastructure and people of Gaza. 
+The baseline data used for this analysis is as follows. These data act as an input to damage assessment, or add another layer of information to estimate the impact of damage to infrastructure and people of Gaza.
 
-1. [ACLED](https://acleddata.com): The Armed Conflict Location & Event Data Project (ACLED) is a disaggregated data collection, analysis, and crisis mapping project. ACLED collects information on the dates, actors, locations, fatalities, and types of all reported political violence and protest events around the world. The raw data is available through a license obtained by the World Bank. 
+1. [ACLED](https://acleddata.com): The Armed Conflict Location & Event Data Project (ACLED) is a disaggregated data collection, analysis, and crisis mapping project. ACLED collects information on the dates, actors, locations, fatalities, and types of all reported political violence and protest events around the world. The raw data is available through a license obtained by the World Bank.
 
-2. [WorldPop](https://www.worldpop.org): Estimated population density per grid-cell. The units are number of people per square kilometre based on country totals adjusted to match the corresponding official United Nations population estimates that have been prepared by the Population Division of the Department of Economic and Social Affairs of the United Nations Secretariat (2019 Revision of World Population Prospects).This data was produced in 2020. 
+2. [WorldPop](https://www.worldpop.org): Estimated population density per grid-cell. The units are number of people per square kilometre based on country totals adjusted to match the corresponding official United Nations population estimates that have been prepared by the Population Division of the Department of Economic and Social Affairs of the United Nations Secretariat (2019 Revision of World Population Prospects).This data was produced in 2020.
 
-3. [OpenStreetMap](https://wiki.openstreetmap.org/wiki/About_OpenStreetMap): This is a crowdsourced dataset used to identify Points of Inetrest within Gaza. There is research that suggests that the OSM data for Israel is better than Gaza. However, it is currently the most accessible dataset we can use to estimate if Points of Interest took damage as a result of the attacks. 
+3. [OpenStreetMap](https://wiki.openstreetmap.org/wiki/About_OpenStreetMap): This is a crowdsourced dataset used to identify Points of Inetrest within Gaza. There is research that suggests that the OSM data for Israel is better than Gaza. However, it is currently the most accessible dataset we can use to estimate if Points of Interest took damage as a result of the attacks.
 
-4. [Sentinel-1](https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-1/overview): Satellite imagery forms the core of this analysis. These data are obatined once in two weeks to detect changes to building structures. 
+4. [Sentinel-1](https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-1/overview): Satellite imagery forms the core of this analysis. These data are obatined once in two weeks to detect changes to building structures.
 
-5. [Google Earth Engine](https://earthengine.google.com): The heights of the buildings were obtained from Google Earth Engine. 
+5. [Google Earth Engine](https://earthengine.google.com): The heights of the buildings were obtained from Google Earth Engine.
+
+(damage-assessment-methodology)=
 
-(damage-assessment-methodology)= 
 ## Methodology
 
-The WB Data Lab Lab team developed a multi-step methodology, designed to reduce costs of conducting the analysis while maximizing certainty and frequency of results for damage inventories. Using freely available, bi-weekly satellite radar data, the team has been running a set of automated algorithms to identify significant “changes” to the underlying infrastructure – especially changes in the heights and shapes of features. This analysis is still experimental; it can result in false positives and negatives.   
+The WB Data Lab Lab team developed a multi-step methodology, designed to reduce costs of conducting the analysis while maximizing certainty and frequency of results for damage inventories. Using freely available, bi-weekly satellite radar data, the team has been running a set of automated algorithms to identify significant “changes” to the underlying infrastructure – especially changes in the heights and shapes of features. This analysis is still experimental; it can result in false positives and negatives.
 
-The damage assessment analysis relies on the similarity measure computed using SAR medium resolution and openly accessible [Sentinel-1](https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-1/overview) and the employed methodology can be split into 3 steps: 
+The damage assessment analysis relies on the similarity measure computed using SAR medium resolution and openly accessible [Sentinel-1](https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-1/overview) and the employed methodology can be split into 3 steps:
 
-### 1. Image similarity computation 
+### 1. Image similarity computation
 
-This similarity measurement, namely interferometric coherence ranging from $[0,1]$, provides values of high similarity (usually higher than $0.6$) over structures on that had not suffered almost any variation, as for example buildings or man-made structures, while exhibits lower values (usually lower than $0.4$) over forest and agricultural areas (especially on large time separation between the acquisition time of the satellite data) over water (usually lower than $0.3$) bodies already between consecutive acquisitions. 
+This similarity measurement, namely interferometric coherence ranging from $[0,1]$, provides values of high similarity (usually higher than $0.6$) over structures on that had not suffered almost any variation, as for example buildings or man-made structures, while exhibits lower values (usually lower than $0.4$) over forest and agricultural areas (especially on large time separation between the acquisition time of the satellite data) over water (usually lower than $0.3$) bodies already between consecutive acquisitions.
 
-We have employed all the Copernicus [Sentinel-1](https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-1/overview) data acquired over the Gaza strip, which consists in 3 satellite orbits, namely ascending orbit 87 and 160 and descending orbit 94, and we computed all the similarity maps with respect to the first image available for each of the orbits acquired in September 2022 to have time series measurements that will allow us to use a statistical approach to determine changes using anomaly detection method. Each orbit has 12 days repeat pass, so new updated products can be added regularly every 12 days. 
+We have employed all the Copernicus [Sentinel-1](https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-1/overview) data acquired over the Gaza strip, which consists in 3 satellite orbits, namely ascending orbit 87 and 160 and descending orbit 94, and we computed all the similarity maps with respect to the first image available for each of the orbits acquired in September 2022 to have time series measurements that will allow us to use a statistical approach to determine changes using anomaly detection method. Each orbit has 12 days repeat pass, so new updated products can be added regularly every 12 days.
 
 ```{figure} ../../docs/images/damage-assessment-calendar.png
 ---
 ---
 Calendar with Copernicus Sentinel-1 acquisition dates after the war started on past 7th October 2023 until 9th January 2023. 
 ```
 
-
 ### 2. Change detection based on time series statistics
 
 For the time series change detection, we have computed all the data pre-war September 2022 until end September 2023 to compute the statistics in non-war situation, and to use those statistics to classify the newer data acquired during the war period October 2023 until the present time with pixels for which had been detected a change (potentially attributable to war damage) using anomaly detection method with different thresholds (i.e. 3 sigma rule and 2.5 sigma rule).  
 
-The 3-sigma rule is more conservative and provides more conservative results with false alarms regarding the change detections. 3-sigma rule considers as anomaly values that are lower than the average minus 3 times their standard deviation which it means that are lower than the 99.6% of the value’s normal distribution (measured in non-war conditions) or are included in the 0.15% of possible values, and hence, detected as anomalous. Similarly happens for the 2.5 sigma rule, which pixels are considered as anomaly the ones being the 0.65%. This 2.5 sigma rule may increase some more false alarms, while the 3 sigma rule is considered more conservative anomaly detection rule. See example of this empirical rule below. 
+The 3-sigma rule is more conservative and provides more conservative results with false alarms regarding the change detections. 3-sigma rule considers as anomaly values that are lower than the average minus 3 times their standard deviation which it means that are lower than the 99.6% of the value’s normal distribution (measured in non-war conditions) or are included in the 0.15% of possible values, and hence, detected as anomalous. Similarly happens for the 2.5 sigma rule, which pixels are considered as anomaly the ones being the 0.65%. This 2.5 sigma rule may increase some more false alarms, while the 3 sigma rule is considered more conservative anomaly detection rule. See example of this empirical rule below.
 
 ```{figure} ../../docs/images/damage-assessment-empirical-rule.jpg
 ---
@@ -49,24 +49,23 @@ scale: 50%
 Illustration of the empirical rule
 ```
 
-
 ### 3. Infrastructural damage assessment using the change maps and the vector layers
 
-For the final assessment of infrastructural damage, in roads, points of interest or buildings, the different layers are overlaid and computed whether each feature has been damage or not. For the different features we have computed their potential damage as follows: 
+For the final assessment of infrastructural damage, in roads, points of interest or buildings, the different layers are overlaid and computed whether each feature has been damage or not. For the different features we have computed their potential damage as follows:
 
-- In case of roads, the layer is split into 10 meters roads, and it is computed whether each of the segments had been damaged or not,   
+- In case of roads, the layer is split into 10 meters roads, and it is computed whether each of the segments had been damaged or not,
 
-- In case of the points-of-interest (POI): 
+- In case of the points-of-interest (POI):
 
-    - Point POIs have been attributed a buffer of 10-meter radius and are overlaid to the change map to detect whether they are likely damaged or not. 
+  - Point POIs have been attributed a buffer of 10-meter radius and are overlaid to the change map to detect whether they are likely damaged or not.
 
-    - Area POIs have been overlaid with the change map to detect whether they are likely damaged or not. 
+  - Area POIs have been overlaid with the change map to detect whether they are likely damaged or not.
 
-- In case of buildings, 
+- In case of buildings,
 
-    - Using OpenStreetMap building layer, they are overlaid with the change map and computed which is the percentage (in $[0,1]$ range) of their area which are likely damaged. OSM layers comes also with their possible landuse information that is provided by the OSM layers. 
+  - Using OpenStreetMap building layer, they are overlaid with the change map and computed which is the percentage (in $[0,1]$ range) of their area which are likely damaged. OSM layers comes also with their possible landuse information that is provided by the OSM layers.
 
-    - Using Microsoft footprint layer, they are overlaid with the change map and computed which is the percentage (in $[0,1]$ range) of their area which are likely damaged, but they do not come with landuse information. 
+  - Using Microsoft footprint layer, they are overlaid with the change map and computed which is the percentage (in $[0,1]$ range) of their area which are likely damaged, but they do not come with landuse information.
 
 ## Limitations
 

diff --git a/notebooks/solar-panel-segmentation/README.md b/notebooks/solar-panel-segmentation/README.md
@@ -0,0 +1,4 @@
+
+# Detection of Solar Panels
+
+This section contains a Computer Vision (CV) project aimed at detecting and identifying solar panels in the Gaza Strip. The project utilizes image processing techniques and machine learning to analyze satellite or aerial imagery for the presence of solar panels.
diff --git a/notebooks/solar-panel-segmentation/panel-detection-gaza.ipynb b/notebooks/solar-panel-segmentation/panel-detection-gaza.ipynb
diff --git a/notebooks/solar-panel-segmentation/requirements.txt b/notebooks/solar-panel-segmentation/requirements.txt
@@ -0,0 +1,16 @@
+pytest==5.3.5
+opencv-python-headless==4.6.0.66
+numpy==1.18.1
+scikit_image==0.16.2
+matplotlib==3.1.3
+requests==2.22.0
+tensorflow==2.2.0
+pandas==1.1.4
+Pillow==8.0.1
+detecto==1.2.1
+scikit_learn==0.23.2
+sphinx_rtd_theme==0.5.0
+h5py==2.10.0
+torch==1.9.0
+torchvision==0.10.0
+protobuf~=3.19.0
diff --git a/notebooks/solar-panel-segmentation/setup.cfg b/notebooks/solar-panel-segmentation/setup.cfg
@@ -0,0 +1,7 @@
+[versioneer]
+VCS = git
+style = pep440
+versionfile_source = panel_segmentation/_version.py
+versionfile_build = panel_segmentation/_version.py
+tag_prefix = ''
+parentdir_prefix = panel_segmentation-
diff --git a/notebooks/solar-panel-segmentation/setup.py b/notebooks/solar-panel-segmentation/setup.py
@@ -0,0 +1,76 @@
+try:
+    from setuptools import setup
+except ImportError:
+    raise RuntimeError('setuptools is required')
+
+import versioneer
+
+KEYWORDS = [
+    'photovoltaic',
+    'solar',
+    'analytics',
+    'analysis',
+    'performance',
+    'PV'
+    'satellite'
+]
+
+
+INSTALL_REQUIRES = [
+    'opencv-python-headless==4.6.0.66',
+    'numpy>=1.18.1',
+    'scikit_image>=0.16.2',
+    'matplotlib>=3.1.3',
+    'requests>=2.22.0',
+    'tensorflow>=2.2.0',
+    'pandas>=1.1.4',
+    'Pillow>=8.0.1',
+    'scikit_learn>=0.23.2',
+    'h5py>=2.10.0',
+    'detecto>=1.2.1',
+    'torch>=1.9.0',
+    'torchvision>=0.10.0'
+]
+
+TESTS_REQUIRE = [
+    'pytest>=5.3.5'
+]
+
+EXTRAS_REQUIRE = {
+    'doc': [
+        'sphinx==3.2',
+        'jinja2<3.1',
+        'sphinx_rtd_theme==0.5.2',
+        'ipython'
+    ],
+    'test': TESTS_REQUIRE
+}
+
+
+setup(
+    name='panel_segmentation',
+    version=versioneer.get_version(),
+    install_requires=INSTALL_REQUIRES,
+    tests_require=TESTS_REQUIRE,
+    extras_require=EXTRAS_REQUIRE,
+    description='A package to segment solar panels from a '
+                'satellite image and perform automated metadata extraction.',
+    url='https://github.com/NREL/Panel-Segmentation',
+    keywords=KEYWORDS,
+    author='Ayobami Edun, Kirsten Perry, Kevin Anderson, Christopher Campos',
+    author_email='[email protected]; [email protected]; '
+                '[email protected]; [email protected]',
+    package_data={
+        'panel_segmentation': [
+            'panel_segmentation/examples/*',
+            'panel_segmentation/tests/*',
+            'panel_segmentation/models/VGG16Net_ConvTranpose_complete.h5',
+            'panel_segmentation/models/VGG16_classification_model.h5',
+            'panel_segmentation/models/object_detection_model.pth'
+        ],
+    },
+    include_package_data=True,
+    license='MIT',
+    packages=['panel_segmentation'],
+    zip_safe=False
+)
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,4 @@

		# Detection of Solar Panels

		This section contains a Computer Vision (CV) project aimed at detecting and identifying solar panels in the Gaza Strip. The project utilizes image processing techniques and machine learning to analyze satellite or aerial imagery for the presence of solar panels.