worldbank · andresfchamorro · Dec 2, 2024 · Nov 15, 2024 · Nov 15, 2024 · Nov 4, 2024
diff --git a/.github/workflows/gh-pages.yml b/.github/workflows/gh-pages.yml
@@ -1,30 +1,44 @@
 name: Publish to GitHub Pages
-
 on:
   push:
     branches:
       - main
 
 jobs:
-  build:
+  build-book:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
-          python-version: "3.10"
+          python-version: "3.11"
           cache: "pip"
-      - name: Dependencies
+      - name: Install Dependencies
+        working-directory: space2stats_api/src
         run: |
-          if [ -f docs/requirements.txt ]; then pip install -r docs/requirements.txt; fi
-      - name: Build Jupyter Book
+          if [ -f pyproject.toml ]; then
+            pip install poetry && poetry install --with docs
+          fi
+      - name: Build Jupyter Book using Sphinx
         run: |
-          jupyter-book build . --config docs/_config.yml --toc docs/_toc.yml
-      - name: Deploy
-        uses: peaceiris/actions-gh-pages@v3
-        if: github.ref == 'refs/heads/main' && job.status == 'success'
+          sphinx-build docs _build/html -b html
+      - name: Upload artifact
+        uses: actions/upload-pages-artifact@v3
+        if: job.status == 'success'
         with:
-          github_token: ${{ secrets.GITHUB_TOKEN }}
-          publish_dir: ./_build/html
-          enable_jekyll: false
+          path: "_build/html"
+
+  deploy-book:
+    needs: build-book
+    permissions:
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4
diff --git a/README.md b/README.md
@@ -190,4 +190,63 @@ This method is helpful for understanding the available fields that can be querie
 with StatsTable.connect() as stats_table:
     columns = stats_table.fields()
     print(columns)  # Output: ['population', 'average_income', ...]
+```
+
+## Documentation
+
+Our documentation is hosted at https://worldbank.github.io/DECAT_Space2Stats/, and is built using Jupyter Book and Sphinx. 
+
+All configuration, markdown, and notebook files relevant for this are stored under `/docs`. 
+
+### Environment Setup
+
+To install the package and docs dependencies:
+
+```bash
+conda create -n book python=3.11
+conda activate book
+pip install poetry
+cd space2stats_api/src
+poetry install --with docs # Installs space2stats, dependencies and jupyter-book
+pip install folium matplotlib mapclassify # for additional notebook visualizations
+```
+
+To build the documentation locally, run the following command:
+
+```bash
+sphinx-build docs _build/html -b html
+```
+
+To remove the generated files:
+
+```bash
+jupyter-book clean .
+```
+
+### Contributing
+
+A Github Action is used to automatically build and deploy the documentation to Github Pages. To contribute to the documentation, follow the steps below:
+
+1. Create a new branch from the latest `main`.
+
+    ```bash
+    git checkout -b new_docs
+    ```
+
+2. Make changes to the documentation, e.g: markdown files, and table of contents (`docs/_toc.yml`).
+3. Build the documentation locally to ensure it renders correctly.
+4. Commit, push the changes to your branch and create a pull request.
+
+    ```bash
+    git add .
+    git commit -m "Update documentation"
+    git push origin new_docs
+    ```
+
+The site will be updated automatically once the branch is merged to main.
+
+Note that the sphinx build command uses the conf.py file. If we need to make changes to _conf.yml, then rebuild the conf.py file by running:
+
+```bash
+jupyter-book config sphinx docs
 ```
diff --git a/docs/_config.yml b/docs/_config.yml
@@ -1,7 +1,7 @@
 # Book settings
-title:
+title: DECAT Space2Stats
 author: World Bank
-logo: docs/images/logo.png
+logo: images/logo.png
 only_build_toc_files: true
 
 repository:
@@ -15,7 +15,7 @@ html:
   extra_navbar: ""
   use_edit_page_button: true
   use_repository_button: true
-  use_issues_button: true
+  use_issues_button: false
   baseurl: https://github.com/worldbank/DECAT_Space2Stats
   extra_footer: |
     <div>
@@ -25,9 +25,6 @@ html:
 # Execution settings
 execute:
   execute_notebooks: off
-  allow_errors: true
-  exclude_patterns:
-    - notebooks/*.ipynb
 
 #######################################################################################
 # Sphinx settings
@@ -36,7 +33,6 @@ sphinx:
     html_show_copyright: false
     html_last_updated_fmt: "%b %d, %Y"
     apidoc_module_dir: ../space2stats_api/src
-  extra_extensions:
-  - 'sphinx.ext.autodoc'
-  - sphinx.ext.napoleon
-  - sphinxcontrib.apidoc
+    extra_extensions:
+      - sphinx.ext.autodoc
+      - sphinx.ext.napoleon
diff --git a/docs/_toc.yml b/docs/_toc.yml
@@ -1,26 +1,26 @@
 format: jb-book
-root: docs/readme.md
+root: readme.md
 
 parts:
   - caption: Concept Note
     numbered: False
     chapters:
-      - file: docs/Introduction.md
-      - file: docs/annexA.md    
+      - file: Introduction.md
+      - file: annexA.md    
   - caption: S2S Sub-tasks
     numbered: False
     chapters:
-        - file: docs/sub_tasks/admin_bounds.md
-        - file: docs/sub_tasks/sub_national_poverty.md
-        - file: docs/sub_tasks/geest.md
-        - file: docs/sub_tasks/geo_enhancement.md
+        - file: sub_tasks/admin_bounds.md
+        - file: sub_tasks/sub_national_welfare.md
+        - file: sub_tasks/geest.md
+        - file: sub_tasks/geo_enhancement.md
   - caption: S2S Examples
     numbered: False
     chapters:
-        - file: notebooks/user-docs/space2stats_py_package_demo.ipynb
-        - file: notebooks/user-docs/space2stats_api_demo.ipynb
-        - file: notebooks/user-docs/space2stats_api_demo_R.Rmd
+        - file: user-docs/space2stats_py_package_demo.ipynb
+        - file: user-docs/space2stats_api_demo.ipynb
+        - file: user-docs/space2stats_api_demo_R.Rmd
   - caption: S2S Python Documentation
     numbered: False
     chapters:
-        - file: docs/api.rst
+        - file: api.rst
diff --git a/docs/annexA.md b/docs/annexA.md
@@ -1,31 +1,30 @@
 # Annex A: S2S Datasets
 |Phase|Category|Variable Name|Source|
 |:----|:----|:----|:----|
-|1|Access|Distance to airport  |Global Friction Surface 2019|
-|1|Access|Distance to market  |Global Friction Surface 2019|
-|1|Access|Distance to port  |Global Friction Surface 2019|
-|1|Climate|Precipitation  |TBD (see here)|
-|1|Climate|Temperature  |TBD (see here)|
 |1|Disaster|Flood|Fathom models|
-|1|Disaster|Severe storm|IBTRACS|
-|1|Economics|GDP (gridded)|UNEP and WB|
-|1|Economics|Ag-GDP|IFPRI and WB|
 |1|Economics|Nightlights  |Lights Every Night|
-|1|Environment|Deforestation  |Global forest watch|
-|1|Environment|Estimated biomass|Above ground biomass|
-|1|Environment|Reforestation  |Global forest watch|
-|1|Environment|Vegetation indices  |Sentinel/Landsat|
-|1|Environment|Terrestrial biodiversity indicators|WB|
-|1|Infrastructure|Road density/road length|OpenStreetMap|
-|1|Physical|Elevation|SRTM 30|
-|1|Physical|Slope  |SRTM 30|
-|1|Social|Conflict events|ACLED|
-|1|Social|Population  |GHS-Pop|
+|1|Social|Population  |WorldPop Demographics|
 |1|Social|Urbanization|GHS-SMOD|
-|2|Climate|XCO2 emissions|Dasgupta, S., S. Lall, D. Wheeler. 2021|
-|2|Climate|Methane emissions  |Sentinel 5P|
-|2|Climate|NO2 emissions  |Sentinel 5P|
-|2|Climate|Heat Waves|Benny's Method|
-|2|Infrastructure|Building statistics (orientation, size, alignment)|Digitize Africa/Google Buildings/Microsoft Buildings|
+|2|Access|Distance to airport  |Global Friction Surface 2019|
+|2|Access|Distance to market  |Global Friction Surface 2019|
+|2|Access|Distance to port  |Global Friction Surface 2019|
+|2|Climate|Precipitation  |TBD (see here)|
+|2|Climate|Temperature  |TBD (see here)|
+|2|Disaster|Severe storm|IBTRACS|
+|2|Environment|Deforestation  |Global forest watch|
+|2|Environment|Terrestrial biodiversity indicators|WB|
+|3|Economics|GDP (gridded)|UNEP and WB|
+|3|Economics|Ag-GDP|IFPRI and WB|
+|3|Environment|Estimated biomass|Above ground biomass|
+|3|Environment|Vegetation indices  |Sentinel/Landsat|
+|3|Infrastructure|Road density/road length|OpenStreetMap|
+|3|Physical|Elevation|SRTM 30|
+|3|Physical|Slope  |SRTM 30|
+|3|Social|Conflict events|ACLED|
+|3|Climate|XCO2 emissions|Dasgupta, S., S. Lall, D. Wheeler. 2021|
+|3|Climate|Methane emissions  |Sentinel 5P|
+|3|Climate|NO2 emissions  |Sentinel 5P|
+|3|Climate|Heat Waves|Benny's Method|
+|3|Infrastructure|Building statistics (orientation, size, alignment)|Digitize Africa/Google Buildings/Microsoft Buildings|
 |3|Climate|Climate change predictions  |WB CCKP|
 |3|Economics|Trade volumes (AIS)|AIS|
diff --git a/docs/api.rst b/docs/api.rst
@@ -0,0 +1,7 @@
+API Reference
+=============
+
+.. automodule:: space2stats.lib
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/docs/conf.py b/docs/conf.py
@@ -0,0 +1,83 @@
+###############################################################################
+# Auto-generated by `jupyter-book config`
+# If you wish to continue using _config.yml, make edits to that file and
+# re-generate this one.
+###############################################################################
+apidoc_module_dir = "../space2stats_api/src"
+author = "World Bank"
+comments_config = {"hypothesis": False, "utterances": False}
+copyright = "2023"
+exclude_patterns = ["**.ipynb_checkpoints", ".DS_Store", "Thumbs.db", "_build"]
+extensions = [
+    "sphinx_togglebutton",
+    "sphinx_copybutton",
+    "myst_nb",
+    "jupyter_book",
+    "sphinx_thebe",
+    "sphinx_comments",
+    "sphinx_external_toc",
+    "sphinx.ext.intersphinx",
+    "sphinx_design",
+    "sphinx_book_theme",
+    "sphinx_jupyterbook_latex",
+    "sphinx_multitoc_numbering",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.napoleon",
+]
+external_toc_exclude_missing = True
+external_toc_path = "_toc.yml"
+extra_extensions = ["sphinx.ext.autodoc", "sphinx.ext.napoleon"]
+html_baseurl = "https://github.com/worldbank/DECAT_Space2Stats"
+html_favicon = ""
+html_last_updated_fmt = "%b %d, %Y"
+html_logo = "images/logo.png"
+html_show_copyright = False
+html_sourcelink_suffix = ""
+html_theme = "sphinx_book_theme"
+html_theme_options = {
+    "search_bar_text": "Search this book...",
+    "launch_buttons": {
+        "notebook_interface": "classic",
+        "binderhub_url": "",
+        "jupyterhub_url": "",
+        "thebe": False,
+        "colab_url": "",
+        "deepnote_url": "",
+    },
+    "path_to_docs": "",
+    "repository_url": "https://github.com/worldbank/DECAT_Space2Stats",
+    "repository_branch": "main",
+    "extra_footer": '<div>\n    <b>All content (unless otherwise specified) is subject to the <a href="https://raw.githubusercontent.com/worldbank/template/main/LICENSE">World Bank Master Community License Agreement.</a></b>\n</div>\n',
+    "home_page_in_toc": True,
+    "announcement": "",
+    "analytics": {
+        "google_analytics_id": "",
+        "plausible_analytics_domain": "",
+        "plausible_analytics_url": "https://plausible.io/js/script.js",
+    },
+    "use_repository_button": True,
+    "use_edit_page_button": True,
+    "use_issues_button": False,
+}
+html_title = "DECAT Space2Stats"
+latex_engine = "pdflatex"
+myst_enable_extensions = [
+    "colon_fence",
+    "dollarmath",
+    "linkify",
+    "substitution",
+    "tasklist",
+]
+myst_url_schemes = ["mailto", "http", "https"]
+nb_execution_allow_errors = False
+nb_execution_cache_path = ""
+nb_execution_excludepatterns = []
+nb_execution_in_temp = False
+nb_execution_mode = "off"
+nb_execution_timeout = 30
+nb_output_stderr = "show"
+numfig = True
+pygments_style = "sphinx"
+suppress_warnings = ["myst.domains"]
+use_jupyterbook_latex = True
+use_multitoc_numbering = True
diff --git a/docs/images/GDF_logo.png b/docs/images/GDF_logo.png
diff --git a/docs/images/WEE_LCA.jpg b/docs/images/WEE_LCA.jpg
diff --git a/docs/images/WEE_PNG.png b/docs/images/WEE_PNG.png
diff --git a/docs/images/boundary_comparison_VNM.png b/docs/images/boundary_comparison_VNM.png
diff --git a/docs/images/geo-enhancement-pipeline.png b/docs/images/geo-enhancement-pipeline.png
diff --git a/docs/readme.md b/docs/readme.md
@@ -0,0 +1,12 @@
+# Space2Stats
+The Space2Stats program is designed to provide academics, statisticians, and other interested data scientists with easier access to regularly requested geospatial aggregate data. The primary deliverable is a database of geospatial aggregates at two official scales:
+1. Official World Bank boundaries at admin level 2  
+2. A global database of h3 hexagons at level 6 (~36km2)  
+
+The Space2Stats program is funded by the World Bank's [Global Data Facility](https://www.worldbank.org/en/programs/global-data-facility), which is a World-Bank hosted global funding instrument for the world's most critical data impact opportunities.
+
+```{image} images/GDF_logo.png
+---
+alt: GDF
+---
+```
diff --git a/docs/sub_tasks/admin_bounds.md b/docs/sub_tasks/admin_bounds.md
@@ -0,0 +1,16 @@
+# Official World Bank Boundaries
+The WBG produces many datasets at the global scale, often in complex geospatial formats. These datasets need to be summarized and aggregated at a smaller administrative level to provide meaningful results – a process which requires a reliable and up-to-date set of global administrative boundaries.  
+
+The Global Administrative Boundaries subtask (P501733) of the Space2Stats program (P180913) seeks to obtain, disseminate, and update a new high-resolution global boundary dataset at Levels 0 (national), 1 (state/province), and 2 (district). Moving forward, the files will have documented versioning as well as a detailed and well-defined coding system so that the boundaries can be easily joined with tabular data. The licensing on the high-resolution files will be public, creating a global public good that can be used by other organizations in the international development community.  
+
+To illustrate the difference between the high and medium resolution products, the following images show the high resolution Administrative 2 boundaries (in blue, digitized at a 1:250,000 scale) and medium resolution (in red, digitized at a 1:1,000,000 scale). While acceptable for regional or global mapping, medium resolution is of substantively lower quality when viewing a small area.  This is especially noteworthy when summarizing other global datasets by administrative area, as the lower-resolution boundaries can significantly affect the results.  
+
+```{figure} ../images/boundary_comparison_VNM.png
+---
+alt: Boundary comparison 
+---
+Differences in administrative boundary resolution around Hanoi, Vietnam. High resolution is blue and medium is red
+```
+The leadership of this project is shared between the DECAT Global Operations Support Team (GOST) and the GCS Cartography Unit. With the amount of data that DECAT produces, the team has a vested interest in ensuring the boundary files needed for summarizing datasets are reliable. Meanwhile, the Cartography Unit currently holds the responsibility of maintaining and hosting the official WBG GAD and distributing it via the Development Data Hub (DDH). 
+
+In summer 2024, the contract for the production of the new high-resolution boundaries was awarded to IMA, a vendor with extensive experience in producing legally-approved boundary datasets for international use. The draft dataset was delivered in October 2024, edited by the WBG’s legal department, and sent back to IMA for a few small edits. The final legally-approved dataset is expected by late November 2024. The boundaries will then be placed on DDH and made publicly available, while the team then focuses on the editing and update plan.