Deploying to gh-pages from @ 0de3c23 🚀

datonic · Dec 11, 2023 · 611d6c3 · 611d6c3
1 parent f73c1be
commit 611d6c3
Show file tree

Hide file tree

Showing 22 changed files with 72 additions and 70 deletions.
diff --git a/README.html b/README.html
@@ -2,7 +2,7 @@
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>
 
 <meta charset="utf-8">
-<meta name="generator" content="quarto-1.4.518">
+<meta name="generator" content="quarto-1.4.525">
 
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 
@@ -94,6 +94,7 @@
     "search-more-match-text": "more match in this document",
     "search-more-matches-text": "more matches in this document",
     "search-clear-button-title": "Clear",
+    "search-text-placeholder": "",
     "search-detached-cancel-button-title": "Cancel",
     "search-submit-button-title": "Submit",
     "search-label": "Search"
@@ -125,7 +126,7 @@
           <div class="collapse navbar-collapse" id="navbarCollapse">
             <ul class="navbar-nav navbar-nav-scroll me-auto">
   <li class="nav-item">
-    <a class="nav-link" href="./docs"> <i class="bi bi-sun" role="img">
+    <a class="nav-link" href="./dbt/docs"> <i class="bi bi-sun" role="img">
 </i> 
 <span class="menu-text">dbt Docs</span></a>
   </li>  
@@ -190,19 +191,20 @@
 <p><img alt="GitHub" src="https://img.shields.io/github/license/davidgasquez/datadex?style=flat-square"> <img alt="GitHub Workflow Status" src="https://img.shields.io/github/actions/workflow/status/davidgasquez/datadex/ci.yml?style=flat-square"> <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/davidgasquez/datadex?style=flat-square"></p>
 </div>
 <p><br></p>
-<p>Open source, serverless, and local-first Data Platform that makes it easy to collaborate on Open Data! Built on top of <a href="https://dagster.io/">Dagster</a>, <a href="https://www.getdbt.com/">dbt</a>, <a href="https://quarto.org/">Quarto</a>, and <a href="https://www.duckdb.org/">DuckDB</a>. You can check two instances of Datadex working in the following repositories:</p>
+<p>Open source, serverless, and local-first Data Platform to collaborate on Open Data! Built on top of <a href="https://dagster.io/">Dagster</a>, <a href="https://www.getdbt.com/">dbt</a>, <a href="https://quarto.org/">Quarto</a>, <a href="https://www.duckdb.org/">DuckDB</a>, and <a href="https://evidence.dev/">Evidence</a>.</p>
+<p>This repository is an up to date toy implementation of the overall pattern. You can check two real world production instances of Datadex working in the following repositories:</p>
 <ul>
 <li><a href="https://github.com/davidgasquez/gitcoin-grants-data-portal">Gitcoin Grants Data Portal</a>. Improve data access and empower data scientists to conduct research and guide community driven analysis and decisions around Gitcoin Grants.</li>
 <li><a href="https://github.com/davidgasquez/filecoin-data-portal/">Filecoin Data Portal</a>. Local-first data hub for Filecoin data! Like Dune, but in your laptop.</li>
 </ul>
 <section id="principles" class="level3">
 <h3 class="anchored" data-anchor-id="principles">💡 Principles</h3>
 <ul>
-<li><strong>Open</strong>: Code and data are fully open. <a href="https://voltrondata.com/codex/a-new-frontier">Use open standards and share data in accesible formats</a>.</li>
-<li><strong>Modular</strong>: Each component can be replaced, extended, or removed. Works well in many environments (your laptop, in a cluster, or from the browser), and with multiple tools (thanks to the Arrow ecosystem).</li>
+<li><strong>Open</strong>: Code and data are fully open source.</li>
+<li><strong>Modular and Interoperable</strong>: Each component can be replaced, extended, or removed. Works well in many environments (your laptop, in a cluster, or from the browser), and with multiple tools (thanks to the Arrow ecosystem). <a href="https://voltrondata.com/codex/a-new-frontier">Use open standards and share data in accesible formats</a>.</li>
 <li><strong>Permissionless</strong>. Don’t ask, fork it and improve the models, add a new source or update any script. No API limits, just plain files.</li>
 <li><strong>Data as Code</strong>. Declarative stateless transformations tracked in <code>git</code>. Version your data as code! Publish and share your reusable models for others to build on top.</li>
-<li><strong>Modern</strong>: Supports types, tests, materialized views, and more.</li>
+<li><strong>Glue</strong>: Be a bridge between tools and aproaches. E.g: Use software engineering good practices like types, tests, materialized views, and more.</li>
 </ul>
 <p>Datadex is not a new tool. <strong>Datadex is a pattern showing an opinionated bridge between existing ones</strong>.</p>
 </section>
@@ -213,7 +215,7 @@ <h3 class="anchored" data-anchor-id="what-can-you-do-with-datadex">🚀 What can
 <li>Explore your data wherever you want. Use Jupyter Notebooks, BI Tools, Excel, …. <a href="https://davidgasquez.github.io/datadex/notebooks/2023-05-06-quarto.html">Share your findings with others</a> by publishing them online as beautiful static websites (thanks to Quarto).</li>
 <li>Add new data sources to the Datadex. The goal is simple; bring data locally and work with it!</li>
 </ul>
-<div class="quarto-figure quarto-figure-center">
+<div class="img-fluid quarto-figure quarto-figure-center">
 <figure class="figure">
 <p><img src="https://user-images.githubusercontent.com/1682202/259458000-92984525-66bc-4410-8cb0-bd1b0cbfaf1d.png" class="img-fluid figure-img"></p>
 <figcaption>Dagster Asset_Group</figcaption>
@@ -226,15 +228,11 @@ <h2 class="anchored" data-anchor-id="setup">⚙️ Setup</h2>
 <section id="docker-dev-containers" class="level3">
 <h3 class="anchored" data-anchor-id="docker-dev-containers">🐳 Docker / Dev Containers</h3>
 <p>The fastest way to start using Datadex is via <a href="https://code.visualstudio.com/docs/remote/containers">VSCode Remote Containers</a>. Once inside the develpment environment, you’ll only need to run <code>make dev</code> to spin up the <a href="http://127.0.0.1:3000">Dagster UI locally</a>.</p>
-<div class="quarto-figure quarto-figure-center">
-<figure class="figure">
-<p><a href="https://codespaces.new/davidgasquez/datadex"><img src="https://github.com/codespaces/badge.svg" class="img-fluid figure-img"></a></p>
-</figure>
-</div>
+<p><a href="https://codespaces.new/davidgasquez/datadex"><img src="https://github.com/codespaces/badge.svg" class="img-fluid"></a></p>
 <p>The development environment can also run in your browser thanks to GitHub Codespaces!</p>
-<p>You can also build the Docker image locally and run it with:</p>
-<div class="sourceCode" id="cb1"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> build</span>
-<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> docker-run</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<p>You can also build the <a href="Dockerfile">Dockerfile</a> image locally and run it with:</p>
+<div class="sourceCode" id="cb1"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">docker</span> build <span class="at">-t</span> datadex .</span>
+<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="ex">docker</span> run <span class="at">-it</span> <span class="at">-v</span> <span class="va">$(</span><span class="bu">pwd</span><span class="va">)</span>:/workspaces/datadex <span class="at">-p</span> 3000:3000 datadex</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </section>
 <section id="python-virtual-environment" class="level3">
 <h3 class="anchored" data-anchor-id="python-virtual-environment">🐍 Python Virtual Environment</h3>

diff --git a/dashboard.html b/dashboard.html
@@ -2,7 +2,7 @@
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>
 
 <meta charset="utf-8">
-<meta name="generator" content="quarto-1.4.518">
+<meta name="generator" content="quarto-1.4.525">
 
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 
@@ -102,6 +102,7 @@
     "search-more-match-text": "more match in this document",
     "search-more-matches-text": "more matches in this document",
     "search-clear-button-title": "Clear",
+    "search-text-placeholder": "",
     "search-detached-cancel-button-title": "Cancel",
     "search-submit-button-title": "Submit",
     "search-label": "Search"
@@ -135,7 +136,7 @@
           <div class="collapse navbar-collapse" id="navbarCollapse">
             <ul class="navbar-nav navbar-nav-scroll me-auto">
   <li class="nav-item">
-    <a class="nav-link" href="./docs"> <i class="bi bi-sun" role="img">
+    <a class="nav-link" href="./dbt/docs"> <i class="bi bi-sun" role="img">
 </i> 
 <span class="menu-text">dbt Docs</span></a>
   </li>  

diff --git a/dbt/catalog.json b/dbt/catalog.json
diff --git a/dbt/graph.gpickle b/dbt/graph.gpickle
diff --git a/dbt/graph_summary.json b/dbt/graph_summary.json
@@ -0,0 +1 @@
+{"_invocation_id": "0fa8a0af-fdb4-4c48-894d-6060a98d1759", "linked": {"0": {"name": "source.datadex.web.energy", "type": "source", "succ": [5]}, "1": {"name": "source.datadex.web.co2_global_trend", "type": "source", "succ": [7]}, "2": {"name": "source.datadex.public.raw_threatened_animal_species", "type": "source", "succ": [6]}, "3": {"name": "source.datadex.public.raw_owid_co2_data", "type": "source", "succ": [4]}, "4": {"name": "model.datadex.climate_owid_co2_by_country", "type": "model"}, "5": {"name": "model.datadex.climate_energy", "type": "model"}, "6": {"name": "model.datadex.climate_theatened_animal_species", "type": "model"}, "7": {"name": "model.datadex.climate_co2_global_trend", "type": "model"}}}
diff --git a/docs/docs/index.html → dbt/index.html b/docs/docs/index.html → dbt/index.html
diff --git a/dbt/manifest.json b/dbt/manifest.json
diff --git a/dbt/run_results.json b/dbt/run_results.json
@@ -0,0 +1 @@
+{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v5.json", "dbt_version": "1.7.3", "generated_at": "2023-12-11T15:31:43.673254Z", "invocation_id": "0fa8a0af-fdb4-4c48-894d-6060a98d1759", "env": {}}, "results": [{"status": "success", "timing": [{"name": "compile", "started_at": "2023-12-11T15:31:43.641686Z", "completed_at": "2023-12-11T15:31:43.657010Z"}, {"name": "execute", "started_at": "2023-12-11T15:31:43.661109Z", "completed_at": "2023-12-11T15:31:43.661123Z"}], "thread_id": "Thread-1 (worker)", "execution_time": 0.02816915512084961, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_co2_global_trend", "compiled": true, "compiled_code": "with\n    source as (select * from 'https://gml.noaa.gov/webdata/ccgg/trends/co2/co2_trend_gl.csv'),\n\n    renamed as (\n        select make_date(year, month, day) as date, trend from source order by 1 asc\n    )\n\nselect *\nfrom renamed", "relation_name": "\"local\".\"main\".\"climate_co2_global_trend\""}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-12-11T15:31:43.648972Z", "completed_at": "2023-12-11T15:31:43.657919Z"}, {"name": "execute", "started_at": "2023-12-11T15:31:43.662338Z", "completed_at": "2023-12-11T15:31:43.662345Z"}], "thread_id": "Thread-2 (worker)", "execution_time": 0.028410911560058594, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_energy", "compiled": true, "compiled_code": "select * from 'https://raw.githubusercontent.com/owid/energy-data/master/owid-energy-data.csv'", "relation_name": "\"local\".\"main\".\"climate_energy\""}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-12-11T15:31:43.651531Z", "completed_at": "2023-12-11T15:31:43.659255Z"}, {"name": "execute", "started_at": "2023-12-11T15:31:43.664714Z", "completed_at": "2023-12-11T15:31:43.664721Z"}], "thread_id": "Thread-3 (worker)", "execution_time": 0.0291597843170166, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_owid_co2_by_country", "compiled": true, "compiled_code": "select country, iso_code, year, co2 from \"local\".\"public\".\"raw_owid_co2_data\"", "relation_name": "\"local\".\"main\".\"climate_owid_co2_by_country\""}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-12-11T15:31:43.654200Z", "completed_at": "2023-12-11T15:31:43.660457Z"}, {"name": "execute", "started_at": "2023-12-11T15:31:43.666350Z", "completed_at": "2023-12-11T15:31:43.666358Z"}], "thread_id": "Thread-4 (worker)", "execution_time": 0.028788089752197266, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_theatened_animal_species", "compiled": true, "compiled_code": "select * from \"local\".\"public\".\"raw_threatened_animal_species\"", "relation_name": "\"local\".\"main\".\"climate_theatened_animal_species\""}], "elapsed_time": 0.2180466651916504, "args": {"use_colors": true, "printer_width": 80, "quiet": false, "cache_selected_only": false, "favor_state": false, "log_level": "info", "log_level_file": "debug", "partial_parse": true, "show_resource_report": false, "compile": true, "strict_mode": false, "indirect_selection": "eager", "log_format_file": "debug", "partial_parse_file_diff": true, "static": false, "use_colors_file": true, "project_dir": "/home/runner/work/datadex/datadex/dbt", "print": true, "log_format": "default", "invocation_command": "dbt docs generate --profiles-dir .", "vars": {}, "select": [], "exclude": [], "introspect": true, "version_check": true, "warn_error_options": {"include": [], "exclude": []}, "profiles_dir": ".", "which": "generate", "macro_debugging": false, "send_anonymous_usage_stats": false, "static_parser": true, "enable_legacy_logger": false, "log_path": "/home/runner/work/datadex/datadex/dbt/logs", "defer": false, "populate_cache": true, "write_json": true, "log_file_max_bytes": 10485760, "empty_catalog": false}}
diff --git a/docs/docs/semantic_manifest.json → dbt/semantic_manifest.json b/docs/docs/semantic_manifest.json → dbt/semantic_manifest.json
@@ -1 +1 @@
-{"semantic_models": [], "metrics": [], "project_configuration": {"time_spine_table_configurations": [], "metadata": null, "dsi_package_version": {"major_version": "0", "minor_version": "4", "patch_version": "1"}}, "saved_queries": []}
+{"semantic_models": [], "metrics": [], "project_configuration": {"time_spine_table_configurations": [], "metadata": null, "dsi_package_version": {"major_version": "0", "minor_version": "4", "patch_version": "2"}}, "saved_queries": []}
diff --git a/docs/docs/catalog.json b/docs/docs/catalog.json
diff --git a/docs/docs/graph.gpickle b/docs/docs/graph.gpickle
diff --git a/docs/docs/graph_summary.json b/docs/docs/graph_summary.json
diff --git a/docs/docs/manifest.json b/docs/docs/manifest.json
diff --git a/docs/docs/run_results.json b/docs/docs/run_results.json
diff --git a/docs/tutorial.html b/docs/tutorial.html
@@ -2,7 +2,7 @@
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>
 
 <meta charset="utf-8">
-<meta name="generator" content="quarto-1.4.518">
+<meta name="generator" content="quarto-1.4.525">
 
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 
@@ -94,6 +94,7 @@
     "search-more-match-text": "more match in this document",
     "search-more-matches-text": "more matches in this document",
     "search-clear-button-title": "Clear",
+    "search-text-placeholder": "",
     "search-detached-cancel-button-title": "Cancel",
     "search-submit-button-title": "Submit",
     "search-label": "Search"
@@ -125,7 +126,7 @@
           <div class="collapse navbar-collapse" id="navbarCollapse">
             <ul class="navbar-nav navbar-nav-scroll me-auto">
   <li class="nav-item">
-    <a class="nav-link" href="../docs"> <i class="bi bi-sun" role="img">
+    <a class="nav-link" href="../dbt/docs"> <i class="bi bi-sun" role="img">
 </i> 
 <span class="menu-text">dbt Docs</span></a>
   </li>
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		{"_invocation_id": "0fa8a0af-fdb4-4c48-894d-6060a98d1759", "linked": {"0": {"name": "source.datadex.web.energy", "type": "source", "succ": [5]}, "1": {"name": "source.datadex.web.co2_global_trend", "type": "source", "succ": [7]}, "2": {"name": "source.datadex.public.raw_threatened_animal_species", "type": "source", "succ": [6]}, "3": {"name": "source.datadex.public.raw_owid_co2_data", "type": "source", "succ": [4]}, "4": {"name": "model.datadex.climate_owid_co2_by_country", "type": "model"}, "5": {"name": "model.datadex.climate_energy", "type": "model"}, "6": {"name": "model.datadex.climate_theatened_animal_species", "type": "model"}, "7": {"name": "model.datadex.climate_co2_global_trend", "type": "model"}}}
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v5.json", "dbt_version": "1.7.3", "generated_at": "2023-12-11T15:31:43.673254Z", "invocation_id": "0fa8a0af-fdb4-4c48-894d-6060a98d1759", "env": {}}, "results": [{"status": "success", "timing": [{"name": "compile", "started_at": "2023-12-11T15:31:43.641686Z", "completed_at": "2023-12-11T15:31:43.657010Z"}, {"name": "execute", "started_at": "2023-12-11T15:31:43.661109Z", "completed_at": "2023-12-11T15:31:43.661123Z"}], "thread_id": "Thread-1 (worker)", "execution_time": 0.02816915512084961, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_co2_global_trend", "compiled": true, "compiled_code": "with\n source as (select * from 'https://gml.noaa.gov/webdata/ccgg/trends/co2/co2_trend_gl.csv'),\n\n renamed as (\n select make_date(year, month, day) as date, trend from source order by 1 asc\n )\n\nselect \nfrom renamed", "relation_name": "\"local\".\"main\".\"climate_co2_global_trend\""}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-12-11T15:31:43.648972Z", "completed_at": "2023-12-11T15:31:43.657919Z"}, {"name": "execute", "started_at": "2023-12-11T15:31:43.662338Z", "completed_at": "2023-12-11T15:31:43.662345Z"}], "thread_id": "Thread-2 (worker)", "execution_time": 0.028410911560058594, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_energy", "compiled": true, "compiled_code": "select from 'https://raw.githubusercontent.com/owid/energy-data/master/owid-energy-data.csv'", "relation_name": "\"local\".\"main\".\"climate_energy\""}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-12-11T15:31:43.651531Z", "completed_at": "2023-12-11T15:31:43.659255Z"}, {"name": "execute", "started_at": "2023-12-11T15:31:43.664714Z", "completed_at": "2023-12-11T15:31:43.664721Z"}], "thread_id": "Thread-3 (worker)", "execution_time": 0.0291597843170166, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_owid_co2_by_country", "compiled": true, "compiled_code": "select country, iso_code, year, co2 from \"local\".\"public\".\"raw_owid_co2_data\"", "relation_name": "\"local\".\"main\".\"climate_owid_co2_by_country\""}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-12-11T15:31:43.654200Z", "completed_at": "2023-12-11T15:31:43.660457Z"}, {"name": "execute", "started_at": "2023-12-11T15:31:43.666350Z", "completed_at": "2023-12-11T15:31:43.666358Z"}], "thread_id": "Thread-4 (worker)", "execution_time": 0.028788089752197266, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_theatened_animal_species", "compiled": true, "compiled_code": "select * from \"local\".\"public\".\"raw_threatened_animal_species\"", "relation_name": "\"local\".\"main\".\"climate_theatened_animal_species\""}], "elapsed_time": 0.2180466651916504, "args": {"use_colors": true, "printer_width": 80, "quiet": false, "cache_selected_only": false, "favor_state": false, "log_level": "info", "log_level_file": "debug", "partial_parse": true, "show_resource_report": false, "compile": true, "strict_mode": false, "indirect_selection": "eager", "log_format_file": "debug", "partial_parse_file_diff": true, "static": false, "use_colors_file": true, "project_dir": "/home/runner/work/datadex/datadex/dbt", "print": true, "log_format": "default", "invocation_command": "dbt docs generate --profiles-dir .", "vars": {}, "select": [], "exclude": [], "introspect": true, "version_check": true, "warn_error_options": {"include": [], "exclude": []}, "profiles_dir": ".", "which": "generate", "macro_debugging": false, "send_anonymous_usage_stats": false, "static_parser": true, "enable_legacy_logger": false, "log_path": "/home/runner/work/datadex/datadex/dbt/logs", "defer": false, "populate_cache": true, "write_json": true, "log_file_max_bytes": 10485760, "empty_catalog": false}}
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		{"semantic_models": [], "metrics": [], "project_configuration": {"time_spine_table_configurations": [], "metadata": null, "dsi_package_version": {"major_version": "0", "minor_version": "4", "patch_version": "1"}}, "saved_queries": []}
		{"semantic_models": [], "metrics": [], "project_configuration": {"time_spine_table_configurations": [], "metadata": null, "dsi_package_version": {"major_version": "0", "minor_version": "4", "patch_version": "2"}}, "saved_queries": []}