Skip to content

Commit

Permalink
Deploying to gh-pages from @ b706031 🚀
Browse files Browse the repository at this point in the history
  • Loading branch information
davidgasquez committed Nov 9, 2023
1 parent 5129b0d commit 8d7e6b4
Show file tree
Hide file tree
Showing 14 changed files with 155 additions and 52 deletions.
64 changes: 58 additions & 6 deletions README.html
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>

<meta charset="utf-8">
<meta name="generator" content="quarto-1.4.487">
<meta name="generator" content="quarto-1.4.489">

<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">

Expand All @@ -20,6 +20,40 @@
margin: 0 0.8em 0.2em -1em; /* quarto-specific, see https://github.com/quarto-dev/quarto-cli/issues/4556 */
vertical-align: middle;
}
/* CSS for syntax highlighting */
pre > code.sourceCode { white-space: pre; position: relative; }
pre > code.sourceCode > span { line-height: 1.25; }
pre > code.sourceCode > span:empty { height: 1.2em; }
.sourceCode { overflow: visible; }
code.sourceCode > span { color: inherit; text-decoration: inherit; }
div.sourceCode { margin: 1em 0; }
pre.sourceCode { margin: 0; }
@media screen {
div.sourceCode { overflow: auto; }
}
@media print {
pre > code.sourceCode { white-space: pre-wrap; }
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
}
pre.numberSource code
{ counter-reset: source-line 0; }
pre.numberSource code > span
{ position: relative; left: -4em; counter-increment: source-line; }
pre.numberSource code > span > a:first-child::before
{ content: counter(source-line);
position: relative; left: -1em; text-align: right; vertical-align: baseline;
border: none; display: inline-block;
-webkit-touch-callout: none; -webkit-user-select: none;
-khtml-user-select: none; -moz-user-select: none;
-ms-user-select: none; user-select: none;
padding: 0 4px; width: 4em;
}
pre.numberSource { margin-left: 3em; padding-left: 4px; }
div.sourceCode
{ }
@media screen {
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
}
</style>


Expand Down Expand Up @@ -183,19 +217,37 @@ <h3 class="anchored" data-anchor-id="what-can-you-do-with-datadex">🚀 What can
</section>
<section id="setup" class="level2">
<h2 class="anchored" data-anchor-id="setup">⚙️ Setup</h2>
<p>The fastest way to start using Datadex is via <a href="https://code.visualstudio.com/docs/remote/containers">VSCode Remote Containers</a>. Once inside the develpment environment, you’ll only need to run <code>make deps</code>.</p>
<p>Datadex consists of several components and requires some setup to get started.</p>
<section id="docker-dev-containers" class="level3">
<h3 class="anchored" data-anchor-id="docker-dev-containers">🐳 Docker / Dev Containers</h3>
<p>The fastest way to start using Datadex is via <a href="https://code.visualstudio.com/docs/remote/containers">VSCode Remote Containers</a>. Once inside the develpment environment, you’ll only need to run <code>make dev</code> to spin up the <a href="http://127.0.0.1:3000">Dagster UI locally</a>.</p>
<p><a href="https://codespaces.new/davidgasquez/datadex"><img src="https://github.com/codespaces/badge.svg" class="img-fluid"></a></p>
<p>PS: The development environment can also run in your browser thanks to GitHub Codespaces!</p>
<p>The development environment can also run in your browser thanks to GitHub Codespaces!</p>
<p>You can also build the Docker image locally and run it with:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> build</span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> docker-run</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</section>
<section id="python-virtual-environment" class="level3">
<h3 class="anchored" data-anchor-id="python-virtual-environment">🐍 Python Virtual Environment</h3>
<p>Clone the repository and run the following commands from the root folder:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Create a virtual environment</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="ex">python3</span> <span class="at">-m</span> venv .venv</span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="bu">source</span> .venv/bin/activate</span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a><span class="co"># Install the package and dependencies</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a><span class="ex">pip</span> install <span class="at">-e</span> .<span class="pp">[</span><span class="ss">dev</span><span class="pp">]</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
<p>Now, you should be able to spin up Dagster UI and <a href="http://127.0.0.1:3000">access it locally</a>.</p>
</section>
</section>
<section id="motivation" class="level2">
<h2 class="anchored" data-anchor-id="motivation">🎯 Motivation</h2>
<p>This small project was created after <a href="https://publish.obsidian.md/davidgasquez/Open+Data">thinking how an Open Data Protocol could look like</a>!</p>
<p>This project started after <a href="https://publish.obsidian.md/davidgasquez/Open+Data">thinking how an Open Data Protocol could look like</a>!</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">👏 Acknowledgements</h2>
<ul>
<li>This proof of concept was created thanks to open source projects like <a href="https://www.duckdb.org/">DuckDB</a> and <a href="https://getdbt.com">dbt</a>.</li>
<li>Datadex name was inspired by <a href="https://juan.benet.ai/blog/2014-03-11-discussion-scienceexchange/">Juan Benet awesome <code>data</code> projects</a>.</li>
<li>This proof of concept was created thanks to open source projects like <a href="https://www.duckdb.org/">DuckDB</a>, <a href="https://getdbt.com">dbt</a>, <a href="https://dagster.io/">Dagster</a>, and <a href="https://quarto.org/">Quarto</a>.</li>
<li>Datadex name was inspired by <a href="https://juan.benet.ai/blog/2014-03-11-discussion-scienceexchange/">Juan Benet’s <code>data</code> projects</a>.</li>
</ul>


Expand Down
1 change: 0 additions & 1 deletion country-data.json

This file was deleted.

44 changes: 22 additions & 22 deletions dashboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>

<meta charset="utf-8">
<meta name="generator" content="quarto-1.4.487">
<meta name="generator" content="quarto-1.4.489">

<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">

Expand Down Expand Up @@ -110,24 +110,7 @@

<body class="quarto-dashboard nav-fixed fullcontent">

<header>
<nav class="navbar navbar-expand-md slim" data-bs-theme="dark">
<div class="navbar-container container-fluid">
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#dashboard-collapse" aria-controls="dashboard-collapse" aria-expanded="true" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="navbar-brand-container">

<div class="navbar-title">
<div class="navbar-title-text"><a href="#">GreyNoise Country Explorer</a></div>

</div>
</div>
<div id="dashboard-collapse" class="navbar-collapse collapse"><ul class="navbar-nav navbar-nav-scroll me-auto" role="tablist"><li class="nav-item" role="presentation"><a id="tab-geo-attributed-attacks" class="nav-link active" data-bs-toggle="tab" role="tab" data-bs-target="#geo-attributed-attacks" href="#geo-attributed-attacks" aria-controls="geo-attributed-attacks" aria-selected="true"><span class="nav-link-text">Geo-Attributed Attacks</span></a></li><li class="nav-item" role="presentation"><a id="tab-malicious-activity" class="nav-link" data-bs-toggle="tab" role="tab" data-bs-target="#malicious-activity" href="#malicious-activity" aria-controls="malicious-activity" aria-selected="false"><span class="nav-link-text">Malicious Activity</span></a></li><li class="nav-item" role="presentation"><a id="tab-about" class="nav-link" data-bs-toggle="tab" role="tab" data-bs-target="#about" href="#about" aria-controls="about" aria-selected="false"><span class="nav-link-text">About</span></a></li></ul></div></div>


</nav>
</header>

<div id="quarto-search-results"></div>
<header id="quarto-header" class="headroom fixed-top">
Expand Down Expand Up @@ -184,7 +167,24 @@
</div>
</div> <!-- /container-fluid -->
</nav>
</header>
<header id="quarto-dashboard-header">
<nav class="navbar navbar-expand-md slim" data-bs-theme="dark">
<div class="navbar-container container-fluid">
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#dashboard-collapse" aria-controls="dashboard-collapse" aria-expanded="true" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="navbar-brand-container">

<div class="navbar-title">
<div class="navbar-title-text"><a href="#">GreyNoise Country Explorer</a></div>

</div>
</div>
<div id="dashboard-collapse" class="navbar-collapse collapse"><ul class="navbar-nav navbar-nav-scroll me-auto" role="tablist"><li class="nav-item" role="presentation"><a id="tab-geo-attributed-attacks" class="nav-link active" data-bs-toggle="tab" role="tab" data-bs-target="#geo-attributed-attacks" href="#geo-attributed-attacks" aria-controls="geo-attributed-attacks" aria-selected="true"><span class="nav-link-text">Geo-Attributed Attacks</span></a></li><li class="nav-item" role="presentation"><a id="tab-malicious-activity" class="nav-link" data-bs-toggle="tab" role="tab" data-bs-target="#malicious-activity" href="#malicious-activity" aria-controls="malicious-activity" aria-selected="false"><span class="nav-link-text">Malicious Activity</span></a></li><li class="nav-item" role="presentation"><a id="tab-about" class="nav-link" data-bs-toggle="tab" role="tab" data-bs-target="#about" href="#about" aria-controls="about" aria-selected="false"><span class="nav-link-text">About</span></a></li></ul></div></div>


</nav>
</header></header>
<!-- content -->
<div id="quarto-content" class="quarto-container page-layout-custom page-navbar quarto-dashboard-content bslib-gap-spacing html-fill-container dashboard-scrolling">
<!-- sidebar -->
Expand Down Expand Up @@ -229,7 +229,7 @@
</div>
</div>
<div class="bslib-grid-item html-fill-item tab-content bslib-grid" data-layout="flow" style="display: grid; grid-template-rows: minmax(3em, max-content); grid-auto-columns: minmax(0, 1fr);">
<div id="geo-attributed-attacks" class="dashboard-page tab-pane show active html-fill-item html-fill-container" data-orientation="rows" data-title="Geo-Attributed Attacks" aria-labelledby="tab-geo-attributed-attacks">
<div id="geo-attributed-attacks" class="dashboard-page tab-pane show active html-fill-item html-fill-container" data-title="Geo-Attributed Attacks" data-orientation="rows" aria-labelledby="tab-geo-attributed-attacks">
<div class="html-fill-item html-fill-container bslib-grid" style="display: grid; grid-template-rows: minmax(3em, max-content) minmax(3em, max-content); grid-auto-columns: minmax(0, 1fr);">
<div class="html-fill-item html-fill-container bslib-grid" style="display: grid; grid-template-columns: minmax(3em, 1fr);
grid-auto-rows: minmax(0, 1fr);">
Expand Down Expand Up @@ -330,7 +330,7 @@
</div>
</div>
</div>
<div id="malicious-activity" class="dashboard-page tab-pane grid-skip html-fill-item html-fill-container" data-orientation="rows" data-title="Malicious Activity" aria-labelledby="tab-malicious-activity">
<div id="malicious-activity" class="dashboard-page tab-pane grid-skip html-fill-item html-fill-container" data-title="Malicious Activity" data-orientation="rows" aria-labelledby="tab-malicious-activity">
<div class="html-fill-item html-fill-container bslib-grid" style="display: grid; grid-template-rows: minmax(3em, max-content) minmax(3em, max-content); grid-auto-columns: minmax(0, 1fr);">
<div class="html-fill-item html-fill-container bslib-grid" style="display: grid; grid-template-columns: minmax(3em, 1fr);
grid-auto-rows: minmax(0, 1fr);">
Expand Down Expand Up @@ -431,7 +431,7 @@
</div>
</div>
</div>
<div id="about" class="dashboard-page tab-pane grid-skip html-fill-item html-fill-container" data-orientation="rows" data-title="About" aria-labelledby="tab-about">
<div id="about" class="dashboard-page tab-pane grid-skip html-fill-item html-fill-container" data-title="About" data-orientation="rows" aria-labelledby="tab-about">
<div class="html-fill-item html-fill-container bslib-grid" style="display: grid; grid-template-rows: minmax(3em, max-content) minmax(3em, max-content); grid-auto-columns: minmax(0, 1fr);">
<div class="html-fill-item html-fill-container bslib-grid" style="display: grid; grid-template-columns: minmax(3em, 1fr) minmax(3em, 1fr) minmax(3em, 1fr);
grid-auto-rows: minmax(0, 1fr);">
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

Binary file modified docs/graph.gpickle
Binary file not shown.
2 changes: 1 addition & 1 deletion docs/graph_summary.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"_invocation_id": "9f22a7f0-c44e-4fab-ae50-1e9fd9fec265", "linked": {"0": {"name": "source.datadex.web.energy", "type": "source", "succ": [3]}, "1": {"name": "source.datadex.web.co2_global_trend", "type": "source", "succ": [5]}, "2": {"name": "source.datadex.public.raw_threatened_animal_species", "type": "source", "succ": [4]}, "3": {"name": "model.datadex.climate_energy", "type": "model"}, "4": {"name": "model.datadex.climate_theatened_animal_species", "type": "model"}, "5": {"name": "model.datadex.climate_co2_global_trend", "type": "model"}}}
{"_invocation_id": "774c7d48-a87b-4a8e-9f92-074691833269", "linked": {"0": {"name": "source.datadex.web.energy", "type": "source", "succ": [3]}, "1": {"name": "source.datadex.web.co2_global_trend", "type": "source", "succ": [5]}, "2": {"name": "source.datadex.public.raw_threatened_animal_species", "type": "source", "succ": [4]}, "3": {"name": "model.datadex.climate_energy", "type": "model"}, "4": {"name": "model.datadex.climate_theatened_animal_species", "type": "model"}, "5": {"name": "model.datadex.climate_co2_global_trend", "type": "model"}}}
2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/run_results.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v4.json", "dbt_version": "1.6.7", "generated_at": "2023-11-07T10:13:00.382610Z", "invocation_id": "9f22a7f0-c44e-4fab-ae50-1e9fd9fec265", "env": {}}, "results": [{"status": "success", "timing": [{"name": "compile", "started_at": "2023-11-07T10:13:00.363733Z", "completed_at": "2023-11-07T10:13:00.374507Z"}, {"name": "execute", "started_at": "2023-11-07T10:13:00.376685Z", "completed_at": "2023-11-07T10:13:00.376698Z"}], "thread_id": "Thread-1 (worker)", "execution_time": 0.01734185218811035, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_co2_global_trend"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-11-07T10:13:00.369892Z", "completed_at": "2023-11-07T10:13:00.374902Z"}, {"name": "execute", "started_at": "2023-11-07T10:13:00.377077Z", "completed_at": "2023-11-07T10:13:00.377084Z"}], "thread_id": "Thread-2 (worker)", "execution_time": 0.017460107803344727, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_energy"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-11-07T10:13:00.371991Z", "completed_at": "2023-11-07T10:13:00.375331Z"}, {"name": "execute", "started_at": "2023-11-07T10:13:00.377503Z", "completed_at": "2023-11-07T10:13:00.377512Z"}], "thread_id": "Thread-3 (worker)", "execution_time": 0.017596006393432617, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_theatened_animal_species"}], "elapsed_time": 0.8801689147949219, "args": {"vars": {}, "empty_catalog": false, "quiet": false, "version_check": true, "use_colors": true, "write_json": true, "populate_cache": true, "invocation_command": "dbt docs generate --project-dir dbt", "exclude": [], "warn_error_options": {"include": [], "exclude": []}, "printer_width": 80, "select": [], "introspect": true, "print": true, "log_file_max_bytes": 10485760, "send_anonymous_usage_stats": false, "log_format": "default", "strict_mode": false, "use_colors_file": true, "log_level": "info", "macro_debugging": false, "indirect_selection": "eager", "favor_state": false, "which": "generate", "static_parser": true, "log_level_file": "debug", "partial_parse_file_diff": true, "log_format_file": "debug", "enable_legacy_logger": false, "compile": true, "cache_selected_only": false, "defer": false, "partial_parse": true, "profiles_dir": "/workspaces/datadex/dbt", "project_dir": "dbt", "log_path": "dbt/logs"}}
{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v4.json", "dbt_version": "1.6.8", "generated_at": "2023-11-09T19:49:38.013925Z", "invocation_id": "774c7d48-a87b-4a8e-9f92-074691833269", "env": {}}, "results": [{"status": "success", "timing": [{"name": "compile", "started_at": "2023-11-09T19:49:37.976692Z", "completed_at": "2023-11-09T19:49:37.999367Z"}, {"name": "execute", "started_at": "2023-11-09T19:49:38.003223Z", "completed_at": "2023-11-09T19:49:38.003250Z"}], "thread_id": "Thread-1 (worker)", "execution_time": 0.033797502517700195, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_co2_global_trend"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-11-09T19:49:37.994978Z", "completed_at": "2023-11-09T19:49:38.000408Z"}, {"name": "execute", "started_at": "2023-11-09T19:49:38.005358Z", "completed_at": "2023-11-09T19:49:38.005373Z"}], "thread_id": "Thread-3 (worker)", "execution_time": 0.03384900093078613, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_theatened_animal_species"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2023-11-09T19:49:37.990740Z", "completed_at": "2023-11-09T19:49:38.001784Z"}, {"name": "execute", "started_at": "2023-11-09T19:49:38.008302Z", "completed_at": "2023-11-09T19:49:38.008317Z"}], "thread_id": "Thread-2 (worker)", "execution_time": 0.03730511665344238, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.datadex.climate_energy"}], "elapsed_time": 0.9608099460601807, "args": {"introspect": true, "print": true, "enable_legacy_logger": false, "vars": {}, "defer": false, "use_colors_file": true, "project_dir": "/workspaces/datadex/dbt", "log_file_max_bytes": 10485760, "cache_selected_only": false, "log_format": "default", "printer_width": 80, "which": "generate", "partial_parse_file_diff": true, "favor_state": false, "macro_debugging": false, "select": [], "indirect_selection": "eager", "quiet": false, "send_anonymous_usage_stats": false, "write_json": true, "log_format_file": "debug", "empty_catalog": false, "profiles_dir": ".", "version_check": true, "use_colors": true, "static_parser": true, "exclude": [], "invocation_command": "dbt docs generate --profiles-dir .", "log_level": "info", "partial_parse": true, "populate_cache": true, "strict_mode": false, "log_level_file": "debug", "warn_error_options": {"include": [], "exclude": []}, "log_path": "/workspaces/datadex/dbt/logs", "compile": true}}
Loading

0 comments on commit 8d7e6b4

Please sign in to comment.