Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parquet loader #1666

Merged
merged 35 commits into from
Jun 26, 2024
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
168dbb1
rename graph/pandas to graph/io
shivam-880 Jun 18, 2024
1826f12
rename utils to panda_loaders
shivam-880 Jun 18, 2024
d433bbc
refactor panda loader related functions to panda_loader from df_loaders
shivam-880 Jun 18, 2024
f0da7b2
init parquet loader
shivam-880 Jun 18, 2024
d3ad3d8
impl and test process_parquet_file_to_df
shivam-880 Jun 19, 2024
4280dba
impl load edges from parquet for graph
shivam-880 Jun 19, 2024
fc9f2ed
impl/test load nodes from parquet and fix names order issue
shivam-880 Jun 19, 2024
db8c036
impl/test load from parquet. load node/edge props from parquet
shivam-880 Jun 19, 2024
69ffe89
add tests for props
shivam-880 Jun 20, 2024
8ab1b64
ref tests
shivam-880 Jun 20, 2024
e09f387
simplify tests
shivam-880 Jun 20, 2024
7744a32
more tests
shivam-880 Jun 20, 2024
76238f1
impl loaders for persistent graphs
shivam-880 Jun 20, 2024
ff0e6eb
impl loader tests for persistent graphs
shivam-880 Jun 20, 2024
ec1e9ab
move load_edges_deletions_from_pandas to panda_loaders
shivam-880 Jun 20, 2024
38a4fb6
impl load_edges_deletions_from_parquet
shivam-880 Jun 20, 2024
5e1dd62
impl edge deletions tests
shivam-880 Jun 20, 2024
5326733
fmt
shivam-880 Jun 20, 2024
96267dc
add py dep
shivam-880 Jun 20, 2024
1323b32
add deps
shivam-880 Jun 20, 2024
330890c
move compression to arrow dep
shivam-880 Jun 20, 2024
116d26c
restruct io
shivam-880 Jun 21, 2024
c48c750
restruct arrow
shivam-880 Jun 21, 2024
4e63229
tmpdir impl
shivam-880 Jun 21, 2024
c32bf86
add feature gates
shivam-880 Jun 21, 2024
7178d01
make parquet loader compatible for rust
shivam-880 Jun 21, 2024
c3048c8
move py impls to python package
shivam-880 Jun 22, 2024
dcf50de
rename PretendDF to DFView, panda_loaders to pandas_loaders
shivam-880 Jun 24, 2024
8d57736
rid parent
shivam-880 Jun 24, 2024
bbb065f
make test create parquet
shivam-880 Jun 24, 2024
f27c9a0
load parquet from dir or file
shivam-880 Jun 24, 2024
f102175
change invalid layers error message to include valid layers
shivam-880 Jun 24, 2024
05a586a
Merge branch 'master' into parquet-loader
shivam-880 Jun 24, 2024
07d12f3
fix issue with valid_layers
shivam-880 Jun 26, 2024
4c7695f
Merge branch 'master' into parquet-loader
shivam-880 Jun 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions examples/rust/src/bin/bench/main.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
use raphtory::{
algorithms::centrality::pagerank::unweighted_page_rank,
graph_loader::source::csv_loader::CsvLoader, prelude::*,
algorithms::centrality::pagerank::unweighted_page_rank, io::csv_loader::CsvLoader, prelude::*,
};
use serde::Deserialize;
use std::{
Expand Down
2 changes: 1 addition & 1 deletion examples/rust/src/bin/btc/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#![allow(dead_code)]

use chrono::{DateTime, Utc};
use raphtory::{core::utils::hashing, graph_loader::source::csv_loader::CsvLoader, prelude::*};
use raphtory::{core::utils::hashing, io::csv_loader::CsvLoader, prelude::*};
use regex::Regex;
use serde::Deserialize;
use std::{
Expand Down
2 changes: 1 addition & 1 deletion examples/rust/src/bin/crypto/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use raphtory::{
pathing::temporal_reachability::temporally_reachable_nodes,
},
db::api::view::*,
graph_loader::example::stable_coins::stable_coin_graph,
graph_loader::stable_coins::stable_coin_graph,
};
use std::{env, time::Instant};

Expand Down
2 changes: 1 addition & 1 deletion examples/rust/src/bin/hulongbay/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use raphtory::{
triangle_count::triangle_count,
},
},
graph_loader::source::csv_loader::CsvLoader,
io::csv_loader::CsvLoader,
prelude::*,
};
use regex::Regex;
Expand Down
2 changes: 1 addition & 1 deletion examples/rust/src/bin/lotr/main.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use raphtory::{
algorithms::pathing::temporal_reachability::temporally_reachable_nodes, core::utils::hashing,
graph_loader::source::csv_loader::CsvLoader, prelude::*,
io::csv_loader::CsvLoader, prelude::*,
};
use serde::Deserialize;
use std::{
Expand Down
2 changes: 1 addition & 1 deletion examples/rust/src/bin/pokec/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ use raphtory::{
centrality::pagerank::unweighted_page_rank, components::weakly_connected_components,
},
db::{api::mutation::AdditionOps, graph::graph::Graph},
graph_loader::source::csv_loader::CsvLoader,
io::csv_loader::CsvLoader,
prelude::*,
};
use serde::Deserialize;
Expand Down
14 changes: 14 additions & 0 deletions python/tests/test_graphdb.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
import math
import sys
import random
import re

import pandas as pd
import pandas.core.frame
import pytest
Expand Down Expand Up @@ -1435,6 +1437,18 @@ def test_layer():
assert g.exclude_layers(["layer1", "layer2"]).count_edges() == 1
assert g.exclude_layer("layer2").count_edges() == 4

with pytest.raises(
Exception,
match=re.escape("Invalid layer: test_layer. Valid layers: _default, layer1, layer2"),
):
g.layers(["test_layer"])

with pytest.raises(
Exception,
match=re.escape("Invalid layer: test_layer. Valid layers: _default, layer1, layer2"),
):
g.edge(1, 2).layers(["test_layer"])


def test_layer_node():
g = Graph()
Expand Down
2 changes: 1 addition & 1 deletion python/tests/test_load_from_pandas.py
Original file line number Diff line number Diff line change
Expand Up @@ -624,7 +624,7 @@ def assertions_layers_in_df(g):
assert g.layers(["layer 3"]).edges.src.id.collect() == [3]
with pytest.raises(
Exception,
match=re.escape("Invalid layer test_layer."),
match=re.escape("Invalid layer: test_layer. Valid layers: _default, layer 1, layer 2, layer 3, layer 4, layer 5"),
):
g.layers(["test_layer"])

Expand Down
Loading
Loading