Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add write support to connector #8

Open
EnricoMi opened this issue Jun 15, 2020 · 9 comments
Open

Add write support to connector #8

EnricoMi opened this issue Jun 15, 2020 · 9 comments
Labels
enhancement New feature or request

Comments

@EnricoMi
Copy link
Collaborator

Spark can write DataFrames to sources in two modes: override (erasing everything first) and append. Appending to (mutating) the Dgraph database would be great.

@EnricoMi EnricoMi added the enhancement New feature or request label Jun 15, 2020
@daveaitel
Copy link

Has there been any progress on this issue?

@stackedsax
Copy link

I thought @EnricoMi added write support as he worked through the https://github.com/G-Research/dgraph-dbpedia/ project. But looking at this again, I think it still needs doing.

@stackedsax
Copy link

@EnricoMi I know you're off at the moment, but can you clarify whether this support made it in?

@EnricoMi
Copy link
Collaborator Author

@daveaitel @stackedsax write is not support yet and is definitively a bigger piece of work. And I suspect it won't scale nicely, so don't expect huge write performance.

@stackedsax
Copy link

Thanks for confirming, Enrico. @daveaitel, what did you have in mind here?

@daveaitel
Copy link

daveaitel commented Sep 18, 2021 via email

@EnricoMi
Copy link
Collaborator Author

EnricoMi commented Sep 19, 2021

@daveaitel so that would mean to write / update a single value per node and modifying any edges. That should scale nicely.

Alternative is of course to use the non-scaling traditional pipeline of writing the PageRank scores into a Dgraph compatible RDF file and use the Dgraph live loader. Of course, writing from Spark directly means a much smaller pipeline.

@daveaitel
Copy link

daveaitel commented Sep 19, 2021 via email

@EnricoMi
Copy link
Collaborator Author

That is right, writing to Dgraph from Spark is not supported.

@EnricoMi EnricoMi pinned this issue Nov 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants