Neptune Analytics Support & Logging #23
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Replaces the queries to fetch edge directions while building the schema. The old queries worked by testing for the existance of any possible edge, which resulted in a large number of queries for graphs with lots of node and edge labels. The new query instead iterates through all the edge labels, and runs a query to fetch the corresponding from-label to-label pairs.
…eptune-Analytics---Logger
Replaces the queries to fetch edge directions while building the schema. The old queries worked by testing for the existance of any possible edge, which resulted in a large number of queries for graphs with lots of node and edge labels. The new query instead iterates through all the edge labels, and runs a query to fetch the corresponding from-label to-label pairs.
…g of test cases, and changed to analytics endpoint
This reverts commit 5e54131.
…ould also be logged to console. Added new loggerError method which logs the message to file and console.error.
…e-Analytics---Logger
…gger Reduce duplicate logging code
…er for analytics vs neptune db.
sophiadt
force-pushed
the
sophiadt/Neptune-Analytics---Logger
branch
from
51a491d
to
12691dd
Compare
…and use parameterized query for getNodeProperties.
…e() function to check relationship type
Replaces the queries to fetch edge directions while building the schema. The old queries worked by testing for the existence of any possible edge, which resulted in a large number of queries for graphs with lots of node and edge labels. The new query instead iterates through all the edge labels, and runs a query to fetch the corresponding from-label to-label pairs.
Cole-Greer
reviewed
Oct 4, 2024
| } | ||
|
|
||
| function sanitize(text) { | ||
| // TODO implement sanitization logic |
There was a problem hiding this comment.
Will this be coming in a future PR?
There was a problem hiding this comment.
Yes - still need to talk with AWS about the logic.
Cole-Greer
reviewed
Oct 4, 2024
src/NeptuneSchema.js
Outdated
| for (let fromLabel of result.fromLabel) { | ||
| for (let toLabel of result.toLabel) { | ||
| edgeStructure.directions.push({from:fromLabel, to:toLabel}); | ||
| consoleOut(' Found edge: ' + yellow(edgeStructure.label) + ' direction: ' + yellow(fromLabel) + ' -> ' + yellow(toLabel)); |
There was a problem hiding this comment.
Is the expectation that calls to consoleOut should always be paired with calls to loggerLog?
…sier maintainability and future customization. Changed the quiet option to set the console log level to 'warn' if enabled ('debug' if disabled) instead of checking for the quiet value before each log. File logging is by default at the 'debug' level but could be customized in the future with a user-provided level. Removed some ARNs and other info from logged statements for privacy concerns. (#4)
…gic (#5) Introduce usage of the neptune graph (analytics) SDK in a couple scenarios: 1. during pipeline creation, if an axios request fails when querying the Neptune Analytics graph and falls back to SDK and 2. if the user has opted to use SDK for the generated lambda (as opposed to http). Summary of changes: -added new dependency on client-neptune-graph version 3.662.0 -created new lambda template which is used if the user specifies --output-resolver-query-sdk option -set additional lambda environment variable for neptune db name which is required to execute queries using the neptune graph SDK -added logic to fall back to neptune graph SDK if Axios request fails during pipeline creation (previous logic threw Error as the analytics SDK was not yet available) -fixed function which retrieves graph summary to use neptune graph SDK if the neptune-type is neptune-graph (the summary endpoint path for neptune-db is not the same for neptune-graph) -fixed CDK pipeline to only fetch cluster info if the type is neptune-db as it is not required for Neptune-graph (analytics) -set isNeptuneIAMAuth to true if the neptune type is detected as neptune-graph -introduced util.js for parsing functions that are used across multiple modules -refactored function which had many params to use an object param instead for better readability -fixed some typos which specified 'neptume' instead of 'neptune' -changed queryNeptune function to have default value of empty object for params -change jest config from json to js to allow for globals to be set from environment variables -changed test cases to reference <AIR_ROUTES_DB_HOST> and <AIR_ROUTES_DB_PORT values which are swapped at test runtime for the environment variable values for easier test configurability and to prevent future accidental commit of real URLs -removed duplicated airports.source.schema.graphql file in test cases and changed to reference single file in parent directory -fixed Case05/case02.json which I believe was accidentally changed from remove to create pipeline -introduced new test case 7 which sets --output-resolver-query-sdk option
…equired for axios credential interceptor in the http lambda.
Refactored console output and spinners to be more like a command line progress UI and introduced debug level file logs. Setting the verbose-log option to true will log more data to the log file. Since the verbose log data can include graph data values it is intended for the user to have to opt in to this verbose logging. Changed pino standard out logger to console as the pino logger was causing interleaving log messages that were out of order with the spinners. Ensured to not log to console if a spinner is spinning as this can have unintended interleaving of output.
|
This PR was broken into several smaller ones: |
Sounds good. Thank you @andreachild! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR to release Neptune Analytics - Logger #19 changes to main
Changes:
neptune-dbtodo.schema.graphqlthat failed to create when utility is used with the todo schemaNote: connecting to a Neptune Analytics graph requires the additional argument:
--create-update-aws-pipeline-neptune-IAM option