Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added folder for structured solution with script. #15

Merged

Conversation

Jesse-Rees
Copy link
Contributor

Script pull csv files from github into dremio.

Additional changes have been made to use environment variables stored locally only rather than prompted inputs.
csv urls are now passed in the command to run the script as detailed in the comments at the top of the script.

@AmirZandiehprojects
Copy link
Collaborator

AmirZandiehprojects commented Sep 12, 2024

@Jesse-Rees Thank you for your work on the Dremio pipeline script. I've identified three critical security issues that need to be addressed ASAP:
everything else looks good

Input Validation: The script accepts URLs without proper validation. Please implement robust input validation for all user inputs, especially CSV URLs.
SQL Injection Risk: Current SQL query construction is vulnerable to injection attacks. Switch to using prepared statements or parameterized queries for all database interactions.
Error Handling and Logging: Implement comprehensive error handling and secure logging practices. Ensure no sensitive information is exposed in logs.

Jesse-Rees and others added 2 commits September 17, 2024 01:23
As requested by cybersecurity team several security based measures have been added like url parsing, pre-made sql statements to avoid sql injections, error handling and also logging of script progress.
@Jesse-Rees
Copy link
Contributor Author

Jesse-Rees commented Sep 16, 2024

@AmirZandiehprojects thank you for your review, I made the several changes you requested. This pull request now also includes the file for a docker container running an api on top of the dremio rest api that only allows select statements to be made by people connected to the deakin network using anyconnect VPN.

@AmirZandiehprojects
Copy link
Collaborator

Awesome thank you Jesse i will approve this pull request !

@AmirZandiehprojects AmirZandiehprojects merged commit 7998a31 into Redback-Operations:main Sep 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants