diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..93dc65b --- /dev/null +++ b/.env.example @@ -0,0 +1,8 @@ +databricks_experiment_name='' +databricks_experiment_id='' +databricks_host='' +databricks_token='' +databricks_username='' +databricks_password='' +databricks_cluster_id='' +databricks_sql_http_path='' \ No newline at end of file diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst deleted file mode 100644 index 5965e4e..0000000 --- a/CONTRIBUTING.rst +++ /dev/null @@ -1,270 +0,0 @@ -.. Generated by synthtool. DO NOT EDIT! -############ -Contributing -############ - -#. **Please sign one of the contributor license agreements below.** -#. Fork the repo, develop and test your code changes, add docs. -#. Make sure that your commit messages clearly describe the changes. -#. Send a pull request. (Please Read: `Faster Pull Request Reviews`_) - -.. _Faster Pull Request Reviews: https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#best-practices-for-faster-reviews - -.. contents:: Here are some guidelines for hacking on the Google Cloud Client libraries. - -*************** -Adding Features -*************** - -In order to add a feature: - -- The feature must be documented in both the API and narrative - documentation. - -- The feature must work fully on the following CPython versions: - 3.7, 3.8, 3.9, 3.10 and 3.11 on both UNIX and Windows. - -- The feature must not add unnecessary dependencies (where - "unnecessary" is of course subjective, but new dependencies should - be discussed). - -**************************** -Using a Development Checkout -**************************** - -You'll have to create a development environment using a Git checkout: - -- While logged into your GitHub account, navigate to the - ``python-storage`` `repo`_ on GitHub. - -- Fork and clone the ``python-storage`` repository to your GitHub account by - clicking the "Fork" button. - -- Clone your fork of ``python-storage`` from your GitHub account to your local - computer, substituting your account username and specifying the destination - as ``hack-on-python-storage``. E.g.:: - - $ cd ${HOME} - $ git clone git@github.com:USERNAME/python-storage.git hack-on-python-storage - $ cd hack-on-python-storage - # Configure remotes such that you can pull changes from the googleapis/python-storage - # repository into your local repository. - $ git remote add upstream git@github.com:googleapis/python-storage.git - # fetch and merge changes from upstream into main - $ git fetch upstream - $ git merge upstream/main - -Now your local repo is set up such that you will push changes to your GitHub -repo, from which you can submit a pull request. - -To work on the codebase and run the tests, we recommend using ``nox``, -but you can also use a ``virtualenv`` of your own creation. - -.. _repo: https://github.com/googleapis/python-storage - -Using ``nox`` -============= - -We use `nox `__ to instrument our tests. - -- To test your changes, run unit tests with ``nox``:: - - $ nox -s unit-2.7 - $ nox -s unit-3.7 - $ ... - - .. note:: - - The unit tests and system tests are described in the - ``noxfile.py`` files in each directory. - -.. nox: https://pypi.org/project/nox/ - -***************************************** -I'm getting weird errors... Can you help? -***************************************** - -If the error mentions ``Python.h`` not being found, -install ``python-dev`` and try again. -On Debian/Ubuntu:: - - $ sudo apt-get install python-dev - -************ -Coding Style -************ - -- PEP8 compliance, with exceptions defined in the linter configuration. - If you have ``nox`` installed, you can test that you have not introduced - any non-compliant code via:: - - $ nox -s lint - -- In order to make ``nox -s lint`` run faster, you can set some environment - variables:: - - export GOOGLE_CLOUD_TESTING_REMOTE="upstream" - export GOOGLE_CLOUD_TESTING_BRANCH="main" - - By doing this, you are specifying the location of the most up-to-date - version of ``python-storage``. The the suggested remote name ``upstream`` - should point to the official ``googleapis`` checkout and the - the branch should be the main branch on that remote (``main``). - -- This repository contains configuration for the - `pre-commit `__ tool, which automates checking - our linters during a commit. If you have it installed on your ``$PATH``, - you can enable enforcing those checks via: - -.. code-block:: bash - - $ pre-commit install - pre-commit installed at .git/hooks/pre-commit - -Exceptions to PEP8: - -- Many unit tests use a helper method, ``_call_fut`` ("FUT" is short for - "Function-Under-Test"), which is PEP8-incompliant, but more readable. - Some also use a local variable, ``MUT`` (short for "Module-Under-Test"). - -******************** -Running System Tests -******************** - -- To run system tests, you can execute:: - - $ nox -s system-3.8 - $ nox -s system-2.7 - - .. note:: - - System tests are only configured to run under Python 2.7 and - Python 3.8. For expediency, we do not run them in older versions - of Python 3. - - This alone will not run the tests. You'll need to change some local - auth settings and change some configuration in your project to - run all the tests. - -- System tests will be run against an actual project and - so you'll need to provide some environment variables to facilitate - authentication to your project: - - - ``GOOGLE_APPLICATION_CREDENTIALS``: The path to a JSON key file; - Such a file can be downloaded directly from the developer's console by clicking - "Generate new JSON key". See private key - `docs `__ - for more details. - -- Once you have downloaded your json keys, set the environment variable - ``GOOGLE_APPLICATION_CREDENTIALS`` to the absolute path of the json file:: - - $ export GOOGLE_APPLICATION_CREDENTIALS="/Users//path/to/app_credentials.json" - - -************* -Test Coverage -************* - -- The codebase *must* have 100% test statement coverage after each commit. - You can test coverage via ``nox -s cover``. - -****************************************************** -Documentation Coverage and Building HTML Documentation -****************************************************** - -If you fix a bug, and the bug requires an API or behavior modification, all -documentation in this package which references that API or behavior must be -changed to reflect the bug fix, ideally in the same commit that fixes the bug -or adds the feature. - -Build the docs via: - - $ nox -s docs - -******************************************** -Note About ``README`` as it pertains to PyPI -******************************************** - -The `description on PyPI`_ for the project comes directly from the -``README``. Due to the reStructuredText (``rst``) parser used by -PyPI, relative links which will work on GitHub (e.g. ``CONTRIBUTING.rst`` -instead of -``https://github.com/googleapis/python-storage/blob/main/CONTRIBUTING.rst``) -may cause problems creating links or rendering the description. - -.. _description on PyPI: https://pypi.org/project/google-cloud-storage - - -************************* -Supported Python Versions -************************* - -We support: - -- `Python 3.5`_ -- `Python 3.6`_ -- `Python 3.7`_ -- `Python 3.8`_ - -.. _Python 3.5: https://docs.python.org/3.5/ -.. _Python 3.6: https://docs.python.org/3.6/ -.. _Python 3.7: https://docs.python.org/3.7/ -.. _Python 3.8: https://docs.python.org/3.8/ - - -Supported versions can be found in our ``noxfile.py`` `config`_. - -.. _config: https://github.com/googleapis/python-storage/blob/main/noxfile.py - -Python 2.7 support is deprecated. All code changes should maintain Python 2.7 compatibility until January 1, 2020. - -We also explicitly decided to support Python 3 beginning with version -3.5. Reasons for this include: - -- Encouraging use of newest versions of Python 3 -- Taking the lead of `prominent`_ open-source `projects`_ -- `Unicode literal support`_ which allows for a cleaner codebase that - works in both Python 2 and Python 3 - -.. _prominent: https://docs.djangoproject.com/en/1.9/faq/install/#what-python-version-can-i-use-with-django -.. _projects: http://flask.pocoo.org/docs/0.10/python3/ -.. _Unicode literal support: https://www.python.org/dev/peps/pep-0414/ - -********** -Versioning -********** - -This library follows `Semantic Versioning`_. - -.. _Semantic Versioning: http://semver.org/ - -Some packages are currently in major version zero (``0.y.z``), which means that -anything may change at any time and the public API should not be considered -stable. - -****************************** -Contributor License Agreements -****************************** - -Before we can accept your pull requests you'll need to sign a Contributor -License Agreement (CLA): - -- **If you are an individual writing original source code** and **you own the - intellectual property**, then you'll need to sign an - `individual CLA `__. -- **If you work for a company that wants to allow you to contribute your work**, - then you'll need to sign a - `corporate CLA `__. - -You can sign these electronically (just scroll to the bottom). After that, -we'll be able to accept your pull requests. - - -****************************** -Attribution -****************************** - -This Contributing guide is adapted from the Contributor Source for release-please, -available at -https://github.com/googleapis/python-storage/blob/main/CONTRIBUTING.rst \ No newline at end of file diff --git a/README.md b/README.md index e5970a9..899e433 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,7 @@ databricks_token='' databricks_username='' databricks_password='' databricks_cluster_id='' +databricks_sql_http_path='' ``` ```python diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index 034e848..0000000 --- a/SECURITY.md +++ /dev/null @@ -1,21 +0,0 @@ -# Security Policy - -## Supported Versions - -Use this section to tell people about which versions of your project are -currently being supported with security updates. - -| Version | Supported | -| ------- | ------------------ | -| 5.1.x | :white_check_mark: | -| 5.0.x | :x: | -| 4.0.x | :white_check_mark: | -| < 4.0 | :x: | - -## Reporting a Vulnerability - -Use this section to tell people how to report a vulnerability. - -Tell them where to go, how often they can expect to get an update on a -reported vulnerability, what to expect if the vulnerability is accepted or -declined, etc. diff --git a/it.py b/example.py similarity index 77% rename from it.py rename to example.py index 7508ed2..0f9929a 100644 --- a/it.py +++ b/example.py @@ -6,12 +6,12 @@ # %% spark_sql = DatabricksSQLSession() # %% -pdf = spark_sql.sql("select * from microchip_logs limit 10") +pdf = spark_sql.sql("select * from samples.nyctaxi.trips limit 10") print(pdf.head()) # %% sqlalchemy_engine = spark_sql.get_session() # %% -df = spark_sql.read(sqlalchemy_engine, "microchip_logs") +df = spark_sql.read(sqlalchemy_engine, "samples.nyctaxi.trips") # %% from databricks_session import DatabricksJDBCSession @@ -28,7 +28,7 @@ spark = DatabricksSparkSession().get_session() # %% -sdf = spark.read.table("microchip_logs") +sdf = spark.read.table("samples.nyctaxi.trips") print(sdf.show()) # %% diff --git a/package.json b/package.json deleted file mode 100644 index 717ab45..0000000 --- a/package.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "version": "0.1.1" -}