#123 Release new version of sqlalchemy exasol package on pypi (#129)

Fixes #123, fixes #113 

Prepare and setup workspace for release(s) 
* Update setup.py
   - Update author(s) and author_email
   - Reformat setup.py
   - Update metadata [classifiers, links, ...]
* Update outdated dependency markers in setup.py
* Update README.rst
* Add additional python versions to CI/CD build matrix
* Add nox session(s) for checking and listing documentation links
  - Add link check to CI/CD pipeline
* Fix broken and outdated links in the documentation
* Update deployment token
* Pin version of deployment action
* Update changelog

.github/workflows/CI-CD.yml

@@ -13,7 +13,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python: [3.6]
+        python: [3.6, 3.7, 3.8, 3.9]
         connector:
           - pyodbc
           - turbodbc
@@ -60,6 +60,9 @@ jobs:
       run: git clone https://github.com/exasol/integration-test-docker-environment.git
       working-directory: ..
 
+    - name: Check documentation links ${{ matrix.python }} using ${{ matrix.connector }}
+      run: nox -s "check-links"
+
     - name: Run Test for Python ${{ matrix.python }} using ${{ matrix.connector }}
       run: nox -s "verify(connector='${{ matrix.connector }}')"
 
@@ -97,8 +100,8 @@ jobs:
     - name: Build sdist and wheel packages
       run: python setup.py sdist bdist_wheel
 
-    - name: Push package to Test Pypi
-      uses: pypa/gh-action-pypi-publish@master
+    - name: Push package to Pypi
+      uses: pypa/gh-action-pypi-publish@release/v1
       with:
         user: __token__
-        password: ${{ secrets.pypi_password }}
+        password: ${{ secrets.pypi_token }}

.github/workflows/CI.yml

@@ -14,7 +14,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python: [3.6]
+        python: [3.6, 3.7, 3.8, 3.9]
         connector:
           - pyodbc
           - turbodbc
@@ -61,6 +61,9 @@ jobs:
       run: git clone https://github.com/exasol/integration-test-docker-environment.git
       working-directory: ..
 
+    - name: Check documentation links ${{ matrix.python }} using ${{ matrix.connector }}
+      run: nox -s "check-links"
+
     - name: Run Test for Python ${{ matrix.python }} using ${{ matrix.connector }}
       run: nox -s "verify(connector='${{ matrix.connector }}')"
 

CHANGES.md

@@ -1,3 +1,16 @@
+# 2.3.0
+* Update supported versions of EXASOL DB to 7.1.6 and 7.0.16 
+* Update supported python versions to 3.6, 3.7, 3.8, 3.9
+* bumped SQLAlchemy dependency to 1.3.24
+* bumped pyodbc dependency to 4.0.32
+* Fixed bug regarding maximum identifier length
+* Fixed bug with custom translate maps
+* Fixed bug regarding non existent error code mappings for EXASOL specific odbc error codes
+* Updated documentation to reflect the latest version changes etc.
+* Updated maintainer and contact information
+* Removed outdated documentation and resources
+* Removed python 2.7 support
+
 # 2.2.0
 * Solved performance problems for large tables/databases. Full information on this [PR](https://github.com/blue-yonder/sqlalchemy_exasol/pull/101)
 * Bumped dependencies

README.rst

@@ -2,8 +2,8 @@ SQLAlchemy Dialect for EXASOL DB
 ================================
 
 
-.. image:: https://github.com/blue-yonder/sqlalchemy_exasol/workflows/CI-CD/badge.svg?branch=master
-    :target: https://github.com/blue-yonder/sqlalchemy_exasol/actions?query=workflow%3ACI-CD
+.. image:: https://github.com/exasol/sqlalchemy_exasol/workflows/CI-CD/badge.svg?branch=master
+    :target: https://github.com/exasol/sqlalchemy_exasol/actions?query=workflow%3ACI-CD
 .. image:: https://img.shields.io/pypi/v/sqlalchemy_exasol
      :target: https://pypi.org/project/sqlalchemy-exasol/
      :alt: PyPI Version
@@ -20,93 +20,96 @@ How to get started
 ------------------
 
 We assume you have a good understanding of (unix)ODBC. If not, make sure you
-read their documentation carefully - there are lot's of traps to step into.
-
-Get the EXASolution database
-````````````````````````````
-
-If you do not have access to an EXASolution database, download EXASolo for free
-from EXASOL: http://www.exasol.com/en/test-drive/
-
-The database is a VM image. You will need VirtualBox, VMWare Player, or KVM to
-run the database. Start the database and make sure you can connect to it as
-described in the How-To from EXASOL.
+read their documentation carefully - there are lot's of traps 🪤 to step into.
 
 Meet the system requirements
 ````````````````````````````
 
 On Linux/Unix like systems you need:
 
-- the packages unixODBC and unixODBC-dev >= 2.2.14
-- Python >= 2.7
-- Download and install the ODBC client drivers from EXASOL >= 5
-- configure ODBC.ini and ODBCINST.ini
+- An Exasol DB (e.g. `docker-db <test_docker_image_>`_ or a `cloud instance <test_drive_>`_)
+
+  - >= 7.1.6
+  - >= 7.0.16
+
+- The packages unixODBC and unixODBC-dev >= 2.2.14
+- Python >= 3.6
+- The Exasol `ODBC driver <odbc_driver_>`_
+- The ODBC.ini and ODBCINST.ini configurations files setup
 
 Turbodbc support
 ````````````````
 
-- Turbodbc and sqlalchemy_exasol as well do now support python 2.7, 3.4 and 3.6.
+- You can use Turbodbc with sqlalchemy_exasol if you use a python version >= 3.6.
 - Multi row update is not supported, see
   `test/test_update.py <test/test_update.py>`_ for an example
 
 
-Setup you python project and install sqlalchemy-exasol
-``````````````````````````````````````````````````````
+Setup your python project and install sqlalchemy-exasol
+```````````````````````````````````````````````````````
 
-::
+.. code-block:: shell
 
-	> pip install sqlalchemy-exasol
+    $ pip install sqlalchemy-exasol
 
 for turbodbc support:
 
-::
+.. code-block:: shell
 
-	> pip install sqlalchemy-exasol[turbodbc]
+    $ pip install sqlalchemy-exasol[turbodbc]
 
-Talk to EXASolution using SQLAlchemy
-````````````````````````````````````
+Talk to the EXASOL DB using SQLAlchemy
+``````````````````````````````````````
 
-::
+.. code-block:: python
 
 	from sqlalchemy import create_engine
-	e = create_engine("exa+pyodbc://A_USER:[email protected]:1234/my_schema?CONNECTIONLCALL=en_US.UTF-8&driver=EXAODBC")
+	url = "exa+pyodbc://A_USER:[email protected]:1234/my_schema?CONNECTIONLCALL=en_US.UTF-8&driver=EXAODBC"
+	e = create_engine(url)
 	r = e.execute("select 42 from dual").fetchall()
 
 to use turbodbc as driver:
 
-::
+.. code-block:: python
 
 	from sqlalchemy import create_engine
-	e = create_engine("exa+turbodbc://A_USER:[email protected]:1234/my_schema?CONNECTIONLCALL=en_US.UTF-8&driver=EXAODBC")
+	url = "exa+turbodbc://A_USER:[email protected]:1234/my_schema?CONNECTIONLCALL=en_US.UTF-8&driver=EXAODBC"
+	e = create_engine(url)
 	r = e.execute("select 42 from dual").fetchall()
 
 
-The dialect supports two connection urls for create_engine. A DSN (Data Source Name) mode and a host mode:
+The dialect supports two types of connection urls creating an engine. A DSN (Data Source Name) mode and a host mode:
+
+.. list-table::
+    :widths: 50 50
+    :header-rows: 1
 
-========  ====================================================================
-DSN url   'exa+pyodbc://USER:PWD@exa_test'
-Host url  'exa+pyodbc://USER:[email protected]:1234/my_schema?parameter'
-========  ====================================================================
+   * - Type
+     - Example
+   * - DSN URL
+     - 'exa+pyodbc://USER:PWD@exa_test'
+   * - HOST URL
+     - 'exa+pyodbc://USER:[email protected]:1234/my_schema?parameter'
 
-*Features*:
+Features
+++++++++
 
 - SELECT, INSERT, UPDATE, DELETE statements
 - you can even use the MERGE statement (see unit tests for examples)
 
-*Note*:
+Notes
++++++
 
-- Schema name and parameters are optional for the host url string
+- Schema name and parameters are optional for the host url
 - At least on Linux/Unix systems it has proven valuable to pass 'CONNECTIONLCALL=en_US.UTF-8' as a url parameter. This will make sure that the client process (Python) and the EXASOL driver (UTF-8 internal) know how to interpret code pages correctly.
 - Always use all lower-case identifiers for schema, table and column names. SQLAlchemy treats all lower-case identifiers as case-insensitive, the dialect takes care of transforming the identifier into a case-insensitive representation of the specific database (in case of EXASol this is upper-case as for Oracle)
-- As of EXASol client driver version 4.1.2 you can pass the flag 'INTTYPESINRESULTSIFPOSSIBLE=y' in the connection string (or configure it in your DSN). This will convert DECIMAL data types to Integer-like data types. Creating integers is a factor three faster in Python than creating Decimals.
-
-Troubleshooting
-```````````````
-
-The unixodbc Stack is not the most friendly for programmers. If you get strange errors from the driver mangager, you might have an issue with the names of the unixodbc libs. Have a look at https://github.com/blue-yonder/sqlalchemy_exasol/blob/master/fix_unixodbc_so.sh to find ideas on how to fix this on Ubuntu. Good luck!
+- As of Exasol client driver version 4.1.2 you can pass the flag 'INTTYPESINRESULTSIFPOSSIBLE=y' in the connection string (or configure it in your DSN). This will convert DECIMAL data types to Integer-like data types. Creating integers is a factor three faster in Python than creating Decimals.
 
 Development & Testing
 `````````````````````
 See `developer guide`_
 
-.. _developer guide: https://github.com/exasol/sqlalchemy-exasol/blob/doc/developer_guide/developer_guide.rst
+.. _developer guide: https://github.com/exasol/sqlalchemy-exasol/blob/master/doc/developer_guide/developer_guide.rst
+.. _odbc_driver: https://docs.exasol.com/db/latest/connect_exasol/drivers/odbc/odbc_linux.htm
+.. _test_drive: https://www.exasol.com/test-it-now/cloud/
+.. _test_docker_image: https://github.com/exasol/docker-db

doc/developer_guide/developer_guide.rst

@@ -94,7 +94,7 @@ Tests
     If something still is not working or unclear, you may want to look into the CI/CD action_ files.
 
 
-.. _action: https://github.com/blue-yonder/sqlalchemy_exasol/actions
+.. _action: https://github.com/exasol/sqlalchemy_exasol/actions
 .. _python: https://www.python.org/
 .. _git: https://git-scm.com/
 .. _Docker: https://www.docker.com/

doc/developer_guide/integration_tests.rst

@@ -18,7 +18,7 @@ This workflow will be executed anytime there's a commit pushed to any branch tha
 
 To run it just commit and push to any non-master branch and watch the workflow run in:
 
-`<https://github.com/blue-yonder/sqlalchemy_exasol/actions>`_
+`<https://github.com/exasol/sqlalchemy_exasol/actions>`_
 
 CI-CD
 -----
@@ -33,6 +33,6 @@ This workflow will be executed anytime there's a commit pushed to **master**, or
 
 To run it just commit and push to master (*Optional:* push a tag in case you want Pypi upload) and watch the workflow run in:
 
-`<https://github.com/blue-yonder/sqlalchemy_exasol/actions>`_
+`<https://github.com/exasol/sqlalchemy_exasol/actions>`_
 
 The status of the CI-CD workflow will always be reflected in the badge called "build" in the README.rst and Home Page of this repository

noxfile.py

@@ -1,14 +1,19 @@
 import os
+import sys
 from contextlib import contextmanager
 from pathlib import Path
 from tempfile import TemporaryDirectory
 from textwrap import dedent
 
+PROJECT_ROOT = Path(__file__).parent
+sys.path.append(f'{PROJECT_ROOT / "scripts"}')
+
 import nox
+from links import check as _check
+from links import documentation as _documentation
+from links import urls as _urls
 from pyodbc import connect
 
-PROJECT_ROOT = Path(__file__).parent
-
 # default actions to be run if nothing is explicitly specified with the -s option
 nox.options.sessions = ["verify(connector='pyodbc')"]
 
@@ -76,7 +81,7 @@ def temporary_odbc_config(config):
 @contextmanager
 def odbcconfig():
     with temporary_odbc_config(
-            ODBCINST_INI_TEMPLATE.format(driver=Settings.ODBC_DRIVER)
+        ODBCINST_INI_TEMPLATE.format(driver=Settings.ODBC_DRIVER)
     ) as cfg:
         env_vars = {"ODBCSYSINI": f"{cfg.parent.resolve()}"}
         with environment(env_vars) as env:
@@ -168,3 +173,26 @@ def integration(session, connector):
             ]
         ).format(connector=connector, db_port=Settings.DB_PORT)
         session.run("pytest", "--dropfirst", "--dburi", uri, external=True, env=env)
+
+
+@nox.session(name="check-links", python=None)
+def check_links(session):
+    """Checks weather or not all links in the documentation can be accessed"""
+    errors = []
+    for path, url in _urls(_documentation(PROJECT_ROOT)):
+        status, details = _check(url)
+        if status != 200:
+            errors.append((path, url, status, details))
+
+    if errors:
+        session.error(
+            "\n"
+            + "\n".join((f"Url: {e[1]}, File: {e[0]}, Error: {e[3]}" for e in errors))
+        )
+
+
+@nox.session(name="list-links", python=None)
+def list_links(session):
+    """List all links within the documentation"""
+    for path, url in _urls(_documentation(PROJECT_ROOT)):
+        session.log(f"Url: {url}, File: {path}")

odbcconfig/odbcinst.ini

@@ -0,0 +1,8 @@
+[ODBC]
+Trace=yes
+TraceFile=/home/nic/Projects/sqla-playground/odbc.trace
+
+[EXAODBC]
+#Driver location will be appended in build environment:
+DRIVER=/home/nic/Projects/sqla-playground/driver/libexaodbc-uo2214lv1.so
+

requirements_dev.txt

@@ -1,3 +1,4 @@
 -r requirements_test.txt
 nox>=2022.1.7
+urlscan>=0.9.9
 

scripts/links.py

@@ -0,0 +1,48 @@
+import urllib.error
+import subprocess
+from itertools import repeat
+from pathlib import Path
+from typing import Iterable, Tuple
+from urllib import request
+
+
+def documentation(root: Path) -> Iterable[Path]:
+    """Returns an iterator over all documentation files of the project"""
+    docs = Path(root).glob("**/*.rst")
+
+    def _deny_filter(path):
+        return not ("venv" in path.parts)
+
+    return filter(lambda path: _deny_filter(path), docs)
+
+
+def urls(files: Iterable[Path]) -> Iterable[Tuple[Path, str]]:
+    """Returns an iterable over all urls contained in the provided files"""
+
+    def should_filter(url):
+        _filtered = []
+        return url.startswith("mailto") or url in _filtered
+
+    for file in files:
+        cmd = ['python', '-m', 'urlscan', '-n', f'{file}']
+        result = subprocess.run(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
+        if result.returncode != 0:
+            stderr = result.stderr.decode('utf8')
+            msg = f"Could not retrieve url's from file: {file}, details: {stderr}"
+            raise Exception(msg)
+        stdout = result.stdout.decode('utf8').strip()
+        _urls = (url.strip() for url in stdout.split('\n'))
+        yield from zip(
+            repeat(file), filter(lambda url: not should_filter(url), _urls)
+        )
+
+
+def check(url: str) -> Tuple[int, str]:
+    """Checks if an url is still working (can be accessed)"""
+    try:
+        # User-Agent needs to be faked otherwise some webpages will deny access with a 403
+        req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
+        result = request.urlopen(req)
+        return result.code, f"{result.msg}"
+    except urllib.error.HTTPError as ex:
+        return ex.status, f"{ex}"