CHANGELOG

2024-10-28 Release 2.2.4:
	- update libhxl-python version

2024-08-21 Release 2.2.3:
	- update docker base image
	- remove legacy deps specifiers

2024-06-06 Release 2.2.2:
	- add Google Tab Manager js lib
	- ability to configure GTM via env var GOOGLE_TAG_MANAGER_ID
2024-02-06 Release 2.2.1:
	- require libhxl==5.2.1:
	  - make logup log level configurable via environment variable
2024-01-18 Release 2.2:
    	- fix pcodes urls
	- require libhxl==5.2:
	  - update URLs for test files on GitHub
	  - fixed bug in aggregators that excluded tag patterns ending in "!"
2023-09-26 Release 2.1:
	- remove all SQL support
	- require libhxl==5.1 to reduce logging output

2023-07-13 Release 2.0.2:
	- update to libhxl==5.0.2 to support the .hxl extension for CSV files

2023-06-01 Release 2.0.1:
	- tweak dependency versions to avoid requests-cache conflicts
	- move setup info from setup.py to setup.cfg
	- add more dependency versions to about.html
	- ensure that the HXL Proxy can read from itself

2023-05-18 Release 2.0
	- from libhxl:
	  - /api/source-info now returns columns for headers and HXL hashtags
	  - URL munging fixed for direct-download HXL Proxy URLs

2023-04-06 Release 1.29
	- from libhxl:
	  - /api/source-info now supports all formats (not just XLSX and XLS)
	  - /api/source-info now returns headers and HXL hashtags
	  - libhxl fixes for API-munging bug
	  - renamed 'hashtag_hash' property from /api/source-info to 'hxl_header_hash'

2023-03-29 Release 1.28.1
	- temporarily deactivate Python-layer timeout (causing random request failures)

2023-03-20 Release 1.28
	- remove controller support for adding or editing saved recipes
	- request code stops running after a configurable timeout (default: 30 seconds)
	- remove skip-verify-ssl option (SSL certificates for remote data must always be valid)
	- add support for structured logs (e.g. JSON)
	- add support for Visual Studio
	- remove all frozen version requirements (except urllib3 for requests), so that we'll pull the latest packages on deployment
	- get rid of warnings during user testing

2022-12-12 Release 1.27.3
	- interim release to upgrade libhxl to 4.27.3 (decreased logging levels)

2022-10-07 Release 1.27.1
	- interim release to address performance issues
	- change Docker config to use threads with nginx
	- make root logging level configurable via ROOT_LOGGING_LEVEL config property (and default to INFO instead of DEBUG)

2022-10-03 Release 1.27
	- add REMOTE_ACCESS_LOGGING_LEVEL to filter "hxl.REMOTE_ACCESS" separately to show remote file access (currently,
	  all messages will be at the INFO level)
	- show the same error message for a non-existant remote data file, or for a URL that contains HTML markup; prevents
	  bad actors from using the HXL Proxy to ping HTML websites
	- don't trap 404 and HTML markup exceptions in the /data/edit endpoint
	- forbid HTML markup or URLs in saved-recipe titles, descriptions, and stubs to cut down on spam attempts

2022-08-11 Release 1.26.1
	- downgrade to mysql-connector-python==8.0.29 to avoid bug in 8.0.30

2022-08-08 Release 1.26
	- add MAX_REQUEST_TIMEOUT configuration option, to place a cap on the maximum allowable timeout for reading external datasets
	- add LOGGING_LEVEL configuration option (defaults to "ERROR")
	- log warnings for exceptions caught and returned as web pages with 400/500-series status codes

2022-07-18 Release 1.25.2
	- update to libhxl==4.25.2 to block tagger exploit

2022-07-14 Release 1.25.1
	- improve logging output

2022-06-29 Release 1.25
	- add /api/source-info to get information about a data source (Excel-only for now)
	- add support for filling merged areas in Excel to UI and data-preview API
	- change references to hxl.io to hxl.input for libhxl-python 1.25+
	- refactored to support new InputOptions class in libhxl-python: all input options are now available for all API calls
	- all unit tests passing

2022-03-16 Release 1.24.4
	- fix cache threshold type

2022-03-16 Release 1.24.3
	- added warning banner to admin pages, per OICT requirements (code committed for 1.25)
	- increase threshold for cache

2021-04-29 Release 1.24.2
	- bugfix release: change debug handling in config for data preview (to detect non-existant sheet in an Excel workbook)
	- add PyPi publish target to Makefile
	- fix unit tests to work in non-debug mode

2021-04-23 Release 1.24
	- fix data-preview api call with sheet param 0
	- cache improvements
	- tagger fix from libhxl

2021-02-08 Release 1.23
	- incorporate release 4.23 of libhxl-python, with stability
	  improvements for sorting and character-encoding errors
	- improved support for redis caching
	- disabled HID controller tests (not currently in use)

2021-01-15 Release 1.22
	- consolidate recent bugfix releases
	- update to libhxl=4.22 for security and Excel improvements and bug fixes
	- fix redis caching for input and refactor/simplify input caching code
	- add additional security unit tests

2020-12-08 Release 1.21.3
	- update to libhxl=4.21.2 to fix rare Unicode encoding error for CSV lookahead buffers

2020-08-18 Release 1.21.2
	- update to libhxl==4.21.1 to fix URL munging for HXL Proxy source links

2020-08-06 Release 1.21.1
	- restored default Flask logging to stdout/stderr

2020-07-23 Release 1.21
	- added a new "Logs" tab to show logs from HXL processing
	- added UI support for Expand lists filter
	- make tagger UI smarter about avoiding duplicates
	- remove "whitelist" and "blacklist" throughout code and comments
	- fix saved-recipe bug that prevented disabling cloning or removing a description
	- added HTML form to set up data-preview API call interactively, and added objects.json format
	- ensure that user always returns to select page after login diversion
	- added /api/from-spec endpoint to read a full JSON spec (including input and tagger)
	- updated to libhxl-python 4.21
	- hide unusable data tabs

2020-05-21 Release 1.20
	- fixed bug with JSONPath &selector
	- added long description in setup.py for PyPi (from README.md)
	- added link files for in-browser tests
	- remove unneeded system library from DockerFile

2020-04-27 Release 1.19
	- add UI support for labeling steps in a recipe
	- added field on data-source page to specify a JSONpath selector to find the HXL data

2020-03-16 Release 1.18
	- add UI support for new implode filter
	- add UI support for flattening JSON lists
	- add optional header field for rename-column filter
	- add optional &filename= param for /api/data-preview

2020-02-18 Release 1.17
	- added UI support for new Implode filter
	- extended Rename column filter to optionally use headers as well as hashtags
	- added optional configuration for redis caching (input and output); more needed
	- added API endpoint /api/data-preview-sheets to fetch a list of sheet titles from an Excel workbook
	- Docker fixes

2019-12-27 Release 1.16.1
	- minor update: fix HXL_WORKERS check for Docker deployment

2019-12-09 Release 1.16
	- update to libhxl v4.16
	- add an admin interface under /admin (need to set a local admin password)
	- add an endpoint /api/data-preview to do a raw (no-HXL) preview of tabular data sources
	- migrate non-core API calls for P-codes, hash, and hxl_test—to under /api (old URLs still work)
	- stay on the recipe page after updating a saved recipe
	- fix bug with saving clone permission
	- fix bugs with /data/advanced page (for direct JSON entry)
	- fix bug in download stub filename (Flask wasn't escaping it)
	- major code refactoring with new Recipe class
	- remove obsolete iati2hxl endpoint
	- improve organisation and documentation for controllers
	- added more controller unit tests
	- change Dockerfile tag for the base Python image

2019-04-04 Release 1.15.1
	- fix caching for validation requests sent via POST
	- removed obsolete basic chart and map demo

2019-03-29 Release 1.15
	- allow user to provide authorisation token for a private CKAN
	  dataset or other link that returns 403 Forbidden
	- proxy will automatically redirect and start a private dataset
	  when it detects an authorisation error
	- fix 404 error for non-existant P-codes

2019-03-12 Release 1.14.1
	- fix typo in config file template
	- switch to Flask 1.x and Flask-Caching packages
	- fix bug in caching for POST validation request
	- increase gunicorn max GET request line to 8K characters

2019-03-04 Release 1.14
	- longer cache for P-codes (1 week by default)
	- cache results of POST validation service (will fill up cache
	  quickly with big files)
	- bump to libhxl 4.14 for new features and bug fixes

2019-02-06 Release 1.13.1
	- update to libhxl 4.13.2 for bug fixes and new round() formula
	  function

2019-01-31 Release 1.13
	- update to libhxl 4.13, which adds support for row formulas and
	  fixes some bugs

2018-12-03 Release 1.12
	- add Concatenate option to Count filter
	- add /hash endpoint for generating a JSON dataset hash
	- support zipped CSV files (via libhxl-python)
	- fixed parsing of YYYY-MM-DD hh:mm:ss SQL dates (via libhxl-python)

2018-08-31 Release 1.11
	- add special user-agent strings to help with analytics and traffic management

2018-07-31 Release 1.10
	- add UI support for libhxl append-filter improvements (external list
	  and row filters)
	- allow saved-recipe editing page to work even when there is a HXL error
	  (to fix broken recipes)
	- updated libhxl to 4.10 (includes better date cleaning and CSV
	  separator detection, as well as the append-filter improvements)

2018-07-05 Release 1.9.1 (bug fix)
	- fix bug in form-validation regex for tag patterns

2018-06-29 Release 1.9
	- added IATI support with /iati2hxl?url=<url> endpoint
	- added POST endpoint /actions/json-recipe for submitting JSON
	  recipes transactionally
	- update forms validation to support new tag patterns ending in "!"
	- refactored validation support to use JSON report from libhxl

2018-06-14 Release 1.8.3
	- add UI support for JSON filter
	- require libhxl==4.8.4 for JSON filter support

2018-06-05 Release 1.8.2
	- require libhxl==4.8.3 for critical features and fixes

2018-05-31 Release 1.8.1
	- hotfix for installation problem with 1.8 (in a clean install)
	- require libhxl==4.8.2
	- DOCKERFILE updates

2018-05-31 Release 1.8
	- add __version__ attribute to module
	- add /about.html
	- change format of HTML validation error report
	- when the validation error format is JSON, report other errors in
	  JSON as well
	- add new POST endpoint /actions/validate (allows either
	  uploading or providing a URL for both the schema and the
	  content). Details: https://github.com/HXLStandard/hxl-proxy/wiki/Validation-service
	- fixed bug preventing changing custom validation schema in
	  /data/validate
	- added PyCharm project file

2018-05-11 Release 1.7.1
	- require libhxl==4.7.1 for date-parsing hotfix

2018-04-30 Release 1.7
	- add a /pcodes endpoint to download lists of P-codes
	- update to support new libhxl-python hxl.datatypes module (remove
	  references to hxl.common)
	- add JSON output option for data validation (/data/validate.json)
	- show expected value in a validation error if available
	- fix bug with logging into saved recipes from some pages
	- allow sending an inline schema with the schema_content param
	  (GET or POST)

2018-03-31 Release 1.6
	- add lat/lon to clean-data filter
	- add purge option to clean-data filter
	- support using MySQL as an alternative to SQLite3 for saved recipes
	- default to an in-memory database for unit testing
	- fix form for advanced JSON recipes
	- fix template error for updating tagger in a saved recipe

2018-02-05 Release 1.5.1
	- bug-fix release: update to libhxl 4.5.1 so to avoid crashes when
	  parsing a bad date
	- use DEBUG config parameter to decide whether to catch exceptions
  	  and display human-readable error messages

2018-01-31 Release 1.5
	- added option to cut-column filter to skip untagged columns
	- better error report for a private Google Sheet (or other cloud
	  file)
	- added files from @teodorescuserban for building a HXL
	  Proxy Docker image
	- fixed updating source for a saved recipe (short URL)
	- support saving data as a JSON list-of-objects (as well as
	  list-of-lists)
	- allow users to specify number format for clean filter

2017-11-22 Release 1.4
	- send JSON error messages instead of HTML when the user is trying to download a
	  JSON or CSV file
	- add subpath support, so that URLs will be correct when the Proxy
	  is mounted in a subpath in WSGI
	- remove H.ID login link for now, since it doesn't do anything,
	  and is potentially confusing
	- require libhxl-python v4.4 for improved exception handling and
  	  JSON support

2017-06-13 Release 1.3
	- require libhxl-python v4.3 for bug fixes
	- fixed reference to old HDX URL (leading to SSL error)

2017-06-06 Release 1.2
	- change SSL verification logic to skip verifying SSL
	- redirect back to source page if any URL fails SSL cert check
	- function changing changes

2017-06-06 Release 1.1
	- add a verify checkbox to the source page; if unchecked, don't
	  try to verify SSL certs (checked by default)
	- add UI support for the new fill-data filter
	- added support for new date_format field in clean filter
	- support reading new JSON-based input (via libhxl-python)
	- use MIME type and extension where avialable to help choose input
	  type (via libhxl-python)
	- use the character encoding specified in the HTTP response when
	  reading a dataset (via libhxl-python)
	- more flexible handing of invalid SSL certs (via libhxl-python)
	- new "is (not)" operator for row queries (via libhxl-python)

2016-12-02 Release 1.0
	- added support for JSON recipes via a &recipe HTTP parameter
	- add row queries for clean, count, deduplicate, replace, and
	  replace-map filters
	- use "recipe" and "step" (instead of "filters") on data recipe page
	- allow sorting filters/steps
	- allow deleting filters/steps
	- remove type selector for an already-defined filter/step
	- &force=on works for input as well as output
	- limit HTML table display to 5,000 rows to avoid timeouts (full
	  dataset still available via CSV or JSON download)
	- add a max_rows parameter to limit the number of rows returned.
	- add a /hxl-test page (available in JSON as well)
	- modify the append-filter form to add URLs on demand
	- support the new multi-URL append filter from libhxl-python

2016-08-30 Release 0.6
	- Add support for requests-cache, so that different views of the same
	  dataset won't require constant redownloading.
	- Add support for a local copy of the p-code GeoJSON shapes tree, for
	  offline demos (or other purposes).
	- Make the list of countries presented in the mapping dropdown menu
	  configurable.
	- Fix exception handling around a missing data source, to avoid
	  unpredictable behaviour (and incorrect claims of missing HXL hashtags
	  on first load from Google Drive).
	- Start adding scaffolding for jquery-sortable to rearrange filters, but
	  not supported yet in the backend.
	- Remove the sidebar (very slow to download the PNGs in a field
	  environment).