ChangeLog

SGAS Luts Server Change Log (from version 3.0.0 and onwords)

3.8.x 

- Add commented out config for newer NGINX (available in CentOS 7.4+). The
_legacy variables are needed since the old variables changed from
/O=Grid/CN=somewhere to CN=somewhere,O=Grid format, which SGAS does not like.



3.8.1

Bug fixes:

- WLCG views: use start_- and end_time instead of "measured" for Storage queries

- Fix host scale factors.

- Fix a problem with loading plugins.

- Support newer twisted versions (old should still work too)


3.8.0

Mostly refactorization of the code with added support for handling
diffrent parts of SGAS/Luts as plugins loaded when needed.

Added tool for basic UR/SR db maintainance.

Added custom query interface from the SNIC/SAMS project.

Some important upgrade notes:

- Backward compatability view -> view:all and insert -> jobinsert in authz.conf are removed.

- Loading of plugins must be done in sgas.conf (see docs/plugins for more information).

- No changes to the database is done.

3.7.2

Add support for Memory tag in UR.

Fixes issues with XML schema from ARC-Jura

3.7.1

Fixes #12 CPUDuration

Fixes a missed DROP FUNCTION in upgrade from 3.6.2.

Fixes missing files in setup.py.

3.7.0

Fixes #7 added nagois plugin.

Fixes #3 Added support for Star v1.2 final. Requires updated database.
See upgrade instructions for more information.

3.6.3

Fixes #2 (GitHub Issue) UTF Multi byte chars are rejected
	Database Encoding might have to be updated. Se docs/upgrading for
	instructions.

Fixes #1 (GitHub Issue) Problem with long running, large jobs

Fix for empty <host> tag. Patch from Erik Edelmann <edelmann@csc.fi>

3.6.2

Added index jobtransferdata_usage_data_id_idx on jobtransferdata(usage_data_id) to speed up delete on entries in usagedata table.

Changed type of processors in usagedata table to integer (from smallint)
for large clusters with more then 32k cores.

Update of date range functions in query interface.
From date is now inclusive (>=) and end date is still exclusive (<).

3.6.1

Cast durations integers into bigint when summing to prevent overflow errors.

Log full stack trace of insertion errors to easy debugging.

Fix foreign key reference in storagedata table.

Set ssl_verify_depth 3; in NginX default configuration to make deep certificate
infrastructures work out of the box.

Add sgas_cert_update script for updating certs, CRLs, and restarting NginX
(forces cert/CRL reread). Contributed by Ulf Tigerstedt.

3.6.0

Support for the EMI StAR storage accounting format (v7). This support is still
experimental is likely to be subject to change. See docs/storage for details.

Support for extracting time since last registration for a host, enabling Nagios
monitoring of registration. See docs/monitoring for details.

Extend database host field to support large hosts set.

Include insert host in aggregated data (required for Nagios monitoring).

Better handling of database restarts/unavailability.

Better error reporting in view engine.

Perform reverse lookup of inserter ip to provide host name for provenence.

Some small changes to graph layout to make them nicer looking and easier to
compare y-axis values.

Log insert rejection, when rejection is done in pre-check (before reading usage
records).

Fix a case in record overwriting in urcreate which would cause the function to
error out.


3.5.2

Increase error verbosity increase maximum client body size in nginx sample
configuration.

Fix bug for rendering HTML table for 2-dimensional data.

Perform modulus 256 on exit code to ensure that it fits into PostgreSQL schema.

Add WLCG specific view (requires wlcgsgas library and configuration).

Automatically size table to fit in CSS.

Enable logging before startup be capturing log statements and emitting them
after startup.

Use proper return statements in javascript helper functions (fixes graph
rendering in Chromium browser).


3.5.1

Convert charge field into integer in order to prevent insertion errors, when
the field is present.

Fix multi-service initiation for database such the hostscaling is started.

Fix update statement and db pool usage in host scaling.


3.5.0

Add workaround for dealing with invalid local job ids from ARC reporters.

View engine overhaul:
- Upgrade to Protovis 3.2.
- Switch URLs for custom view to be under view/custom.
- Two build in views: Administrators manifest and machine view.
- Unified CSS file for all views.

Support for host scaling, see docs/hostscaling.

No more support for direct TLS/SSL. Reverse proxy setup is now required.

A number of schema normalization upgrades. Saves a bit of space, and makes
it possible to have a richer data model for aggregated data.


3.4.2:

An incorrect variable was referenced when parsing upload information. This
caused either an error or some download data to be used as upload data.

Switch file size from 4-byte integer to 8-byte integer to avoid overflow
errors in PostgreSQL for large files.


3.4.1

Make OrderedDict implementation not rely on _zip_longest, as it was broken
(using len instead to provide equality check).

Updates to query documentation (fixes and updates in 3.4.1)

Make the query result return min and max dates in the resulting records, and
not the query ranges.

Updates resourceutil module for getting hostname, and use constants when
accessing headers.

Fix a copy/paste error in a query engine error message.

Always include vo name/project name in query results.

Fix the query parser to correctly read vo_name query parameter.

Fix PostgreSQL database setup to be Python 2.7 compatible.


3.4.0

Support for CouchDB has been removed.

New query interface for extracting information from SGAS.

New authorization engine for significantly more flexible authorization.

SGAS can now log job transfer statistics from ARC (Note that ARC does not yet
report this in the current release).

The configuration option check_whitelist is no longer supported, use
insert:machine_name=FQDN in sgas.authz instead.

Better detection of fake VO information when creating aggregated information.

New discrete_scatterplot and line graph types.

Support relocation using --home in installation.

Add hash index to to schema for significantly faster aggregation updates.

Use embedded database function for aggregation update to avoid ping-pong with
database during updating.

Write friendly greeting with version number to log when starting up.

NginX sample config file is now automatically installed.

Updated view documentation with configuration overview.

Better Python 2.5 compatability.

Various clean-ups.


3.3.2:

Avoid update race condition by running updater in serialized transaction mode.

Deal with closed database connection errors (typically caused by db restarts)
in PostgreSQL database module.


3.3.1:

Flip legend for stacked columns to fit with stack order. Also make a bit more
room for the legend to avoid column-legend overlap, increase legend spacing
a bit, and increase graph size slightly.

Use OrderedDict for parsing configuration file to make the view list on the
SGAS view page the same order as in the configuration file.

Add a PostgreSQL script for doing complete aggregation recalculation in
database (without using SGAS).

New implementation aggregation updater as the old one had a race condition.

Update NginX documentation with details on bug with SSL session and client
certificates.

Ensure that graph y-value height is always at least 1.

Fix group id setting in init.d script.

Correctly calculate maximum y-value for grouped column charts.

Be less aggresive in the aggregation updater timer when the updater is already
running.


3.3.0:

New (rewritten from scratch) visualization engine. The graphs are accessable
via HTTPS and hence viewable ih the browser. Data sources are specified via SQL
in the configuration file. Currently supported graph types are columns, stacked
columns, grouped columns. View authorization happens via the sgas.authz file.

SGAS now supports running from behind a reverse proxy (nginx). This is partly
due to some strange behaviour from the Python SSL wrapper and that the new
visualization engine must service .css and .js files, for which Twisted Web is
not well suited for. For the visualization engine to work, the LUTS must run
behind a reverse proxy.

Support for UR processor field. Previously node_count has been used for
counting processors (incorrect). The LUTS will detect old loggers only using
node_count and adapt the usage record to the new schema. The schema should
now also be compatable with PostgreSQL 8.3.

Minor updates to the PostgreSQL schema after getting feedback on the 3.2 schema
(some integer -> numeric conversions and longer fields).

Added white list option for bypassing the host - machine name consistency check
as some loggers must be able to register usage records on behalf of other
sites.


3.2.0:

New database backend, which adds support for PostgreSQL along with the existing
CouchDB support. The PostgreSQL backend performs automatic aggregation of the
usage records.

Better record duplicate detection and overwriting, making it possible to log
the same job from multiple places, and keep the record with the most
information (required for setups using both ARC and LRMSURGen).

Host-record consistency check during insert. Hosts can now only register
records with machine names in their own domain (matched with the X509
identity). Strictness is configurable.

Support and configuration option for running as non-root user.

Possible to specify logfile in configuration file.

There is an experimental query and view engine but it is not finalized.


3.1.0:

New _id scheme to conserve space in CouchDB, and perform faster view
generation.

Added a change log.


3.0.1:

Documentation overhaul, including a better structured installation guide, and
an explanation of the main concepts as it was a bit tricky to understand for
new people. Some additional view examples have been added as well.

Return error code 500 (server error) instead of 400 (client error) when an
error happens on insertion.

Better error messages if the underlying database is unavailable.

Some more test cases have been added.

Test for host certificate, key, and CA dir during startup and fail at startup
instead of starting up fine and reporting the error in the log (should confuse
admins a bit less in case of a misconfiguration).

More flexible view engine. Can now report complex results (i.e., multiple
values per table cell) as well as having a single column (1-dimensional
results).

Disables graph rendering of results, as it was buggy and largely useless.

View engine have had a bit of an overhaul, though no visible changes. Mainly to
prepare for more flexible view usage in the future. A view overview page is now
available and will list the custom defined views (URL: /sgas/view).

Slightly less verbose logging as the service wrote quite a bit to the log file.


3.0.0:

New, completely rewritten service.

Seperate backend database. We are now using CouchDB as backend database for
storing usage records. Using a proper database allows easier generation of
summary data.

Data insertion and retrieval now uses ordinary TLS and HTTP. This means that it
is possible to use stock tools as cURL and a Browser to use the LUTS.

Signifanatly easier installation and configuration compared to previous SGAS
versions.

View engine for creating custom views of aggregated data. This part is still a
bit rough but is is possible to create view of aggregated data. These can be
displayed directly in the browser or be fetched with an HTTP client