Release Notes

Current release versions can be downloaded at https://www.heimdalldata.com/download-self/.

Release Build 20.07.10.1 (July 10, 2020)

1. Resolve issue in systemd config file to only restart on error, not always, which can impact updates.  Forcing a restart by rebooting the central manager after updating to this code will resolve this problem.
2. Resolve issue where heap and stack dump options on status tab wouldn’t work unless the proxy was started by the central manager;
3. Add max cache object count and max expiry options for local cache to help control garbage collection overhead with large heaps;
4. Added drain option to status tab, including support to drain on SIGTERM signal to assist with Kuberneties operations.

Release Build 20.07.05.1 (July 5, 2020)

1. Add connection pool new connection rate control (see source documentation for details);
2. Add in new MySQL driver with maxResultBuffer feature;
3. Allow “none” for resetQuery in pool properties to set no reset query, but default to “discard all” for Postgres;
4. Better transaction handling with Postgres, in particular with delayed transactions configured, and when “begin read only” is used to start a read-only transaction, or when “set transaction read only” is used to change the session characteristics;
5. Add better proxy controls via the status screen, including stack trace and heap profile dump options;
6. Improved TLS compatibility;
7. Resolve issue where with read/write split and Postgres, the catalog may not be set properly on the read-only server connection;
8. Doc updates;
9. Resolve regression in MySQL with some character set mappings;
10. Add support for Redis database number configuration;
11. Adjust log behavior from /var/log/heimdallserver.out.log to /opt/heimdall/log/heimdall.log and internalize log roll behavior, to help support docker environments.  Log rolling will save up to 20 logs, of up to 50MB each.  Logs will also be sent to stdout, to enable docker logging to be used as well;
12. Adjust instance installer to use a systemd service for heimdall instead of using supervisord, when present;
13. Add the ability to pull a “host” variable from postgres startup messages or SQL driven authentication, and ${host} dynamic variable in JDBC URL’s for dynamic large-scale hosting configurations;
14. Use Amazon Linux 2 w/ Amazon Corretto Java for AWS Marketplace instances, other instances default to Ubuntu, but this can be adjusted to other distributions;
15. Add Alpine Linux to the installer compatibility list;
16. Updated cloud formation template to correct an issue with the scale up/down policies;
17. Enable support for compressed MySQL connections;
18. Enabled support to cache up to 64MB results in the SQL Server proxy code, it had been limited at 1000 rows.

Release Build 20.06.21.2 (June 21, 2020)

1. Support mysql com_sleep packet (deprecated feature);
2. Provide pool rule property of “reuse” to disable/enable reuse on return of connections at runtime;
3. Remove run-time reload of modules, to better support rolling upgrades without module/driver version conflicts;
4. Add ability to restart specific clients via the status tab;
5. If the server is set to debug mode, then trigger heap dumps on OOM on central manager managed proxies to assist with debugging;
6. Improved connection management in the pool to avoid “lost” connections;
7. Improved client-side idle cleanup behavior;
8. Overall better connection pool behavior;
9. Provide connection borrow overhead times in show pool command;
10. Remove testOnReturn default (can still be enabled in pool configuration);
11. Documentation updates;
12. Binary protocol improvements;
13. Disable multiplexing while a temporary table is detected;
14. Improve behavior for SQL Server cursor open functions, including prepared cursor opens.

Release Build 20.5.6.1 (May 6, 2020)

1. Resolve TLS issues (high cpu/handling of large queries/session reuse), and log the cipher suites used during heartbeats;
2. Reduce logging when using TCP health checks with Postgres;
3. Remove error after a configuration commit when using read/write split or forwarding logic that prevented the reader from being used;
4. Remove some verbose errors that could fill logs;
5. Log to both cloudwatch and central manager when cloudwatch is enabled;
6. Add cloud formation template for proxy auto-scale clusters to documentation;
7. Improve compatibility with Ruby PG driver and Node MySQL drivers;
8. Support SQL Server table parameters to stored procedures;

Release Build 20.4.20.1 (April 20, 2020)

1. Remove data source & time selection in analytics and logging to simplify the interface;
2. Remove the admin notification section, as including the mail code in the driver triggered conflicts with some environments, and was rarely used;
3. Resolved an issue with Mysql when using Replication & loadbalance JDBC URLs;
4. All prior test build issues rolled in for release.

Test build 20.4.14.1 (April 14, 2020)

1. Resolved issue with the Postgres proxy causing problems with Ruby when using prepared statements.

Test build 20.4.14.1 (April 14, 2020)

1. Improved logic for AWS cluster tracking, providing a per-driver template system allowing RDS/Aurora node information to be populated with customization
2. Greatly improved wireline compatibility for all proxies
3. Autodetect prepared statements, and disable multiplexing on the connection using it
4. Multi-threaded Postgres result-set processing, to allow large results to be processed at a faster rate and with less memory overhead
5. Cleaned up Proxy UI for authentication to make it more intuitive
6. Improved handling of SQL driven authentication, including vastly improved documentation
7. Better handling of Redis cluster vs. non-cluster detection for Elasticache for Redis
8. Improved behavior on login errors when connection pooling is enabled, and streamlining of the resulting alerts
9. Add the ability to filter on ports and source IP addresses

Release Build 20.2.9.1 (Feb 9, 2020)

1. Improve pooling behavior, including handling the case where the database closes the connection better, in respect to per-user connection pool limits
2. Improved cloudwatch logging and statistics, including adding a heap available metric
3. Support wrapping queries in a transaction for the purposes of large postgres result sizes–the postgres driver must have the defaultRowFetchSize option set for this to work properly
4. Allow the driver and server to be uploaded via the modules interface to allow updates for simplified patching
5. Remove an issue that prevented updates from kicking in properly–note:  the heimdallserver.sh script must be manually replaced if the base image was from the 19.12.30.1 install.  Contact support for assistance if needed.
6. Added an API to allow easy orchestration integration for database failover tools
7. Improved support for SQL based authentication methods, including per-user ldap configuration as needed
8. Improved Odoo compatibility w/ mutiplexing and pooling enabled
9. Adjust dns caching to 5s by default–may be overridden using java standard methods
10. Reduce use of Java standard Regex due to possible cpu consumption cases
11. Various bug fixes and compatibility improvements for all proxy types
12. Improved documentation for many features

Release Build 19.12.30.1 (Dec 30, 2019)

1. Revised AWS Marketplace base image, including Java 11 to support TLS 1.2, and other improvements including a pure proxy mode for auto-scaling clusters of proxies.  Please see the AWS environment documentation in the new build for details on configuring this
2. Two rare compatibility issues with SQL server resolved
3. All changes and resolved issues in test builds since 19.9.28.1

Test Build 19.12.17.1 (Dec 17, 2019)

1. Revised UI for managing updates to make it more intuitive
2. Update blob handling in JDBC mode to support caching of MySQL blobs
3. Add an option to require TLS to connect to the proxy
4. Provide a new custom Postgres driver with the “maxResultBuffer” option, to limit memory usage while buffering result sets, to prevent out of memory issues.  This is to be submitted to the official Postgres JDBC maintainer.
5. Detect when a row fetch size is set in Postgres, and use this as a hint on how much to buffer before sending data to the client.  Results larger than this will not be cached in proxy mode.
6. Detect invalid regex patterns, and alert on the UI
7. Many low-level optimizations to reduce CPU usage and improve performance
8. Include tested DB2 driver for use in JDBC mode (no proxy is available for DB2 at this time)
9. Added a “simple” bind mode for LDAP authentication, and a group filter for non-simple bind mode
10. Added statement and connection listeners for J2EE environments
11. Numerous other minor updates and enhancements

Test Build 19.10.10.1 (Oct 19, 2019)

1. Improved JDBC compatibility for Websphere and other J2EE environments via PooledDataSource (note:  any XA data source is likely to fail at this point)
2. Updated Lettuce/Redis library code to resolve an issue with the Redis cluster interface, as used with AWS Elasticache clusters with cluster enabled
3. Enabled TLS support on proxies, with auto-generation of self-signed certificates on first use (certs can be loaded as desired)
4. Enable multiple use of ${catalog} variables in a regex field
5. Explicit password sync capabilities for Active Directory (to be documented)

Update Release 19.9.28.1 (Sept 28, 2019)

1. Improved documentation
2. Improved compatibility with SQL Server .Net applications using “Transaction Manager” calls to start/complete transactions.
3.  Adjusted the proxy port option to accept multiple ports, separated by commas.  Each port will be listened to by the VDB.
4. Added a “ports” rule option to add rules based on the port the query was received on.
5. Revamped the connection pooling logic.  Previously, the logic was built on the Apache Tomcat connection pool, which had very poor handling of large numbers of distinct users connecting through the pool.  Now, the logic is based on a “pool of pools” system, where each database+user combination has a “user pool” with it’s own distinct resource limits, along with an overall set of resource limits.  This supports large-scale user connectivity in a much more robust way.  Some pool options have been removed as part of this logic change, as they no longer make sense (such as an initial connection pool).
6. Added the “pool” action, to provide rule-based tuning of user connection pools, allowing individual users to have their own settings.  This will allow service accounts to have large connection pools as desired.  This rule type also allows per-user multiplex settings to be configured.
7. Added an option in the AD configuration to support trusting untrusted certs on the ldap server, and to do the same for HTTPS connections for the software update code.
8. Added copy support for Postgres, to support pg_dump and pg_restore.  Note:  Please upgrade any existing Postgres driver to 42.2.6 if using this feature, as there is a performance regression in 42.2.5 that impacts the copy logic.
9. Adjusted the output stream for proxies to use a buffered stream, which should reduce the number of packets sent and should improve performance.  Most tested with Postgres so far–there is a risk that at times, this may cause stalls in the protocol stream, so please report any issues observed.
10. Adjusting the packaged Postgres driver to 42.2.6 and MySQL to 5.1.48.
11. Add “randomUser” option for VDB, specifically for testing.  This will allow a proxy configuration with proxy auth to randomize the actual login user after a user authenticates.  This helps with tools like Jmeter that only connects with one username to test connection pooling with many users internally.  Accepts a boolean value (true/false).
12. Added inline “show” commands, documented in the “basics/Inline commands section”, including to show connection and pool information.
13. Added the option “hdRewriteInserts” to trigger rewriting of inserts to the source, when async execute is enabled.  This can improve performance significantly over simple transaction batching.  Prior builds would attempt to rewrite automatically.
14. Removed a deadlock on exceptions with rewritten inserts in async.
15. Removed code that purposely limited the number of new connections/second, as it impacted the behavior of Multiplexing in a dramatic way.
16. Set the Postgres driver to default the option “preferQueryMode” to “simple” for performance reasons.  This can be overridden in the connection settings.
17. Resolved an authentication issue (blocking instead of allowing) due to the AD code being added for Postgres.
18. Numerous smaller fixes, in particular surrounding the multiplex logic.
19. Support “wrapping” queries and overriding the user’s ownership of connections, to support reusing Postgres connections across multiple users.  Please contact Heimdall support if you are interested in using this function, designed for high connection Postgres workloads, and in conjunction with Active Directory access controls.
20. Allow triggers to be set to “onlyOnce” as an optimization, such as to configure the database on certain conditions.
21. support for Active Directory (LDAP) authentication for Postgres and SQL Server.  Includes support to add rules against the user’s LDAP group membership

Update Release 19.7.21.1 (July 21, 2019)

1. Improved logging
2. Improved batch rewrite compatibility
3. Handle positional parameters in Postgres prepared statements properly
4. support query patterns larger than 500 characters in the analytics
5. disable attempting to use cloudwatch when cloudwatch wasn’t enabled
6. Resolve a GUI authentication issue when the stub “auth” module was loaded (this is available in source form to allow custom authentication if needed)
7. Support environments where IPv6 is enabled on the instance (this was breaking the ability for the proxy to receive traffic)
8. Allow authentication passthrough with the proxy if “proxy auth” is disabled on Postgres and SQL Server.  Please note, in this case, the username and password will be passed to the DB to authenticate.  If this option is selected for MySQL, then no authentication is performed.
9. Allow a polling interval for Async batching, to better support Redshift data ingestion.

Previous updates:

Please note–release notes are behind in being updated, but since Aug 15th, 2018, the following features have been added (main documentation is updated however):

1. Support for Azure Postgres and MySQL to inject the actual server hostname as part of the user;
2. Improved Async batching behavior, including rewriting simple inserts into a multi-row insert, as well as the ability to delay batch processing until a given amount of time.  This allows optimal performance for large block databases, such as Redshift;
3. Logging of logs to Cloudwatch, not just metrics
4. When Postgres and SQL Server proxies are set to not provide authentication, the passwords will be passed through to the back-end database, reducing the overhead of password management
5. Password rotation support, so new database passwords can be transparently added without downtime
6. Paranoia mode in the logs, to prevent even SQL statements from being included in the logs for debugging
7. Support for MySQL 8 clients, which changes the authentication behavior
8. Parse SQL Server stored procedures to more accurately handle caching.  For both Postgres and SQL Server, views are also parsed to allow proper caching.
9. Leverage Postgres’s pub/sub interface to add invalidations, which also allows stored procedures to control invalidation
10. Improve invalidation api
11. Added the ability to block on async DML operations until a result or commit is made, to improve data safety in highly threaded environments (such as IoT data ingestion)
12. Huge improvements in the overhead of debug logging
13. Numerous bug fixes and compatibility improvements.
14. Added multiplexing to support pg-bouncer replacement

Update Test build 18.8.15.0 (Aug 15, 2018) (TEST)

A major test build update in preparation for a new major release, including:

1. IMPORTANT:  Initial release of our module system–this will reduce the memory and size footprint of the server and driver, by only loading the code that is necessary.  On initial install, a minimum set of modules will need to be manually installed to provide the functionality necessary, i.e. for the proxy support necessary, or the desired caching module.  Modules can be added and removed at runtime, allowing hotfixes for a particular piece of functionality to be provided without restarting the server.
2. Trigger functionality, to allow sql or external callouts to be executed before, during or after particular SQL traffic is encountered.  This includes functions to limit the rate of callouts, delay them, etc.
3. Improved failover behavior, including support for zero exceptions on a connection on failover for particular cases (when autocommit is enabled in particular) and improved connection preservation on failover.
4. SQL Auto-retry functionality via rules, to configure what types of exceptions can trigger an automatic retry, i.e. for a Galera deadlock when multiple nodes are accessing a table.
5. A large number of bug fixes
6. Add initial connection rate limiting, useful for failover with large numbers of connections
7. Set the pool maxActive default to 1000, as too many people were testing with the default of 100, and it wasn’t sufficient.
8. Add a module to allow proxy authentication to be driven via the data source for the VDB.  This can be customized to allow external callouts as needed, contact Heimdall support for example source to compile custom authentication modules.
9. Query per second stats per client on status screen
10. Add support for Apache Ignite/Gridgain and Oracle Coherence for caching
11. Adjust compile version on a per-module basis.  Each module may require a distinct java version, based on the libraries they use.  The core proxy/driver supports java 6, but the following modules have higher java restrictions (Note, the management server now has a minimum of Java 8+):
    1. SQL Server Proxy-Java 7
    2. Apache Geode/Gemfire-Java 7
    3. Apache Ignite/Gridgain-Java 8
12. Add and/or logic to support multiple conditions on a rule, for example a regex query pattern and a table specifier
13. Improved compatibility with Postgres dblink and other protocol level functions
14. Provide an option for structured/unstructured query plan extraction when available
15. Support ${catalog}, ${database}, ${user}, ${password} in SQL Server connection URL, to extract values from the Login request, or the password property on connect, allowing one data source to support many back-end catalogs at once.
16. Fixes provided in the LTS branch.

Updated release 18.4.16.4 (June 29, 2018) (LTS)

A minor update to the 18.4.16.3 test build, including:

1. Adjust the query plan extraction to only operate if a query returns a result-set.  This is to avoid issues with some queries that can terminate transactions when the explain is attempted, as noted on Postgres.
2. Adjust the Java compatibility to allow the driver to operate on Java 6-10 and the server to operate on Java 8-10.  Java 11 has not yet been tested, but is expected to also work.
3. Improve temporary table tracking
4. Prevent autocomplete in the password fields in the VDB and Source tab
5. Better handle read/write split when using mixed capitals in table and catalog names
6. Handle renames of data sources when rules are pointing to the source

Updated release 18.4.16.3 (May 21, 2018) (test)

As the software hasn’t been released yet as anything but a test build, the update version ID hasn’t been incremented, but the following fixes have been made:

1. In the case of a “use <catalog name>” statement in the MySQL proxy, the catalog name was being converted to lower case.  The case is now preserved;
2. If a query was made with a fully qualified catalog.tablename format, it would not be considered the same table as another query with a simple table name for the purposes of cache invalidation.  Now, if the table name starts with the catalog name and a period, then for internal tracking, it will not prepend another copy of the catalog name;
3. Prevent certain management server hangs when statistics were summarized or the delete all logs option was used.

Release version 18.4.16.3 (May 16, 2018) (test)

Test update with bugfixes, in prep for LTS update:

1. Slight change to import api call URL, due to a conflict causing api issues;
2. When using the management server on an alternate port via the application.properties, configure the management-server started proxy to use the proper port to pull the configuration;
3. Avoid a NPE if no rules are attached to a vdb;
4. Improve SQL parsing to handle more cases;
5. Improve prepared statement handling for Postgres via the proxy;
6. Provide a proper error message if a read-only user attempts to dismiss an alert;
7. Better handle automatic rule and source attachment to the VDB;
8. Improve GUI behavior when a filter is applied;
9. Prevent the analytics engine from locking and disabling the GUI under certain situations;
10. Prevent duplicate lines in analytics with one being cached and the other being not-cached, merge these together and report correct overall cache hit rate;
11. Prevent redundant done_sql and trace log generation;
12. Resolve issue where read/write split wouldn’t properly handle last table update times;
13. Allow rewrite of prepared statements before the prepare is executed, to prevent SQL syntax errors from occurring in certain cases;
14. Handle source name rewrite when a source is attached to a rule for forwarding/async, etc.

Release version 18.4.16.2 (Apr 22, 2018) (test, release, lts)

Minor bugfix patch:

1. Security: resolve a security issue where read-only users could use the export option to export a complete configuration including passwords;
2. Adjust the api calls for import/export to start with /api/config vs. just /api for consistency moving forward;
3. Adjust security so that when using a read-only user, it more reliably provides an accurate error of why a write operation was not completed;
4. Fix the link on the login page for login help;
5. If a port is not configured in the wizard, don’t create a jdbc string with “:null” as the port;
6. On importing a rule list, the parameters were appearing blank if commit was triggered, even though the values were present;
7. Update vendor JDBC driver versions in the install package to the currently released versions. Upgrades will not change the jdbc driver versions used, this needs to be done manually to avoid changing underlying JDBC behaviors.

Release version 18.4.16.1 (Apr 16, 2018) (test, release, lts)

A major revision, updating the UI, changing log formats (slightly), and many improvements.  As part of this release, we will be diverging releases into three build cycles:  test, normal releases and LTS (long-term support) releases.  In the software administration section of the admin tab, the release that is monitor for updates can be selected, or no monitoring done at all.  Each type of release can be installed from here as well.  The LTS release is intended to be bug-fix only for an extended period of time, while new features will be in the normal release builds, and the test builds are the cutting edge and not intended for production use unless advised to do so.  As of the moment of the writing, all three release builds are identical, but will diverge moving forward.

New features in this include:

1. Greatly enhanced UI for ease of use;
2. Improved accuracy in the logs
3. Improved compatibility with all proxy drivers
4. Improved docs, which are now included in the software vs. being online links
5. Improved performance of the analytics
6. Nocache rule type to replace the cache rule with ttl=0, to make it clearer what the intent is
7. Improved XA compatibility, including with DB2
8. Tables listed in the expanded query view, to simplify building rules based on tables
9. Improved error detection for common error cases
10. UI Filter capabilities, to help remove clutter when a large number of configuration objects are created or for multi-tenant functions
11. Improved JDBC compatibility when applications use db specific behaviors, such as getBytes on a Blob object
12. Many many bug-fixes

Release version 18.2.16.1 (Feb 16, 2018)

A bug-fix and usability update that includes:

1. Auto-restart the management server if the heap goes down to less than 10% and garbage collection can’t resolve it
2. The ability to download test images via the admin interface, if directed to do so by support
3. Attempt to auto-detect if a proxy port is allowed by a security group when configured in AWS (requires instance IAM access to the security group configuration)
4. Handle cases in MySQL where a trailing “;” is transmitted on the wire
5. Optimize the behavior of the internal database for analytics, query plan and exception tracking
6. Provide a visual indicator on what queries have exceptions and/or query plans associated with them
7. Provide a timestamp on exceptions and query plans to indicate when they were gathered
8. Add configuration object import and export, as well as to duplicate them
9. Add the ability to rename configuration objects
10. Merge configuration object action buttons into a drop-down list
11. Provide a drop-down list of connection pool options with help tooltips
12. Provide drop-down lists of sources in the analytics and log tab for filtering
13. Adjust unix socket setup to only occur if localhost binding is selected
14. Added the ability to specify a specific IP for TCP binding, i.e. when using docker bridge mode networking is enabled, to bind to the docker host IP for system-wide use
15. Add a filter on the Analytics to filter for only SQL exceptions
16. Several minor protocol compatibility improvements with proxies
17. Improved logging and alerting on various critical error conditions
18. Adjusted the proxy-start.sh script to download the newest driver from the management server by default

Release version 18.2.2.1 (Feb 2, 2018)

A bug-fix update that includes fixes on both the driver side and server to resolve:

1. Prevent a “hung” or non-response management server from blocking processing on the driver (Critical)
2. Resolve a memory management issue that could result in garbage collection overhead blocking the processing of driver updates (and thus blocking the driver from processing data) (Critical)
3. Resolve an issue in the duplicate vdb option
4. Improved log ingestion performance for display on the analytics tab
5. Add alerting for connection limits being reached by a connection pool
6. Show a query plan and/or exception indicator on each line of the analytics screen, if the data is present
7. Adjusted the valid object name filter
8. Improve the log behavior for old SQL Server clients to alert on such old clients and their incompatibility
9. Resolve an issue where data for multiple VDBs were used in computing analytic statistics
10. Added an access/secret key to allow proper startup of a proxy without any other authentication configured

Release version 18.1.26.1 (Jan 26, 2018)

A minor update on the data processing path, but numerous improvements to the GUI and usability, including:

1. Additional columns in the analytics for cache vs. db performance stats
2. Improved alerting on a variety of error conditions to help reduce debug overhead
3. Added distinct “access” and “secret” key to support starting the proxy without authentication configured
4. Updated scheme management for the back-end analytics database
5. The download logs option will now download heimdall logs from /var/log/heimdall* as well
6. Added time synchronization detection logic to detect if proxy servers have drifted in time, which can impact proxy hit rates
7. Improved IE 11 compatibility, for users of Windows server 2016 in particular
8. Added duplicate configuration button
9. Autodetect if in an AWS paid-instance, and adjust license behavior accordingly
10. Resolved many small bugs where data on the screen isn’t rendered properly unless refreshed
11. Test data source button now tests the LB servers individually
12. Removed compatibility issue with date/time conversion with Sql Server’s DATETIME2TYPE
13. Show the cache rule ID when displaying “cached” on the Analytics tab, and provide hover-over list of matching rules
14. Provide errors generated when trying to do an explain, to show why a plan may not be available
15. Added notes field to the rules, to provide comments on why the rule exists for example
16. When a pattern matches a cache rule with a ttl value of 0, show as “!cached” instead of “cached”
17. Support running custom log rotation scripts, to upload, compress or otherwise manage the flat log files
18. Prevent chrome from creating extremely long lines, wider than the screen, in the analytics tab
19. Trim queries with more than 10 adjacent positional parameters to use “…” to indicate “and many more”
20. Adjust wizard defaults to behave as more users expect
21. Updated tomcat version used in the management server
22. Alert on low-heap levels in the proxy
23. Reduce the rate of invalidation messages on tables with frequent invalidations, and instead invalidate for a longer period

Release version 18.1.4.1 (Jan 5, 2018)

A minor update, but including significant usability improvements, includes:

1. Context help for rule actions and parameters
2. Drop-down selector list for parameters available for a given rule
3. Resolve issue where the name of the driver may not display properly in the data source tab
4. Add Google Authenticator 2-Factor authentication support for users
5. Hide unused columns when using the Postgres “show db side statistics” in Analytics
6. Resolve issue that could trigger  SQL errors when using the evaluate plan logic (only for SQL Server)
7. Log the IP of failed server login attempts

Release version 17.12.22.1 (Dec 22, 2017)

Signficant update, which requires a coordinated update of the driver and server to account for a change in log record format.

This release includes:

1. Adding the user to the log record, to track who did what against the database.
2. The ability to specify columns to log in a log record, for result-set logging.  This data is logged into the /heimdall/log/data/ directory, and is (currently) indexed by the hash of the resultset to avoid duplication of results.  The hash is inserted into the log records to avoid bloating the actual log records.
3. Auto-explain for query plan details for MySQL, Postgres, SQL Server and Oracle via the “evaluate plan” action.  The results are exposed on the Analytics tab for easy access in analyzing slow query patterns.
4. Provide a “pop out” option on each query pattern in the GUI to provide a nicely formatted view of the SQL, in particular for long queries.
5. Initial Azure integration code (internal).
6. Add initial support for Pivotal Gemfire/Apache Geode (requires Java 8 to function).
7. Add support for Pivotal Greenplum.  The Postgres driver should be used when in proxy mode, but Greenplum will be detected and the behavior of the PG driver will be adjusted to account for this.
8. Add alert system to the GUI for server state changes and critical alerts.
9. Add cache test button in VDB, to verify the cache is initializing properly.
10. Add option in the “reader eligible” policy to ignore lag when lag detection is used, for cases where lag isn’t important to monitor.
11. Support uploading a specific version of Hazelcast to use on the admin console, in the event the included version is not satisfactory.
12. Add option to download logs in the admin tab.
13. Numerous bugfixes, compatibility and logging improvements.
14. Happy Holidays!

Release version 17.11.21.1 (Nov 21, 2017)

Minor update, including:

1. Fix for a case when using the MySQL proxy, prepared statements, UTF8 (not UTF8MB4) encoding and read/write split, which could result in the query not being passed properly to the database.
2. Resolve case sensitivity issue in rule processing with tables: style regex matching.  Now, all tables will be matched without case sensitivity.
3. Adjust how the Postgres code installs the dblink extension when using cluster tracking, which is used to connect from the standby to the master as part of the cluster detect logic.
4. Adds the option to show server stats in the analytics tab for Postgres, which provides database-side data to supplement the server-side data.
5. Various cluster tracking and behavior enhancements.
6. Happy Thanksgiving!

Release version 17.11.8.1 (Nov 8, 2017)

Minor update, including:

1. Improved lag detection logic, and including static lag configuration for read/write split.  If the lag window buffer is set, but lag detection is not, this value will be used to compare against the last write time for a table to determine the safety of a read against read-only servers when reader-eligible rules are used.  When lag detection is used, then this value is added to the computed lag detection, to provide an extra element of safety.
2. Improved cluster monitoring logic for multiple database types, including Postgres, MS-SQL and Galera clusters, to handle more failure cases.  Also provides read and write weight targets to allow ordering of cluster nodes selectively, and a db configurable “heimdall.servers_info” table that allows for mapping of front-end to back-end names and IP’s to account for NAT, non-public IPs, etc.  To be documented in the data source tab help.  A table with default values will be configured if cluster tracking is enabled to simplify the configuration process.
3. Updated the use of the flag “pgmon” to trigger postgres pg_stat_statements tracking.  This value should name the vdb that the stats should be logged against, which generally should be a placeholder vdb for this purpose.  This feature is demonstrated in our current demo via the vdb odoo-alt-vdb.
4. Improved catalog tracking on connections for reconnected connections in various failover cases and read/write split.
5. Improved various GUI behaviors for more intuitive operation, including revised and normalize auto-detected server names with cluster monitoring.
6. Added automatic detection of parent application Hazelcast libraries, and usage of such libraries to prevent conflicts in the JDBC use case.

Release version 17.10.18.1 (Oct 18, 2017)

Minor update, including:

1. Test VDB button to provide a simple baseline traffic generator for testing failover and that the VDB is fundamentally online and able to handle traffic.  It supports MySQL, MS-SQL, PostgreSQL, Oracle and DB2.
2. Improved cluster tracking logic for several database types, including MySQL (Replication & Galera), SQL Server and Postgresql.
3. On Galera clusters, if a temporary lock issue is observed due to a failover or write to alternate server, retry logic will be attempted for up to 4 times and 400ms.
4. Improved failover performance for Postgresql and other databases.
5. Support multiple VDBs via a single JDBC driver
6. Allow filtering of dashboard statistics by VDB
7. Automatically add /libs/*.jar in the application’s cwd into the application classpath, to override any classes included in Heimdall, or to support JDBC drivers that do not support remote loading, such as the Amazon Redshift driver.  In the case of Redshift, do not attach the driver to the actual driver configuration, but otherwise configure all the other parameters, but instead use the ./libs/ folder to include the driver OR otherwise put the Redshift driver in the classpath of the application.
8. Pull cache misses from the grid cache for the first five minutes after driver start, even if using key tracking on the grid-cache.  This will help offload the database during a cold-start, but may reduce the performance benefit of the cache during this time.

Release version 17.9.30.2 (Oct 2, 2017)

Tiny update, including:

1. Fix for the VDB tester when run on Windows
2. Small HTML changes for IE compatibility on the GUI
3. Adjusted logic for SQL Server Mirror cluster handling slightly

Release version 17.9.30.1 (Sept 30, 2017)

Moderately significant revision, including:

1.  Changing the version number scheme to reflect the constant rolling update schedule, now the versions reflect the build date, and a minor version to support hotfixes without any other significant changes.
2. Resolved an issue with the prior revisions SQL-Server driver packaging that would result in errors about class not found when installed on MS-Windows using the MSI installer.
3. Improved cluster detection logic, particularly with SQL Server Always-on Availability Groups.
4. Lag detection logic, to support intelligent read/write split, so that reads are not performed against a server that is not updated to guarantee fresh content.
5. Improved overall failover time, in particular with cluster failover
6. Improved TCP connection maintenance during failovers, to allow applications that can’t reliably reconnect on the loss of the database connection to continue operating
7. Improved SQL Server compatibility, including support for a few previously unsupported SQL Server specific types
8. Added a VDB tester, which will trigger a small amount of traffic on ten parallel connections, in order to assist with failover behavior testing
9. Added new rule type specifically to support read/write split called “Reader eligible”.  The old JDBC “readOnly” support of read/write split has been removed with this rule’s addition.
10. Use the data source properties to test data sources–this allows a source to connect that requires particular settings, such as the use of SSL, etc.  Now, the test source button will be hidden until committed.
11. Added configuration information for Hazelcast in AWS environments in the GUI to configure the necessarily parameters for AWS autodiscovery.
12. Added the ability to pull the hazelcast configuration from the central server to reduce management overhead.
13. Added a PCI/HIPAA cache configuration option, which will leverage grid caches for synchronization, but will not use them for the actual cache store, preventing sensitive data from being transmitted unencrypted.
14. Support adding the max-rows parameter (if set) to the cache key, to prevent responses that had different values from being treated the same
15. prevent the use of fonts from a remote server, and include in our local GUI files, to avoid external network calls
16. use the host application’s classpath to configure Hazelcast if available, otherwise fall back to an included Hazelcast 3.8 build in our driver file.  This is to allow Hazelcast versions to be explicitly controlled and used by the local application without conflict.
17. Various other logging improvements and minor bug fixes.

Release version 1.0.20.1 (Sept 11, 2017)

Hotfix to resolve a few issues:

1.  When using the jdbc call “setMaxRows”, exceptions could be printed that implied an error.  While a cosmetic issue, this could cause performance problems if the call was used too frequently.
2. Updated the JDBC driver for MS-SQL that is packaged from our custom version to a more formal release that supports the SRV Variant type.  1.0.20 without this updated driver using MS-SQL would have issues with this type.
3. Resolve some issues when a call returns multiple result sets in MySQL.

Other changes:

1.  Include the version number in the Manifest for the driver and server jar files, and report the server version in the GUI, including the timestamp of the build.
2. Resolve some lingering issues with the Wizard, to improve compatilibity with AWS instances.

Release version 1.0.20 (Sept 7, 2017)

Minor revision including the following changes:

• Added api call /api/config/enabled/{object name}/[enable|disable] to enable or disable configuration objects easily
• Added log throttling to remove frequently called log messages
• Added per-method cumulative execution time tracking, reported on the daily or sighup generated log for the driver/proxy code
• Remove the printing of passwords from the properties list when reporting connection errors
• Improve cluster compatibility for auto-cluster tracking, and disable the editing of options that are managed by the cluster tracking code when enabled
• Optimizations, particularly in the MySQL proxy code
• Improved status page, allowing pinning of nodes to help identify when they have gone offline and are not passing traffic.  Nodes are updated on connections made, so with connection pooling, they may not update as often (to be resolved in a future version of code).
• GC tracking code in the server, sending a SIGHUP on systems supporting this will provide the detailed data on memory and CPU usage for the server.
• Allow drivers of up to 50MB to be uploaded without adjusting any other configuration
• More bug fixes and behavior enhancements to improve compatibility with various applications

Release version 1.0.19 (Aug 17, 2017)

Minor revision including the following changes:

• Further improvement with SQL Server, MySQL proxy compatibility, as well as improved Oracle JDBC compability.
• Optimized Cache behavior, including a filter to prevent caching in the local memory cache any resultset larger than 1/1000th of the configured memory cache size, to prevent pushing other useful content out.  The grid-cache interface is used for the retrieval of such objects however.
• Resolved an issue that could prevent the test-source button from functioning in certain situations
• Updated cluster detection, configuration, and monitoring logic
• Initial testing for AWS Aurora for Postgres to verify it functions properly with the PostgreSQL proxy.

Release version 1.0.18 (Aug 6, 2017)

Minor revision including the following changes:

• Improved compatibility with SQL Server, including a fix to an issue that could hang connections or trigger exceptions with cached content when using the proxy
• Prevent errors if a database server isn’t ready in AWS when using the wizard
• Added support links on the GUI to make it easier to access support
• Honor the VDB disable flag when using the proxy, to terminate requests to an offline VDB, generally while the database is in maintenance
• Upgrade the Windows installer to use Java 8, necessary for SQL server proxy support
• In MS SQL proxy, add additional logs to help debug why some queries are not cached
• Detect several clusters types when configuring a data source, and automatically configure the cluster peers including SQL Server Always On, Galera and stock MySQL clusters.  Postgresql clusters are pending for the next release.
• Adds throttling of repeated log messages, to prevent flooding of the log file

Release version 1.0.17a (July 20, 2017)

Single fix patch, which resolves a timing related deadlock that could occur on driver/proxy initialization, most often with the Sql Server proxy.

Release version 1.0.17 (July 19, 2017)

In this release:

This is a minor bug-fix release, including:

• Improvements to the SQL Server proxy for compatibility with various drivers and use-cases
• Improved transaction state tracking for the MySQL proxy
• Resolve class loading issues with optional extensions for PostgreSQL
• Resolved a few table name extraction issues for various SQL dialects

One new feature was added to this new build however–when using the MySQL and Postgres proxies, on non-Windows systems, they will automatically attempt to setup and listen to Unix sockets that match the default sockets for MySQL and Postgres at the following locations:

• MySQL:  /tmp/mysqld.sock and /var/run/mysqld/mysqld.sock
• Postgres: /tmp/.s.PGSQL.$port and /var/run/postgresql/.s.PGSQL.$port

This should improve ease of use with applications that connect to “localhost” as this defaults to using Unix sockets.  The exact sockets attached to will be listed in the proxy startup, i.e.:

[2017-07-20 13:35:03,506] [2017-07-20 13:35:03,506] localhost jira: Listening on 5050
[2017-07-20 13:35:04,116] [2017-07-20 13:35:04,116] localhost jira: Proxy: Listening on socket: /var/run/postgresql/.s.PGSQL.5050
[2017-07-20 13:35:04,122] [2017-07-20 13:35:04,122] localhost jira: Proxy: Listening on socket: /tmp/.s.PGSQL.5050

Release version 1.0.16 (July 11, 2017)

In this release:

This is a minor bug-fix release, including:

• Improvements to the SQL Server proxy for compatibility with various drivers and use-cases, as well as some lesser-used field types
• Support configuration of a SQL Server proxy via the wizard
• Improve the error messages when an old version of TDS is used with the SQL server proxy
• Allow connection closes to be handled properly for the MySQL proxy, in particular when canceling a query
• Support larger than 16MB responses that use multiple MySQL packets in the proxy
• Include more information about the client on the status screen
• prevent spaces in configuration names, now they will be replaced with underscores automatically
• Adjust the health-check behavior on an exception from the underlying JDBC driver
• Resolve an NPE when using a heimdall configuration file to set the default hduser and hdpassword
• Add an option to the heimdall configuration file to disable testing for AWS.  Use “aws=false” in the file to avoid the error that AWS was not detected

Release version 1.0.15 (June 26, 2017)

In this release:

Most of the changes in this build are bug-fixes, but it does contain a few new features:

1. Improved MS-SQL proxy support, approaching release quality.  We are testing with a wide variety of applications, and most appear to work correctly.  Only SQL Server 2008+ is supported at this time, and this requires the proxy code to run on Java 7+, while other functions only require Java 6;
2. Added the ability to pause on the dashboard, and other cosmetic changes to the GUI to improve usability;
3. Added support to control the initial password for the server based on either the environment or a /etc/heimdall.conf file.  Set the options as “hduser=” and “hdpassword=” to override any defaults on startup;
4. Improved invalidation logic to resolve some corner cases where it would not be honored;
5. Improved logging to help find problems;
6. Other small issues corrected that in general didn’t impact behavior but resulted in errors in the logs.

Release version 1.0.14 (June 11, 2017)

In this release:

1. A new configuration wizard is now available.  This combines the capability of the original AWS wizard with generic configuration support, to help bring online an application with a minimum amount of effort.  The wizard allows for simple configuration of baseline cache and read/write split along with many other features on a vdb.
2. Grid-clustering capability to allow redundancy of a management server–to enable, select the “Cluster manager via cache” option in the cache settings.  This feature operates on a per-vdb basis, and uses the vdb name as a unique key to synchronize the configurations across nodes.  This feature replaces the “peercache” configuration option previously supported to allow nodes to share cache information.
3. A new status page has been added, to allow the health status of databases, as well as what clients are connecting to be viewed;
4. A new log tab has been added, to allow inspection of up-to the last 1000 log entries on the server;
5. AWS CloudWatch support on a per-vdb level, providing per-instance and vdb statistics;
6. Configuration caching support on the driver.  To use, specify a Heimdall JDBC URL parameter of “localconfig=<pathToDirectory>”.  When used, new configurations from the management servers will be written to this directory, allowing the driver to come online when a central manager is not available;
7. An option to ship the proxy or driver logs directly to the management server, avoiding the need to review application logs to find Heimdall messages;
8. Exposing the VDB level debug option as a checkbox to simplify debugging–per client Heimdall JDBC URL use of the debug option will continue to be supported to allow individual nodes to enable debug mode;
9. Adjusted the “delay” rule property to operate only after cache logic is complete.  This effectively allows before and after cache performance to be evaluated in conjunction with WAN latency simulation;
10. On the MySQL proxy, allowing the proxy to honor the database catalog “use” command, as well as the initial catalog desired on connect.  The proxy now supports multiple catalogs through a single proxy instance–PostgreSQL does not support the “setCatalog()” JDBC api to support this.
11. On all proxies, the support for multiple user logins, including adjusting users at runtime without a proxy reload.  If the user is not allowed to connect to the database, the connection will be refused.  This option, along with the catalog selection, are connection pooling compatible, but may reduce the effectiveness of the connection pool when used heavily.
12. user:<user1>,<user2> matching support to allow any rule type to apply to a given user;
13. Forwarding now has two new options:  setcatalog and readonly.  If readonly is true, then a read-only server in the target source will be considered eligible to serve forwarded requests.  If the setcatalog option is true, then the catalog on the target source will be forced to match the source connection’s catalog.  This provides the capability to redirect queries to alternate catalogs if set to false.  The combination of the two allows for effective read-write split, and is used when read-write splitting is selected via the wizard.
14. Improved read-only account security, to prevent password leakage for restricted users;
15. Removed Memcached, EHcache and Jcache interfaces, as they were insufficient to provide the best experience for customers.  Currently, only Hazelcast and Redis are supported.  The main limiting feature was the lack of pub/sub interfaces, which are now used for messaging between nodes for optimal efficiency;
16. Simplification of configuration to remove unnecessary options, including those that could be auto-detected based on the environment (such as Redis cluster mode);
17. Adjusted how driver dependencies are packaged, to make them invisible to host applications in JDBC mode.  Grid-cache libraries are now included inside the jar files, and don’t need an external lib directory (removed from the packaging).  Please report any other class loading conflicts or issues if observed;
18. Many other minor bugs and improvements;
19. Initial preview build of Microsoft SQL Server support as a proxy.  This is still a work in progress, and should not be used in production.  Currently, transaction tracking is not fully online, and the SQLVariant result type is not yet supported, along with other corner cases.

Release version 1.0.13 (Apr 17, 2017)

Note:  Release 1.0.12 was provided to many customers in a variety of pre-release forms, so it was decided to skip 1.0.12 as a formal release

In this release:

1. Improved compatibility for both the PostgreSQL and MySQL proxies, including full prepared statement support for both;
2. Improved performance for proxy mode operations, both in memory usage as well as CPU overhead.
3. The MySQL proxy automatically sets “best practice” properties for the JDBC driver, to provide compatibility to non-Java applications.  These properties are displayed on startup, and can be overridden in the data source properties configuration.  Connection pool properties are also set this way as well and also able to be overridden.
4. tables:<tablename1>,<tablename2> matching support to allow any rule type to apply to a given table;
5. ${catalog} or ${database}, ${user}, and ${vdb} expansion in rule properties and table selection to improve dynamic configuration for simplified rules;
6. Catalog selection support in the MySQL proxy, allowing “use <database>” commands to be honored.  To best use this feature, specify a JDBC URL without the database/catalog specified;
7. Forwarding support to specify the use of read-only servers in the target data source, as well as to either inherit the parent connection’s database/catalog, or allow it to be overridden;
8. The ability to ignore matching queries from statistics, to allow the dashboard to reflect only user-impacting query patterns, i.e. to exclude background batch operations;
9. Improved logging in debug mode (specify a VDB property of “debug” with a value of “true” to activate), including a prefix specifying which module the log entry applies to;
10. Enhanced diagnostics on non-Windows systems, by sending the proxy/driver the HUP signal.  This data is also generated every 24 hours automatically.  Windows does not support a HUP signal.  If the parent application in the JDBC case has registered the HUP signal, it will attempt to register the signal handler on SIGUSR2–this will be logged on startup.  This will include per-thread stack traces to debug deadlocks conditions, per-connection SQL activity, all SQL patterns observed in the last two hours, and per-table performance statistics;
11. Streamlined the per-minute statistics generated by the system to only include relevent information, including process cpu usage, garbage collection stats, and overall performance stats;
12. Huge number of bug-fixes and overall optimizations;
13. Build times are now displayed on startup, so that the actual build version can be tracked independently of the release version;
14. Additional environment information, such as jar files in the classpath, are displayed on startup, to help debug class load issues;
15. When using the MySQL proxy, if the server isn’t set to negotiate to UTF8MB4, warnings will be displayed on startup.  This is necessary to fully support all character sets.

Release Version 1.0.11 (Feb 28, 2017)

New in this release:

1.  Improved auto-tune functionality
2. MySQL JDBC defaults are now built-in when using the MySQL proxy.  These JDBC settings are displayed on proxy startup, and can be overridden in the data source property configuration.
3. Improved PostgreSQL protocol handling.
4. AWS Configuration Wizard–allows RDS and Elasticache settings to be read out of the AWS configuration, to populate the VDB, Data Source, and Rules tabs automatically.
5. General bug fixes.

Release Version 1.0.10 (Jan 24, 2017)

This release is a smaller release, and provides the following:

1.  Auto restart on update–before, the upgrade of the server would not restart the server.  As this can cause java class load issues, it will also restart the server after mailing any notification on a successful update.
2. Autotune on the cache configuration–when enabled, the vdb will process in a learning mode before caching, and will disable caching for tables that are written to often, or that result in cache hit rates that are non-beneficial.  This tuning adjusts based on the cache configuration, and will account for overheads in putting objects into the cache that end up not providing a benefit.
3. MySQL character set debugging information is now available on startup of the driver, to help insure that the character sets are correct.  For correct behavior, utf8mb4 should be used on the server, and should be inherited on the client without overrides.  Please see the sample configuration for a full list of recommended MySQL configuration options.
4. Improved compatibility for XA and non-XA data source configurations.
5. Improved PostgreSQL proxy compatibility.  Current, binary transfer formats are not supported.

Release Version 1.0.9 (Dec 15, 2016)

This release includes significant changes to the overall configuration of caching.  Instead of cache configuration parameters being in the cache rule, it is now configured in the VDB section, and provides GUI driven options to simplify the process.  In addition, proxy settings are now configured in the VDB tab as well, and the proxy can be configured to be started and managed by the central manager as desired.  Postgres proxy support is now added in addition to the MySQL proxy, although these proxy options should be considered beta for the moment.  The Linux & vendor neutral packaging has been updated to include Docker and Packer samples for creating docker containers for the central manager/proxy and AWS AMI builds.

A public AMI has been made available (please search for “Heimdall” in community AMIs in the US-East region) to simplify testing.  The default GUI user is “admin” and the password will be the instance ID of the system, and the AMI has been scanned and passed in preparation for AWS Marketplace submission.

Release Version 1.0.8 (Oct 24, 2016)

Release 1.0.8 includes the following

1.  VDB level properties–these should be used to configure the cache instead of applying them to the cache rules.
2. Debug mode–if the VDB property “debug=true” is set, then this will activate verbose logging of the driver.  Please use caution when enabling this, as it will impact performance.  Most actions will now account for their operation when this log is enabled, to help verify that the desired behavior is occurring.
3. Support for Hazelcast-client instead of embedded behavior.  Please set the vdb property “cacheconfig” to point to the hazelcast-client.xml file to configure the upstream servers (this should be in the current working directory of the application).  The server/embedded XML configuration can be specified this way as well.
4. Improved support for Redis and Redis cluster configurations.
5. Added clear all logs button to Analytics dashboard to simplify the test process.
6. Added the ability to specify ${vdb}, ${user} or ${catalog} in the forwarding rule source properties, to have these values filled in dynamically, improving the reusability of rules in certain situations.
7. All grid-cache libraries are now merged into the heimdalldriver.jar to prevent classpath issues loading the drivers.  If class conflicts cause issues in your application,  please contact Heimdall support for a stripped driver that requires external dependency jar files.
8. Parameter help from the driver–on startup, the Heimdall Driver will dump a list of all parameters and the context they can be used, along with default values.  These will be exposed directly from the server in the next revision of code, to guarantee help documentation matches the version of software being used.
9. Per-driver table usage statistics–each driver will print a running documentation of tables observed, and statistics about their usage.  These can help tune table-level cache configuration.  Future revisions of code will expose this data on the analytics tab.
10. Fixed a bug that could account for non-cached queries twice in the query per second statistics under certain situations.

Release Version 1.0.7 (Oct 4, 2016)

Release 1.0.7 includes all the fixed intended for 1.0.6, but as several customers had been using internal builds of 1.0.6, we have chosen to skip 1.0.6 as a formal release to insure that update notifications operate properly for those users.  Additionally, this release includes:

1.  Fix for EHCache performance issues, but also reduces the ability to control the size of the cache.  It was found that the logic to control cache size by bytes was extremely expensive with EHCache, so we have tuned it to dynamically compute the number of cache objects based on size/1000.
2. Provide an option for “Verify Cache” on the VDB–this will result in the object being retrieved from the cache AND from the back-end database, then compared.  Alerts will be written to stdout indicating if an object compared differently, and will provide a diff of the difference or other details on what was different.  This is to be used to validate that the cache is performing properly, and returning correct results before being used in production.
3. Release packaging changed from heimdalldemo to just “heimdall” due to confusion over if it was a beta release only or not.

Release Version 1.0.6 (formal release skipped)

IMPORTANT:  Release 1.0.6 changes the password encoding format to exactly match RFC-2307 for SSHA.  Previously, it was incorrectly encoded in hex vs. base64.  As such, upon upgrade, any users will need to have their password fields reset.  If you need assistance with this, please contact Heimdall Support.

1. Provide a Heimdall JDBC URL option of peerconf=true, enabling one JDBC instance to leverage Hazelcast to pull configuration information from another JDBC instance, in the event the management server is down.  Both instances must be configured with this option at startup for this feature to function.
2. Access control changes:
   • Move password encoding to strictly follow RFC-2307 for SSHA.
   • Fixed password changes not changing the password.
   • Added a read-only flag for users.  This flag can be used with client/application server logons to better secure the configuration management.
3. GUI Changes & Fixes
   • Improve PDF handling to prevent large pages from being cut short.
   • Improved graph behavior and memory utilization.  Stats API now includes streamlined data structures for use with the Charts.js package to simplify custom dashboard creation.
   • Added SQL Highlighting support to make the analytics tab easier to read.
   • Resolved the placement of one-click rules, so that they don’t always to go the top of the rule list.
   • Added a one-click Async configuration option.
   • Resolved a Null pointer exception that would be triggered by using the VDB cache reset button.
   • Added an API to force reloading of configurations from the disk via /api/config/reload.
4. Numerous fixes to various compatibility issue in resultset Metadata, which in rare situations could impact an application.  Now, testing pulls through Heimdall and direct to the database server, and verifies that all fields and data match to the byte, including in Blob and Clob field cases.  These fixes also resulted in a measurable performance boost, in particular in high-cache hit rate situations.
5. Improved simplifying of SQL, in the case where a field includes an escaped quote.
6. Added “rediscluster” as a cache option, leveraging the Lettuce libraries (non-cluster Redis still uses Jedis).
   
7. Default “gridcache=jcache” to use Hazelcast’s jcache class provider for easier testing of Jcache.
8. Added a cache option of “uselocal” which if set to “false” will prevent the use of any local (non-grid) cache.  May cause performance slowdowns in particular on heavy cache writes.
9. Added a cache option for “serialize” to force serialization of objects when stored in the local cache.  While this adds overhead on both store and retrieval, it allows much more accurate cache size numbers to be tracked.  Memory usage of the grid-cache interfaces will continue to be unaccounted for in most cases.
   
10. When serialization is not in use, cached objects are now measured by the number of rows, and assumed to be 1kb per row for memory utilization purposes.  This may result in over or under-estimated cache usage.
11. Added the ability to specify a Hazelcast configuration XML to override built-in configuration options.
12. Enabled enforcement of the enable/disable flag on the data source configuration objects.
13. Beta test support for a MySQL protocol level proxy.  This code allows applications such as WordPress, Magento and others to leverage Heimdall functionality for caching, security and reliability.  Please contact Heimdall support if you wish to test against this beta feature for more information.

Release Version 1.0.5 (May 23, 2016)

Release 1.0.5 is primarily a bug-fix release, resolving the following issues:

1.  A packaging issue with the 1.0.4 release vendor neutral package is resolved, which if installed in a non-root account could cause problems with driver download on an application start;
   
2. Resolved two issues preventing the cache button from properly identifying patterns that were covered by a cache rule already;
3. If the Heimdall server component was restarted after the application was started, and the server had incremented the configuration counter, then the driver may not pull new configurations until the configuration version matches what had been set by the previous instance of the server;
4. Improved database failure detection with multiple application servers;
5. Improved thread behavior on the server to help increase the performance of logging.

Additionally, two new features have been introduced:

1. A “stack trace” rule has been added, to help identify the source of queries, including an option to only generate a trace if a query is slow;
2. A default option on tag rules, to allow the creation of a default rule template for a long rule list.

Release Version 1.0.4 (May 13th, 2016)

1. Add rate limit policy parameter to specify the queries matching the policy to X queries per second.   Allows bursting for up to 10s worth of queries, then will back down.  Rate limiting follows the “leaky bucket” model.  To use, specify the parameter “ratelimit” and a value in queries per second allowed.  Traffic exceeding this will be throttled to the limit.  The maximum burst size is based on ten seconds worth of traffic by default, although this can be overridden using the “maxburst” parameter, and is the number of seconds worth of traffic that can be burst before pacing occurs.  Once throttling is enforced, pauses will be for no smaller of an interval than 1ms.
   
2. Add a match rate limit parameter.  Similar to the rate limit, but restricts how often a particular policy will be matched.  This is particularly useful to limit the volume of logs to at most X log entries per second to prevent log overflow and reduce overhead on logging.  To use, specify “matchlimit” with an integer numeric value, for matches/second allowed.  The “maxburst” option is also used by this parameter.
   
3. Add a “stop” parameter, to indicate rule processing should not continue after a match on this rule, accepts the value of “True” or “Yes”.
4. Call action for a parameter–this allows nested rule-lists to be created, i.e. to allow one rule-list to call another rule-list for cache policies, another for firewall policies, etc.  Execution is continued on a rule-list after the called rule-list completes. 
5. Add a “learn pattern” function to learn query parameters.  On a match to the learn pattern policy, a new policy will be created in the named rule-list, of the type specified.  If no rule-list is named, it will be added to the current rule-list, and if no type is specified, it will be an “allow” action.  The primary purpose of this is to allow learning of normal traffic, and automatic generation of whitelist rules, allowing a drop of unknown traffic to be usable.  It can be used to generate other rule types as well however, allowing fine-tuning of traffic on a per-query pattern basis.
6. Allow action type–in conjunction with the learning mode, an allow action will force the bypass of any further “allow”, “drop” or “learn pattern” rules.  This bypasses all rule processing except for sample, exclude or matchlimit processing.
7. In support of the call function, now multiple rule lists can be attached to a vdb.  If not attached, a called rule-list will not be executed, as it will not be provided as part of the configuration for the driver.
8. Enhanced configuration server logging to report the status of the monitored servers in a more visually useful way.
9. Added a summary tab, to provide an overview of the benefits that optimization is providing, if enabled.

Release Version 1.0.3 (May 1, 2016)

1. Added PosgreSQL statechange script, with docker integration for full redundancy of PostgreSQL.
2. Added full Postgres+Jira install script in the jira directory in the platform neutral install package.  Tested on Ubuntu 16.04 LTS, and allows fast, easy setup of redundant Postgres servers and the latest Jira via Docker.  When run, it will setup the Heimdall instance to manage the PostgreSQL instances and will configure Jira to use them.
3. Improved logging for failovers.
4. Added the ability to flag a rule with a “capture=true” property, allowing property values to be modified by regex capture groups, much like transformation does.  Example rule using this:
   regex: “^/\* (.*) \*/”, action: “forward”, parameter: target=”source-$1″, capture=true.  Target name would extract from the SQL comment and add to the data source name, for dynamic data sourcing.
5. Improved logging of connections, to match what is logged in the query logs.
6. Added the ability to monitor PostgreSQL pg_stat_statements on a data source by adding a connection property “pgmon=name”, where name is the vdb the data will be associated with.
7. Resolved an issue with batch execution, where it would report an exception, even though the batch had been executed properly.
8. Added logging of newly observed patterns in the logs, even with SQL logging disabled.  This can be useful for helping track security issues after the fact, yet in most configurations will result in very little logging.
9. Add several new action types:
   • Drop:  Results in an SQL exception for a matching query
   • Evict:  Accepts a list of tables to evict, or the keyword “all”.  Failure to list tables will evict all.  Also accepts a parameter of “olderthan” which can specify an age an object can be to avoid expiry.  This value can be negative to prevent objects from being returned from the cache for a given period of time.  This value is in ms, so 10,000 would be 10 seconds.
   • Ignore:  Accepts the name of a ruletype following to ignore, will honor order of operations.  Table cache rules can’t be ignored this way.
   • Tag:  A no-op action that accepts global parameters, i.e. tables and delay.
10. Added new global option “printtables” to print the tables extracted for queries to the app’s stdout, for debugging table extraction issues.

Release Version 1.0.2 (March 2, 2016)

A number of issues were resolved:

1. If a rule was not bound to a vdb, then the “log sql” option was not logging SQL.  Even an empty rule would resolve this.
2. In certain corner cases, SQL parsing was triggering a loop in isolating tables.
3. When using table caching, and a whitelist configuration, grid caches were not being initialized.
4. A memory leak was resolved when applications were not closing connections due to connection pooling, and were not reusing statements.

Many improvements were also made:

1. Extensive optimization to lower overhead.
2. Improvements were made in the performance logging to more accurately log the execution time of queries.
3. The trace action has been removed, and rolled into logging, to simplify configurations, and to better support the analytics only setup.
4. Added a generic local cache layer above grid caches.  This improves performance on ingestion as well as maintains a small number of cache entries in local heap for faster retrieval.  The size of this local cache is computed from the cache size, at 1/1000 of the configured cache size, but in entry count.  A 50MB cache setting will result in up to 50k objects being maintained locally.  To only use the local cache, the grid cache name of “none” can be specified.
   each page now has an export to pdf option to save the contents of the page.
5. updated the default grid-cache to Hazelcast 3.6, and Hazelcast near-cache was removed, as a generic equivalent was added for all grid caches.
   
6. removed any direct use of the java class “sun.misc.Unsafe” in advance of Java 9 being released, which may remove support for this.  Grid libraries may still use this however.
   
7. Heartbeat packets to the central server now report memory usage by the JVM, to help track down heap utilization issues after the fact.  Future versions of code may allow this to be analyzed in the GUI, but currently it is logged for offline analysis.

Notes on behavior:
The memory reporting and control of cache size is very inexact, as with Java, it can actually be costly to compute the size of objects in memory.  The local cache simply uses an estimate that each object will take up 1000 bytes, but on the grid-cache level, it will query, when available, an appropriate api to ask the grid how much memory it is using.  This is supported on Hazelcast (when using the default binary format) and EHcache, but not on other caches.  In such a case, the memory usage of these caches will not be reported at all.

Version 1.0.1 Initial Release (Jan 18, 2016)

The significant change in this build is the release model.  As of this build, pooling, logging and analytics are free.  Additional features including any rule processing and load balancing are premium features.  Please contact us for a quote for your install if you are interested in these features.  Additional changes include:

1)  Resolved a set of issues involving JDBC batch execution, both with logging and actual response exceptions;
2)  Minor cosmetic changes

License behavior:  In order to ensure that customers don’t ever have an outage caused by a license issue, the license model is currently an advisory model–it will tell you if your use of the system is out of license, so you can correct it or contact sales to obtain a new license.  Otherwise, features are free!  Please check the “Files” tab for information on what is allowed.  For the first 30 days, all features will be enabled for up to 4 client systems/application servers and up to 4 load balanced database servers.  Clients not accessing VDBs with rules will not count to this limit, nor will data sources that are not load balanced.

As always, please contact support if you have any issues with this product.

Version 1.0 Beta 4 Release Notes (Dec 29, 2015) — Major Upgrade
IMPORTANT NOTES: 

1) For caching to operate properly, either the Hazelcast jar file needs to be in the classpath used by the driver, OR an alternate grid cache provider needs to be specified in the cache configuration.  As part of a partnership with Hazelcast, we are now defaulting to using Hazelcast as the primary cache provider.  This will automatically provide shared cache resources if multiple JVMs are used on the same server as an added benefit.

2) The interface between the client and server has been adjusted, and requires that both be upgraded together.

New Features:
Hazelcast is now default:  The internal cache subsystem has been removed, and Hazelcast has been added as a default.  It is configured with a near-cache as well as a local shared cache setup, allowing processes to share their cache without additional configuration.  As a side-effect of this, the cache memory graph is only an estimate of the memory used, and is often higher than is actually being used.

Table level locking during Asynchronous Modifications.  When using the Async Execute method, the table the query is executed against is detected, and the table locked from reads until that execute has completed, if the execute is made on the same JDBC connection.  This should prevent most cases where a read expects a write to have been completed.  Further, in conjunction with our table based expiry, it will also insure that the data read will not be stale content if caching is also in play.

Cache clear button:  Each VDB now has a button to purge the cache for that VDB.  This button will not actually purge the cache memory, but sets a watermark that any object retrieved from cache older than the current time will be considered stale, and purged on retrieval.

Log management:  The files tab now has log management buttons.  Clearing the raw log data (or all logs) will clear both the on-disk logs and in-memory log data that has not yet been written, allowing a clean start when comparing behavior changes.

Automatic log analysis on log roll:  On a log roll, the data in the log will be summarized automatically, resulting in significantly faster “analysis” operations, in particular under high log volume.  Log analysis is also done in a parallel manner, with one thread per VDB, to speed up analysis.

Removal of the trace action:  The logic for trace has been rolled into the log action, and removed from the menu.  If a query returns a result-set, a trace record will be generated.  If it was a cached result-set, only partial trace data will be included, to support timing operations.

Behavior Changes:

Async batching removed:  When using Async Modifications, no batching will occur.  This conflicted with the locking to prevent reads, and impacted performance during batch creation, so was removed for now.  This may be added back in as an option in the Async command if requested.

Catalogs need to be specified in table caching:  In order to guarantee consistent behavior across table caching, table purging, and table blocking for async operations, the table name has been normalized to include the catalog.tablename.

Autocommit caching:  To prevent accessing the database to find out the state of Autocommit, the value is now cached in the Heimdall driver.  If a command outside of the JDBC API causes this to change, it will not be tracked by the driver.

Analytics tab columns:  The “cache benefit” column was deemed confusing in intent, and has been broken into a server time % and duplicate query and response column.  The previous behavior was to multiply these together, but this ended up giving unreasonably low numbers on what can be expected.  These are now broken up for individual analysis.

VDB Cache Memory limit:  When used with EHCache, this is enforced, but as Hazelcast doesn’t have an explicit memory control, it is not enforced at this time.  Instead, this value is divided by 1000 to set the object limit in the cache.

Bug Fixes:
Properly escape single quotes when used as part of a prepare-execute, for logging and other purposes.
Improved response time reporting in several cases.  The logs should be more accurate than before.
Removed reference to some SSL functions in the driver code.  This may have impacted some environments using SSL depending initialization order.
General cleanup of code and consistency of behavior.

Version 1.0 Beta 3 Release Notes (Nov 30, 2015) — Major Update
Improved compatibility was added for trigger based expiry for databases other than MySQL.  The trigger expiry system now provides example configuration for Microsoft SQL Server, and allows customization of the stored procedure calls made by the Heimdall driver to the database.  The driver will now also attempt more than one connection try during a health-check based on the data source connection parameter “failcount”, which defaults to a value of 3.  The state change script now executes each output command as they are received from the script, allowing multi-state “commits” of the configuration, allowing all data sources to be disabled for a period while orchestration brings online new data sources.  Help pages have been updated to document the behaviors, including a new load balancing section in the data sources help.  Help for the Analytics tab has also been added, providing detailed explanation of the result columns.

Version 1.0 Beta 2.2 Release Notes (Nov 22, 2015) — Minor update
Further improvement in HA, to improve the behavior of a server with a weight of “0”.  This value can now be set to specify a “server of last resort” for a data source, which can be used immediately by a driver when other higher weight servers are not accessible.  This is best used when a server cluster is setup as active-active, but only one is intended to be used at a time, but both in theory could be used at once.  This allows nearly instant failure to be performed without synchronizing with the central server.  For best results, this should be used in conjunction with a fail-over script that will guarantee consistent behavior between all nodes in the event of a split-brain network issue in a short period of time.

Additionally, the script name that is used to initiate centralized failure configuration changes has a new naming scheme.  Now, instead of it being a static name of “statechange.(sh|ps1|bat|py|pl)” it will now append the name of the data source that failed when finding the script to “statechange-“, i.e. instead of statechange.bat, it will look for statechange-dbdemo-mysql.bat for the servers in the example configuration.  An issue in locating this script on Unix systems was also resolved.

Version 1.0 Beta 2.1 Release Notes (Nov 21, 2015) — Minor update
Minor update improving HA functionality, including adding a connection property of “failCount”, specifying the number of times a health check needs to fail before a server is flagged as down.  If not specified, the default value of “3” is used, corresponding to a three second outage..  Download link and documentation remains the same.  Failover should also be more reliable under all situations (pooled and non-pooled setups).

Version 1.0 Beta 2 Release notes (Nov 17, 2015) — Major update

New Features:
Asynchronous Database Modification
Insert, Updates and Deletes can now be flagged in the rules to operate in an asynchronous way.  Any such calls will be immediately returned with a success flag, and will be executed in batch mode against the database.  When performed against many inserts of the same type, this can result in a 10x improved insert response, depending on driver support for executeBatch.  It is suggested that when using this feature on MySQL, the connection parameter of “rewriteBatchedStatements=true” be used.  Note:  If the call has been flagged to provide auto-generated keys, then it will automatically bypass the async feature for proper operation.  Additionally, only queries flagged or detected as an “update” will be processed this way.

Database Trigger Synchronized Tables
This feature allows the Heimdall Driver to synchronize its cache with the database server, to insure that cache entries are expired within one second.  This feature requires the use of trigger configuration on the server, which is a feature supported by all major RDBS vendors.  An example script for the MySQL database is available at http://download.heimdalldata.com/mysql-triggers.sh and can easily be adopted to other environments.  Contact customer support if you wish to have an example script created for other database products.  This feature will autodetect if the proper configuration is in place on the server side, and will report in the application log if it was disabled after probing.

JSR107 aka JCache support
A generic JCache interface has been created, and now replaces the Hazelcast option, as Hazelcast was being used in JCache mode.  Please see rule documentation for details on how to specify this.  Note:  Apache Ignite has been tested, but appears to have an issue where memory garbage collection can impact performance even under under heavy read load.  The EHcache interface does not have this issue.

Explicit Tables and Update Flag
As stored procedures may or may not make modifications to tables, rule parameters of “tables” and “update” has been added to allow the internal logic to be overruled, and tables to be specified and update to be flagged.  This allows proper purging of tables on a stored procedure call.  This can also allow better support for purging of underlying tables when a view is used on the database.

Induced Delay on Rule
In order to help determine if a cache policy will impact performance, a simple, if counter-intuitive option is now available:  induced delay.  By adding a cache policy with a TTL of 0, then a second attribute of “delay=x” in milliseconds, you can add additional latency to queries, then measure the application performance change.  If by doubling the average response time, you notice only a 1% change in actual application performance behavior, then caching will at best have an inverse performance benefit.  This can provide a safer mechanism for testing, as the risk of stale data is not present when used this way.

In addition, many issues and minor improvements have been resolved:
1. Reduced overhead of resultset handling, improving performance by up to 66% on high-cache hit workloads, along with improved performance when using prepared statements.
2. Allowed sub-second cache TTL even on grid caches that don’t support this (ehcache, redis, memcached, jcache, etc)
3. Impoved TTL handling on Ehcache and other grid caches, including randomizing an object’s TTL value from 90% to 100% of the specified TTL, so that if a large number of cached items are inserted at once, they will not expire at the same time.
4. Corrected an issue with the analytics tab when running on Linux and similar environments, where only the most recent log data was processed.
5. Improved table extraction to account for more cases
6. Fixed an issue when Clob responses had the toString() method called on them, when served from cache, and if the clob getSubString() method had a length specified of zero.
7. Adjusted license enforcement to only count clients and database nodes, cpu cores, jvms and other license artifacts have been removed.
8. Always log rollback and commit methods, to allow transaction time to be computed on a connection.  Additionally, any method that takes over 100ms to execute will be logged as well.
9. Improve support for older ehcache versions, to as old as 2.1.0 (May, 2010)
10. Adjusted JSON processing to ensure that all messages are sent to the log server and processed correctly.
11. Adjust the log reporting of prepared statements to report the connection the resulting statement was prepared on instead of the prepare itself (for ease of debugging)

Known issues:
HEIM-45:  In Internet Explorer and Edge, dashboard graphs do not refresh autocratically.  They have been tested to work properly on Firefox and Chrome.
HEIM-124:  If you delete an active ruleset from the GUI, the change isn’t pushed out.  To remove a ruleset in use, first unbind it from the VDB, then delete the ruleset.
HEIM-135:  Due to compatibility with Java 6, many Java 7 and Java 8 JDBC methods are not yet supported.  Please let Heimdall support know if particular methods are required for your application.  Most applications and frameworks limit JDBC calls to those supported in Java 6, so this should not impact most applications.

Version 1.0 Beta Release Notes (Oct 30, 2015)
Initial public release made.  Installation guide can be found at http://download.heimdalldata.com/downloads/Heimdall_Data_Access_Platform_Installation_and_Setup_Guide.pdf, and the Windows 64 bit release image is at http://download.heimdalldata.com/downloads/heimdalldemo.msi.  The platform neutral install package is pending, and will be available at http://download.heimdalldata.com/downloads/heimdalldemo.tgz shortly.

Known issues:
HEIM-45:  In Internet Explorer, dashboard graphs do not refresh automatically.

HEIM-124:  If you delete an active ruleset from the GUI, the change isn’t pushed out.  To remove a ruleset in use, first unbind it from the VDB, then delete the ruleset.

HEIM-133:  When a query contains two or more table names in the from clause, if there isn’t a space after the comma dividing them, then it can result in the table name not being extracted properly.  This can impact table based caching and table based purge during writes. (resolved in Beta 2)

HEIM-135:  Due to compatibility with Java 6, many Java 7 and Java 8 JDBC methods are not yet supported.  Please let Heimdall support know if particular methods are required for your application.  Most applications and frameworks limit JDBC calls to those supported in Java 6, so this should not impact most applications.