Release Notes

Current release versions can be downloaded here.

Known issues: If a query contains more than 32k positional parameters for Postgres, this can cause a query to fail. This is due to a limit in the underlying JDBC driver for Postgres. Postgres itself is limited to 64k positional parameters. To resolve, ensure that no more than 32k positional parameters are used.

Important note on driver updates!

In new versions of the Heimdall packaging, we are removing the actual driver files for the JDBC connection.  This is for various reasons, including legal ones in the case of the MySQL driver.  If your instances have internet access, the download of the correct drivers should be done transparently on update however, and no further action is required.  If it is firewalled from the internet, you can manually load the drivers with the following steps.

1.  Download the driver
   1. SQL Server: https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/9.4.1.jre8/mssql-jdbc-9.4.1.jre8.jar
   2. MySQL: https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar
   3. Postgres: https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.1/postgresql-42.3.1.jar
2. Install the driver by choosing the file downloaded in the associated driver section;
3. (Optional) lock the driver in place by unchecking the “Auto update” option at the top of the driver
4. Commit.

If you have any problems with the new drivers coming online even with internet access, please go to the driver tab, select the driver you use, and then select commit.  At times this appears to be necessary after the download, due to a timing issue that we are working to resolve.

Development build 22.01.09.1 (Jan 9, 2022)

1. Swap instanceid and vdb in AWS metrics to allow easier sorting by vdb;
2. Remove excessive logging in verbose debug mode for lag detection;
3. Remove documentation for the “get keymap” command which is not currently available;
4. Resolve NPE in a particular case with multiplexing enabled;
5. Resolve issue where drivers are not always registered on the initial startup, requiring the driver to be re-downloaded or a commit issued on the driver tab;
6. Correct error when an incorrect password is used, and a TLS error is generated for the Postgres proxy;
7. Update parsing with “with” statements, where some tables were not extracted in certain cases;
8. Significant code cleanup to remove tech debt.

Release build 21.12.21.1 (Dec 21, 2021)

1. Library updates, including the removal of all libraries under the CDDL, MPL, EPL and GPL w/ CPE licenses.  This is to simplify the release process for GCP, due to their strict interpretation of how these licenses need to be handled;
2. Implement a parameter consolidation change to simplify queries with a large number of parameters;

Release build 21.12.14.1 (Dec 14, 2021)

1. Due to a new version of log4j being released to completely fix the security issues found, we have updated log4j to 2.16.0.  Further, we have updated nearly every other library to ensure that as of now, even the strictest scanners will validate the security of all the libraries used.
2. Add code to wipe central manager analytics if memory usage becomes too high automatically, as this is the most likely cause of any issue;
3. Added code to condense parameters in queries for logging to a few, avoiding issues where millions of parameters would result in queries not being logged due to their size;
4. Removed an extra debug line in the previous build that could cause excessive log entries, even when verbose debug mode was not enabled;
5. Added retry logic on auto-driver downloads to avoid issues if there is a transient error;
6. For AWS Marketplace images, add the RDS CA certificates to the image by default to allow drivers to validate the CA cert issuing the database cert.

Release build 21.12.11.1 (Dec 11, 2021)

1. While we believe there is no remote exploit available, log4j version was updated to 2.15.0 to ensure we are not vulnerable to cve-2021-44228 and of course: https://xkcd.com/2347/;
2. Resolved an issue with AWS autodetection of regional clusters;
3. Dramatically improved regression tests, although these didn’t result in any actual code changes;
4. Improved warnings if there are overlapping ports for DNS or HTTP monitoring;
5. Throttle certain warnings that could generate large numbers of messages;

Release Build 21.11.24.1 (Nov 24, 2021)

1. When using the test source button, use the same default properties that the driver uses internally, so that they behave the same.  This is in particular important for TLS behavior, so that the test is accurate;
2. Dramatically improve lag detection logic, so that it can track much closer to the real lag.  Now, it uses a stored procedure or function call on each database as part of the lag detection;
3. Resolved issues with SQL Server using array parameter types;
4. Resolved an issue preventing pgbench from working when loading data via the COPY operation;
5. Add the option to run proxies as systemd services, even when on the same server as the central manager.  This allows manager restarts without disrupting traffic through the proxy;
6. Improved the AWS RDS detection code;
7. Removed (currently) unused MySQL8 driver configuration and files;
8. Resolved other customer encountered bugs and discrepencies;
9. Improve the naming of servers when using AWS autodetect and cluster tracking, so they are in alignment;
10. Added the ability to route traffic to the nearest reader node by latency, for global deployments;
11. Added many more regression test cases.

Test Build 21.10.16.1 (Oct 16, 2021)

1. Resolve an issue when updated from older code that may result in the MySQL JDBC driver not being present;
2. Resolved a compatibility issue with SQL Server when using “table value” parameters in prepared statements, while also using forwarding or read/write split;
3. Added a new “multiplex” rule type that allows saving of named parameters (such as cursor names) when disabling multiplexing, and removing them to enable multiplexing again.  This allows nested cursors and other types of queries to trigger multiplex disabling in complex situations;
4. Resolve an issue with parsing when a comment is at the end of the query;
5. Resolved an issue with Postgres with array parameter types in prepared statements;
6. Improve nocache reasons in several cases.

Release Build 21.10.01.1 (Oct 1, 2021)

1. All changes carried forward from prior test build
2. NOTE:  If you have issues connecting to the MySQL database after upgrading, please enter the following value in the “maven id” in the MySQL driver section:  “mysql:mysql-connector-java:5.1.49”.  This should resolve the issue.

Test build 21.09.29.1 (Sept 29, 2021)

1. Integration with and support for RDS Global databases for auto failover;
2. Support for latency based reader selection in multi-az and region configurations;
3. added metrics to track jvm heap allocation rate and hiccups;
4. updated RHEL install logic for newer Redhat versions;
5. Fixed issue with the self-diagnostic http health check where it was using the non-lb url for all checks, and didn’t adjust with changing configurations after start;
6. Added synchronization of TLS certificates so all proxies will use the same certificate and updated tls certificate management documentation;
7. Added a knob to enable TLS 1.0 and 1.1 for older drivers that don’t support tls 1.2+;
8. Significant improvements to the Cloudformation template to make it more flexible and cover more customer environment configurations;
9. Resolved issues with using TLS for proxy to manager communication, and changing the base image to use HTTPS by default for such communication;
10. Added to DNS redirection mode the ability to redirect to AWS public IP’s, not just private;
11. Resolved issues with Hazelcast AWS autodetect logic;
12. Allow downloading of drivers from Maven central to avoid packaging the MySQL driver for legal reasons;
13. Resolved issue with DNS LB when more than 10 proxies were available, impacting proper load distribution;
14. Improve documentation for heimdall sync user functionality;
15. disabled vdbs will now be put at the bottom of the status tab;
16. Improve Postgres copy support;
17. Removed the “track” option from the admin software tab, now alerts will only be for release versions;
18. Alerts added if the cache client isn’t ready;
19. Fixed missing SQL queries for SQL server stored procedure calls;
20. Added support to cache MySQL stored procedure calls, although they require a table attribute before cache will work;
21. Provide additional tooltip info for servers on the info tab;
22. Add 1GB of swap space by default to AWS images to help avoid out of memory situations;
23. Vastly improved nocache reasons, including the ability to report multiple reasons at once;
24. Improved compatibility with all database protocol types;
25. Detect when binary values are used for primary key inserts and handle it properly when using client-side prepares at the driver level for MySQL;
26. Adjust images so that initial console logging can be observed with “journalctl -u heimdall.service”;
27. Huge improvements in test infrastructure to improve regression coverage.

 

Release build 21.07.30.1 (July 30, 2021)

Note:  This build was pushed out to update the AWS ARM instance only for now

1. Add a 1GB swapfile to avoid out of memory errors on the proxy
2. Change the systemd configuration so that the stdout and stderr logs will go to /var/log/messages vs. being thrown out, to assist in debugging issues

Release Build 21.07.23.1 (July 23, 2021)

1. Adjust the yellow alert on the status tab to show on 5% heap vs. 10% heap free;
2. Generate gc overhead alerts based on total cpu usage, not just a single core;
3. Add missing useServerPrepStmts to mysql property list;
4. Ensure a self-signed certificate is generated by the management server on first startup, and port 8443 opened for secure access;
5. Provide an option to download a software image backup on a software update, allowing old versions to be easily archived by users for downgrades;
6. Resolve an issue with mysql with multiple result-set responses and another with prepared statements in certain rare situations;
7. Improved the overall readability of the status tab;
8. Include dumping the SQL Server AG status on a cluster detect operation, to allow debugging if the detection process fails to operate cleanly;
9. Significant performance optimizations on the cache path with multiplexing, resulting in 7x improvement in qps for 100% cache hit situations.

Test Build 21.07.19.1 (July 19th, 2021)

1. Added the ability to store central manager configurations into the AWS Secret store (see the AWS help topic for details);
2. Added cpu load onto the status and dashboard tabs;
3. Added an example log roll script that compresses log files on roll.  This can be modified to include transmit to S3, splunk, etc.;
4. Disable spaces in configuration names;
5. Remove duplication of query patterns in analytics due to temporary tables;
6. Resolved an issue with cluster detect logic, where the jdbc driver may not load properly;
7. Improve the no-cache reason when no cache rule is active;
8. Resolve issues with MySQL and results containing multiple results for a single query;
9. Resolve issue with Postgres COPY operation to allow whitespace before the copy;
10. Add vdb “rejectPrepated” parameter to refuse prepared statement executes to help track down where they are happening;
11. Add vdb “reparsePrepared” parameter to avoid query pattern buildup when prepares are not fully parameratized, with this, it could cause central manager memory buildup;
12. Added support for Jumpcloud LDAP authentication, including specific examples;
13. Added option for “Duplicate request tracking” to only ingest queries that have been observed twice in the maxTTL interval, to avoid ingesting content that is never accessed a second time;

Test Build 21.07.07.1 (July 7th, 2021)

Note:  AWS images are being regenerated due to errors in the version numbering published, but the base image has not changed since the prior build.  There is no need to update the AMI if they have been already updated.

1. Remove unnecessary log entry that was filling some logs;
2. Adjust timing of proxy start when the management server was starting to ensure the management server was fully initialized.

Test Build 21.07.05.1 (July 5th, 2021)

Note:  This test build is the base build for AWS images, but can be downgraded to the current “release” build as needed until the next release build is approved.  

1. Adjust the Java version from 8 to 11 again, and changes the default garbage collector to the Shenandoah collector, resulting in far more consistent behavior under heavy cache ingest (requires a new base image);
2. In the Amazon Linux builds, install the RDS certificates into /etc/ssl/certs/rds-combined-ca-bundle.pem (requires a new base image);
3. Limit the number of IO threads the Redis library Lettuce uses, to prevent certain thread saturation issues;
4. Set the Lettuce command timeout to 3s vs. the default 60s;
5. Adjust the cache logic so that the string () is not checked for if the unconditional flag is set.  Normally, this can trigger caching to stop due to possible indeterministic functions being used;
6. Add ports in the status tab to help identify overlaps;
7. Remove internal default limits for cache object count and ttl, to ensure more “expected” behavior.  In conjunction with the new garbage collector, this should help ensure optimal usage of memory resources;
8. Generate an alert if garbage collection takes more than 50% of one cpu core over a minute;
9. Improved the extraction of tables in some cases, in particular with sub-selects, where the string “(” had been detected as a table name;
10. Sorted disabled VDB’s on the status tab to the bottom;
11. Added a counter to JMX to allow tracking of grid ingest overflows, earlier it resulted in a log message, which could become very verbose;
12. Adjusted MySQL transmit buffer from 8k to 32k to better use larger TCP window sizes.

Test build 21.06.30.1 (June 30, 2021)

1.  Adjust the mysql defaultFetchSize to override the internal API call to set the same value, which is set to the Integer.min_size, needed to enable streaming of result-sets.  This default may trigger a slight performance degredation, so setting the value to 0 in the connection properties will override this, but revert to full response buffering.  The possible regression is being investigated.
2. Adjusted the MySQL proxy transmit buffer from 8k to 32k, to improve performance;
3. Include logging when the max cache object size is reached for MySQL in the nocache reason (pending for other proxies);
4. Hide the “drain” option in the status tab proxy menus, as it isn’t fully implemented for most use cases;
5. Detect when garbage collection is taking more than 15% of cpu time, and disable caching for the next minute;
6. Disable aggregating transmit buffers for non-cacheable results, which should improve memory behavior for non-cache use cases like pure read/write split;
7. Trigger a back-end connection drop when a MySQL change-user request is received, which should trigger any reset logic at the connection pool level and return the connection to the pool;
8. Display the advanced features enabled even if the advanced feature tab is closed on the vdb;
9. Add counter to JMX to track how often a result is dropped due to the cache ingest queue being full for the remote cache (CacheIngestQueueFullCount);
10. Adjust the self-balancing DNS mode to “band” load by 10% increments, and randomize query responses within those bands, to avoid fast-start benchmarks from triggering many clients connecting to the same proxy node;
11. Flag if a known bad version of java is used that can impact TLS;
12. Make the Postgres COPY command case insensitive when parsing for it;

Development build 21.06.24 (June 24, 2021)

1. Enable tcp keepalive on the client connections, set to “ping” every 300 seconds.  This is to avoid issues with NLB’s hard timeout of 350s, which results in connections not cleaning up properly when behind a load balancer;
2. Added DNS redirection capabilities to the Heimdall proxy, when in auto-scaling mode.  This allows delegation of a DNS name directly to the proxies themselves to allow load balancing based on load;
3. Added additional fields to the debug and queryinfo properties to allow tracking of overhead for cache requests, db execution time, and others;
4. Changed the default MySQL configuration to not use the maxResultBuffer option, and to default to using streaming result-sets.  This should resolve most issues with very large result-sets not processing properly;
5. Resolved an issue where the analytics database would grow without bound;

Development build 21.06.03 (June 3, 2021)

1. Migrate from using the MySQL 5.1.x branch to the 8.0.x branch of code to access the database;    This change has been reverted for now, due to compatibility issues found in testing
2. Update other JDBC drivers with minor changes;
3. For Postgres, we now support handling listen requests, optimizing the number of connections to the server needed.  If multiple clients request the same listen configuration, only one connection will be made for a proxy to the database, and all the connected clients will be serviced with the one connection;
4. For Postgres, we now have an option of “hdNotifyDBs” which provides a csv list of databases to listen to for invalidations.  Wildcards and exclusions are supported, and when a wildcard is used, the list is refreshed every five minutes;
5. With the MySQL proxy, support auto-redirection of clients to balance load on the proxy nodes, avoiding NLB and other overhead.  This requires special drivers (contact Heimdall support for more assistance on this);
6. For all drivers, auto-redirection can be done via DNS as well.  When auto-scaling mode is enabled, a DNS listener can be activated, which provides an ordered list of proxy IP addresses based on load, with the first being the best proxy to select.  
7. Configurable result-set size limits are now available in the cache settings, to limit the size of an object to ingest.
8. MySQL Proxy: Improved behavior to allow streaming of result-sets via the driver in more cases, avoiding issues with oversized results generating an exception.
9. With Azure MySQL, we now provide a MySQL JDBC driver (based on Oracle’s 8.0.x) that supports redirect for their “single server” offering.  The code patches for this will be submitted as open source, but not guaranteed to be accepted by Oracle;  this feature is still pending compatibility testing with the MySQL 8.0 driver codebase.
10. License handling has been redesigned, to allow most customers to be “licensed” via DNS configuration by Heimdall, avoiding the need for distinct licenses to be issued.  Improvements have also been made to detect cloud purchased offerings from AWS, Azure, and GCP;
11. Improved JMX statistics, including per-server per-second statistics equivalent to what is used to plot our own dashboard, and also allow ${ip} for the “jmx hostname” field to detect the local IP for JMX redirection.  With JMX, you connect to the host you specify in your tool, but the server then redirects you to another host or IP, which needs to match.  This allows configurations with dynamic IP’s to work cleanly, such as with docker;
12. Improved multiplexing and delayed transaction compatibility, including the ability to flag disabling multiplexing for X queries by rule;
13. Improved internal CI/CD pipeline to test patches and push them as our “development” branch;
14. Re-enable Geode support–for Gemfire use, contact Heimdall for a module compiled against Gemfire instead;
15. Adjust the memory graph on the dashboard to make it easier to observe with many nodes;
16. And… include all other fixes from prior builds, as well as many small documentation changes and updates.

Release build 21.04.05.1 (April 5, 2021)

1. Improved compatibility with delayed transactions;
2. Resolved issue where commands like commit and rollback would result in a sql log entry for the previous SQL command;
3. Added logic to trim prepared statements on a connection to no more than 2000 prepares to prevent memory overflow.  Ruby trims to 1000, and Java to 256, so this should be plenty for most environments;
4. Added logic to forget the last used parameters of a prepared statement after use, to reduce memory overhead;
5. Added support for hstore types in binary format for Postgres;
6. Added better support for commands like drop, alter, etc. that do not actually touch a table, but other metadata.  These will be flagged as modification queries, so that read/write split doesn’t send them to the wrong server;
7. Added to the sql server pool logic a default resetQuery of exec sp_reset_connection;
8. Improved SQL parsing for many different query formats to improve table extraction;
9. Avoid using the /tmp directory created by tomcat for temporary file uploads, as centos and derived systems clean up directories over time, and can break the update and other logic;
10. Added a filter on keys used by Heimdall in external grid caches to prevent non-heimdall keys from being tracked in the cache logic, adding to memory overhead;
11. Added a limit of 10x the number of objects specified in the cache object limit (if set) for tracking remote keys, again to limit memory usage;
12. Fix various issues with prepared SQL statements that include ? as part of the content, which impacted java prepared statement formats;
13. Improved temporary table detection logic, in particular for SQL Server;
14. Improvement in TLS handling, including disabling tls 1.3 with SQL Server, as it doesn’t support it yet, and trying to negotiate it with SQL server libraries often breaks them;
15. Improved accuracy of Prepared statement performance in the Analytics tab;
16. Prevent alerts from prior to a clean restart of the management server from showing back up after restart.

Release build 21.02.27.1 (Feb 27, 2021)

1. Critical: Fix a problem with MySQL when using prepared statements and read/write split (or anything that expands the prepared query to raw SQL) introduced in the last release build. This also impacted SQL logging in some cases as well;
2. Critical: Disable TLS 1.3 only for SQL Server–clients often will attempt to negotiate tls 1.3 if the proxy supports it, but they aren’t tested with tls 1.3, and end up breaking. SQL Server itself doesn’t support TLS 1.3; This would be observed as a connection hang on any attempt to connect. This was triggered by tls 1.3 being backported by Oracle into Java 8 in late 2020, which is used by default by Heimdall;
3. Change the downloaded log filename format for .zip files to include yyyyMMddHHmm, previously it was using minutes for the month position;
4. Add logging of full query hash in debug logs, along with logging some other error conditions for TLS, etc;
5. Resolved issue where a rule disabling multiplexing was not taking effect due to a rule processing optimization;
6. Resolved an issue with insert queries with more than 500k fields inserted (across all rows) due to limit in regex processing that limited the number of capture groups that could be created;
7. Removed a limit of 1024k for a single Postgres packet on query read, which limited the size of an insert that could be done as a single operation;
8. Added DML detection for generic create and drop commands for non-table oriented SQL. This prevents such queries from being directed to a read node with read/write split;
9. Ignore quotes and question marks in comments when looking for unbalanced quotes or parameters to fill in for SQL expansion;
10. Add further compatibility with the PHP PDO drivers for MySQL;
11. Correct an issue where rows read was not reported correctly for SQL Server at the protocol level;
12. Resolved issue where proxy auth being disabled for Postgres still resulted in Proxy auth being enabled in the resulting wizard configuration.

Release Build 21.02.13.1 (Feb 13, 2021)

1. Minor usability improvements in the wizard
2. Detect Azure marketplace deployments for licensing
3. Improved TLS handling and error logging
4. Support binary uploads with prepared queries for PDO MySQL (PHP)
5. Improved handling of temporary tables
6. Improve manual update process in low memory situations
7. Resolve issue with the proxy where it wouldn’t update the code on a restart (requires updating the heimdall-entrypoint.sh script)
8. Add example configuration for PHP’s MySQL PDO to extract the certificate from the keystore
9. Properly handle table names/aliases with question marks when expanding prepared queries for read/write split or transformation.

Release Build 21.01.24.1 (Jan 24, 2021)

1. Resolve issue with Data Direct MySQL drivers that prevented queries from completing execution;
2. Improved logic for Postgres failovers in the Azure cloud environment in the parsing of the replication connect string and to support the hostaddr option;
3. Corrected some cases where the defaultCatalog option was not being used properly to establish connections;
4. Resolved an infinite loop that could occur if using TLS and the client connection was disconnected unexpectedly;
5. Updated the default included driver database and driver names;
6. Remove the creation of new configuration files if the update server API was used;
7. Updated Hazelcast to 4.1.1 and include yaml configuration file support;
8. Resolve issue with internal commands like show pool with MS-SQL Server proxy configurations;
9. Added an option to allow recursive transform calls, to allow the transformation to be applied more than once on a single query;
10. Resolve a memory leak when connections were dropped by the client under very high new connection/second loads;
11. Improved logging (as normal).

Previous release notes can be found here.