The Heimdall Proxy Community Edition supports the following features:
- Connection Pooling with Multiplexing
- Authentication and Authorization through LDAP Group/Role synchronization
- SQL Performance analytics
- Database metadata filtering: Improves performance for user-interface tools (e.g. DBeaver)
- Licensed free of charge for Greenplum deployments
Query Results Caching
Choose the look-aside cache (e.g. Redis, Hazelcast, GemFire), and the proxy provides the caching and invalidation logic. For business applications like Tableau, customers have experienced improved response times over 20x. No application changes are required. View our video below.
Connection Pooling
- Connection Pooling: Multiple client connections are associated to a backend connection. Unlike PgBouncer, we limit connections per user and per database.
- Connection Multiplexing: Establishes a backend connection for only active, non-idle queries. The net result is, 1) lower total memory, 2) more active queries processed, and 3) reduced database costs.
- Authentication / Authorization for LDAP: Automated Group / Role synchronization. Avoid having to preconfigure users on the database.
Batch Processing
- 1. Improved application response times due to fewer commits
- 2. Improved DML scale
Our fast failover is a must-have for any Enterprise desiring the database be always-on. Upon a failure, Heimdall queues up the front end connection and transparently fails over to the backend standby; our proxy persists the application connection. This removes application errors and exceptions upon a failover. Check out our video.
Hybrid Transactional Analytical Processing (HTAP)
As explained in our technical blog, companies choose Greenplum for its large-scale MPP (Massively Parallel Processing). However, there are some drawbacks:
- Higher latency: Many nodes must coordinate to generate an answer;
- No support for materialized views
- Frequent queries slowing down processing
The ideal solution to these drawbacks requires no code changes such as the Heimdall Proxy. Heimdall uses a separate Postgres database to maximize Greenplum performance. One benefit is materialized view support.
Fast materialized views is very important in analytics environments. When reports are generated, or dashboards are viewed, a subset of data is pulled from the back-end data store and processed. Heimdall provides the following functionality:
- Queries against a materialized view are routed to an alternate database (e.g. Postgres) acting on behalf of Greenplum. Postgres answers queries offloading Greenplum.
- Flexibly configured triggers provide an auto-refresh of the view. Heimdall is aware of updated views from Greenplum and when data was loaded that may impact the view. The net result is faster reports and a lighter load on Greenplum.
Visit our technical blog.
Heimdall Proxy for Greenplum allows customers to deploy applications for both OLTP and OLAP workloads, often called HTAP (Hybrid Transactional/Analytic Processing).
Why Heimdall Data for Greenplum?
The Heimdall Proxy improves Greenplum performance up to 100x. Our proxy is deployed between the application and data source to improve query response times (e.g. query caching, connection pooling, batching singleton DML, materialized view offload).
We offer a free version, the Heimdall Proxy Community Edition that supports 1) Connection Pooling, 2) Authentication/Authorization for Active Directory, and 3) Metadata filtering for user-interface tools (e.g. DBeaver) resulting in response times. Download here.
How is Heimdall Data deployed?
1) VM, Docker Container or Standalone instance between the application and database
2) Sidecar process, an agent, or JDBC driver installed on each application instance to maximize performance and reduce latency:
Deployment requires no application or database changes. Just change the connection string or networking setting of the application to route through the Heimdall Database Proxy. SQL performance visibility and optimization is managed by a Heimdall Data central console.
What are other Heimdall for Greenplum Use Cases?
HTAP (Hybrid Transactional / Analytical Processing):
Problem: As a result of queries being issued by a client, it is necessary to perform (trigger) actions that perform data synchronization and/or updates to refresh data. This can be used in conjunction with query routing to aggregate or batch data to load into a large-scale data warehouse for example. In this case, the inserts may be made against a front-end fast with a small database such as Postgres, then after a period of time during which no more updates occur, the data is synchronized as a whole into the large-scale data warehouse.
HTAP Solution: Heimdall is implementing a generic system whereby actions (SQL calls or external programs) can be triggered while processing a query, and the trigger can be executed either before the matching SQL, after, or in parallel. When in “parallel” this means in another thread. Further, the parallel execution can be delayed, with repeated calls being aggregated into a single call.
Summary: The Heimdall Proxy optimizes Greenplum performance for both query reads and writes.
Batching DML Operations: Inserting data row-by-row, with aggregation for a bulk insert into the data warehouse when there is a lull in insert activity. This allows the back-end to have low overhead, even while large amounts of individual row data is being inserted, potentially by many clients. Use-case: When using IoT devices, which are providing large amounts of data into a data warehouse, a front-end database can be used as a buffer, with a periodic flush of the data into the warehouse, without additional logic being needed at the application layer.
Automatic Materialized View Management:
Heimdall serves as a traffic manager:
- Queries against a materialized view are routed to an alternate database (e.g. Postgres) acting on behalf of Greenplum. Postgres answers queries, offloading Greenplum. Also, frequent queries of small data sets are best deployed using a general-purpose Postgres DB than Greenplum. Heimdall routes traffic to the appropriate data source for the best performance.
- Heimdall triggers a refresh of the view automatically. Heimdall is aware of updated views from Greenplum and when data was loaded that may impact the view. The net result is faster reports and a lighter load on Greenplum.
How does Heimdall help Greenplum with frequent singleton transactions?
Heimdall transparently batches singleton transactions. Details of the solution are found here.
How does query caching work?
Our proxy uses a grid-cache as a look-aside results cache. We provide the caching and invalidation logic to intelligently offload SQL traffic from Greenplum and cache into the grid-cache of your choice (Redis, Amazon ElastiCache, Pivotal GemFire, Hazelcast, GridGain).
For this use case, Heimdall is transparently storing SQL results into cache and serving out results from the corresponding query.
- Heimdall treats GemFire as a Key/Value pair: Key = query and Value = Result set
- Heimdall is NOT writing SQL into GemFire nor is GemFire used as a Read-through or Write-through cache.
- No code changes are required for automatic cache invalidation.
How does Heimdall's Automated Failover help Greenplum?
What concurrency levels can you expect with the Heimdall solution?
What transactional databases does Heimdall support?
Heimdall supports any SQL database (Postgres, MySQL, SQL Server, Oracle etc.)
How is Heimdall priced?
Heimdall is priced by Greenplum CPU Core. For specific Heimdall Data for Pivotal pricing, please email sales (at) heimdalldata.com