The Gateway to Algorithmic and Automated Trading

Hazelcast IMDG integrates with Apache Spark

First Published 10th January 2017

Hazelcast in-memory data grid adds connector support for Apache Spark open source data processing engine.

Greg Luck, Hazelcast

Greg Luck, Hazelcast

Palo Alto, California - Hazelcast, the open source in-memory data grid (IMDG), has announced a new solution which integrates Hazelcast's IMDG and Apache Spark - an open source data processing engine.

By combining the two technologies, developers now have access to an open source solution that provides data storage and compute capabilities for big data requirements to address the historical limitations of a single Java Virtual Machine (JVM).

To demonstrate the potential of integrating Apache Spark into a Hazelcast IMDG application, BetLeopard, an example sports betting application, has been developed. BetLeopard is a bet engine that scales across multiple JVMs with the sharing of events via Hazelcast IMDG partitions, with a query engine that uses Spark to provide real-time risk and analytics of future events.

Hazelcast has clients for several programming languages including Java, .Net/C#, C++, Python, Node.js and Scala, while Spark supports Java, Scala, Python and R out of the box. Consequently, Hazelcast and Spark can be used across stacks that comprise multiple languages.

Greg Luck, CEO of Hazelcast, said: "The feedback we get back from the community is that any big data solution needs to be able to distribute processing and storage across machines whilst maintaining a flexible and convenient programming interface. Without these functionalities, it becomes impossible to build enterprise applications which are expected to process more and more data."