If you have ever worked with large amounts of data, you are probably familiar with Kafka, an open source service that lets folks easily manage incoming streams of data. At it’s ongoing re:invent conference, Amazon today announced a brand new managed service that will let folks offload the heavy lifting to AWS, while easily utilizing all the benefits that Kafka offers.
Launched as the Amazon Managed Streaming for Kafka, the service is basically a Kafka implementation as a managed, highly available AWS.
Here is a short description of the service, as put forward by Amazon.
Apache Kafka clusters are challenging to setup, scale, and manage in production. When you run Apache Kafka on your own, you need to provision servers, configure Apache Kafka manually, replace servers when they fail, orchestrate server patches and upgrades, architect the cluster for high availability, ensure data is durably stored and secured, setup monitoring and alarms, and carefully plan scaling events to support load changes. Amazon Managed Streaming for Kafka makes it easy for you to build and run production applications on Apache Kafka without needing Apache Kafka infrastructure management expertise. That means you spend less time managing infrastructure and more time building application
Speaking about the issues that the service would resolve, at re:Invent, AWS CTO Werner Vogels said:
It’s a nightmare having to restart all the cluster and the main nodes. This is what I would call the traditional heavy lifting that AWS is really good at solving for you.
Interestingly, AWS already has a service called Kinesis, that helps folks manage incoming data streams. However, it seems as if Kafka’s popularity has prompted the company to launch an additional product, even at the risk of cannibalizing Kinesis.
Amazon Managed Streaming Service for Kafka is now available as a public preview for the populace. Pricing for a basic Kafka instance starts at around $0.21 per hour, but expect to pay additional charges for things like storage etc.
