With the growth of Kafka adoption in all major streaming initiatives across large organizations, the operational and visibility challenges associated with Kafka are on the rise as well. Kafka users want better visibility in understanding what is going on in the cluster across the four key entities in Kafka: producers, topics, brokers, and consumers.
Different personas in the organization have different challenges with Kafka. For a platform operations team, it is the lack of visibility at a cluster and broker level and the effects of the broker on the infrastructure it runs on and vice versa. While for a DevOps/App team, they want to know how data flows between the four entities and understand the key performance metrics (KPMs) of these entities. For the governance and security teams, the questions revolve around chain of custody, audit, metadata, access control and lineage. SMM caters to all these diverse needs across different personas.
Hortonworks Streams Messaging Manager (SMM) is an operations monitoring / management tool that provides end-to-end visibility in an enterprise Apache Kafka® environment. With SMM, you can gain clear insights about your Kafka clusters. You can understand the end-to-end flow of message streams from producers to topics to consumers. SMM helps you troubleshoot your Kafka environment to identify bottlenecks, throughputs, consumer patterns, traffic flow etc. SMM enables you to analyse the stream dynamics between producers and consumers using various filters. You can optimize your Kafka environment based on the key performance insights gathered from various brokers and topics. With the tight integration of Apache Atlas, you can gain complete data lineage across multiple Kafka hops, producers and consumers with powerful data flow visualization.