Comments on: Multi-Cluster Deployment Options for Apache Kafka: Pros and Cons https://www.altoros.com/blog/multi-cluster-deployment-options-for-apache-kafka-pros-and-cons/ Insight Thu, 18 Jun 2020 17:29:02 +0000 hourly 1 https://wordpress.org/?v=6.0.6 By: Medhavi Chetan https://www.altoros.com/blog/multi-cluster-deployment-options-for-apache-kafka-pros-and-cons/#comment-801 Mon, 13 May 2019 10:01:33 +0000 https://www.altoros.com/blog/?p=31657#comment-801 Hi, If I have a source which can connect to only single kafka cluster, but I want to produce data to two different clusters without using mirroring type of option (cluster to cluster data copy). is there any thing in the market which can be used to read data from source and write to two kafka clusters together

]]>
By: Andrei Bushyk https://www.altoros.com/blog/multi-cluster-deployment-options-for-apache-kafka-pros-and-cons/#comment-548 Sat, 19 May 2018 09:59:58 +0000 https://www.altoros.com/blog/?p=31657#comment-548 In reply to Rajiv Onat.

Hi Rajiv, that is a very good question.
Within a given cluster, Kafka consumers keep track of their offsets in a topic called “__consumer_offsets”. The obvious idea of replicating this topic will not work, because of the different situations, which may cause the offsets to not point to the same messages between the clusters.
Instead, message creation timestamp should be used to find consumer offset before a failure.
Different KIPs (Kafka Improvement Proposal) have been implemented to provide this feature:
KIP-32 – Add timestamps to Kafka message,
KIP-33 – Add a time based log index,
KIP-79 – ListOffsetRequest/ListOffsetResponse v1 and add timestamp search methods to the new consumer,
KIP-122: Add Reset Consumer Group Offsets tooling.
Thus, consumer offset can be reset to the offset, derived from a timestamp before a failure, either from a client application, using the Kafka consumer API, or outside of the application, using Kafka command line tool.

]]>
By: Andrei Bushyk https://www.altoros.com/blog/multi-cluster-deployment-options-for-apache-kafka-pros-and-cons/#comment-726 Sat, 19 May 2018 09:59:58 +0000 https://www.altoros.com/blog/?p=31657#comment-726 In reply to Rajiv Onat.

Hi Rajiv, that is a very good question.
Within a given cluster, Kafka consumers keep track of their offsets in a topic called “__consumer_offsets”. The obvious idea of replicating this topic will not work, because of the different situations, which may cause the offsets to not point to the same messages between the clusters.
Instead, message creation timestamp should be used to find consumer offset before a failure.
Different KIPs (Kafka Improvement Proposal) have been implemented to provide this feature:
KIP-32 – Add timestamps to Kafka message,
KIP-33 – Add a time based log index,
KIP-79 – ListOffsetRequest/ListOffsetResponse v1 and add timestamp search methods to the new consumer,
KIP-122: Add Reset Consumer Group Offsets tooling.
Thus, consumer offset can be reset to the offset, derived from a timestamp before a failure, either from a client application, using the Kafka consumer API, or outside of the application, using Kafka command line tool.

]]>
By: Rajiv Onat https://www.altoros.com/blog/multi-cluster-deployment-options-for-apache-kafka-pros-and-cons/#comment-547 Fri, 11 May 2018 06:06:37 +0000 https://www.altoros.com/blog/?p=31657#comment-547 How did you manage consumer offset syncs between active-active?

]]>
By: Rajiv Onat https://www.altoros.com/blog/multi-cluster-deployment-options-for-apache-kafka-pros-and-cons/#comment-725 Fri, 11 May 2018 06:06:37 +0000 https://www.altoros.com/blog/?p=31657#comment-725 How did you manage consumer offset syncs between active-active?

]]>