[DISCUSS] Custom Metadata on Topics — Inconsistent Behavior Between Broker and Topic Config in Kafka 4.2.0
Hello Kafka Community, We have a use case that requires attaching custom metadata to topics — things like SLA, durability guarantees, and PagerDuty account references. While exploring this in Kafka 4.2.0, we noticed an interesting inconsistency between broker-level and topic-level configuration behavior. Broker-level custom configs work fine: Running the following command succeeds without errors: ./bin/kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type brokers --entity-name 1 \ --alter --add-config 'durability.level=1' And describing the broker returns the config as expected: ./bin/kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type brokers --entity-name 1 \ --describe Output: Dynamic configs for broker 1 are: durability.level=null sensitive=true synonyms={DYNAMIC_BROKER_CONFIG:durability.level=null} Topic-level custom configs throw an exception: Running a similar command on a topic: ./bin/kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type topics --entity-name test-topic-metadata \ --alter --add-config 'durability.level=3' ...results in an InvalidConfigurationException. Our question: Is this asymmetry intentional? Is there a design reason why unknown/custom configurations are permitted at the broker level but rejected at the topic level? And is there a recommended approach for attaching arbitrary custom metadata to topics in Kafka? Any guidance or pointers to relevant KIPs would be greatly appreciated! Thanks in advance [image: :pray:]
Comments
Post a Comment