RE: [EXTERNAL]Re: KSQLdb Stream, Getting Topic Key

This worked for me, thank you for the help!

Andrew Greer

-----Original Message-----
From: Daniel Hinojosa <dhinojosa@evolutionnext.com>
Sent: Tuesday, August 10, 2021 2:33 PM
To: users@kafka.apache.org
Subject: [EXTERNAL]Re: KSQLdb Stream, Getting Topic Key

CAUTION:This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.



I had to verify myself, I wrote an example that uses a key that is not in the payload and I did where I specify the KEY and now it is a field. Notice the state down below

CREATE STREAM my_avro_orders (total BIGINT, shipping VARCHAR, state VARCHAR KEY, discount DOUBLE, gender VARCHAR) WITH (kafka_topic='my_avro_orders', value_format='AVRO');

This came out as the following with a describe:

ksql> describe my_avro_orders;

Name : MY_AVRO_ORDERS
Field | Type
-----------------------------------
TOTAL | BIGINT
SHIPPING | VARCHAR(STRING)
STATE | VARCHAR(STRING) (key)
DISCOUNT | DOUBLE
GENDER | VARCHAR(STRING)
-----------------------------------

Then querying it, I get:

ksql> select * from my_avro_orders emit changes;
+---------------------+---------------------+---------------------+---------------------+---------------------+
|STATE |TOTAL |SHIPPING |DISCOUNT
|GENDER |
+---------------------+---------------------+---------------------+---------------------+---------------------+
|ME |48025 |TWO_DAY
|0.019999999552965164 |FEMALE |
|MD |32813 |NEXT_DAY
|0.10999999940395355 |FEMALE |
|SC |99302 |TWO_DAY
|0.09000000357627869 |MALE |
|SD |35621 |TWO_DAY
|0.17000000178813934 |FEMALE |
|SD |5698 |NEXT_DAY
|0.12999999523162842 |MALE |
|VT |8567 |NEXT_DAY
|0.05000000074505806 |FEMALE |
|WA |13737 |NEXT_DAY
|0.07999999821186066 |MALE |
|NH |60216 |TWO_DAY
|0.05000000074505806 |MALE |
|UT |47470 |TWO_DAY
|0.05000000074505806 |MALE |
|IN |69220 |TWO_DAY
|0.1899999976158142 |FEMALE |


On Tue, Aug 10, 2021 at 12:11 PM Daniel Hinojosa < dhinojosa@evolutionnext.com> wrote:

> Ah, nevermind, just saw the title that this is KSQLDB.
>
> On Tue, Aug 10, 2021 at 12:09 PM Daniel Hinojosa <
> dhinojosa@evolutionnext.com> wrote:
>
>> The keys are already part of the stream. When you run builder.stream
>> or builder.table it returns a Stream<K,V> or a Table<K,V>. From there
>> every operation has a lambda that accepts both key and value. You can
>> use map for example to accept the key and do something with that. Let
>> me know if you have any other questions or if I didn't understand correctly.
>>
>> On Tue, Aug 10, 2021 at 10:58 AM Greer, Andrew C <
>> Andrew.C.Greer@conocophillips.com> wrote:
>>
>>> Hello, I am trying to create a Stream that will accept the data from
>>> my topic and be able to use the message keys in the stream as unique
>>> identifiers for the sensors the data originated from. The data in
>>> the messages does not have anything that would be able to identify
>>> which sensor it came from, hence trying to get the keys used in the
>>> topic. I have seen various posts online about a ROWKEY column, but
>>> that doesn't appear to be an option on my version at least. I have
>>> had no success finding a way to incorporate the keys into my Stream
>>> and was hoping I would be able to find some help here.
>>>
>>> Thank you.
>>>
>>> Kafka version 6.0.1
>>>
>>> Andrew
>>>
>>>