Kafka Connect in Action: Elasticsearch

This is a walkthrough of how to stream data from #ApacheKafka to #Elasticsearch using #KafkaConnect and the Elasticsearch sink connector. You can try the whole tutorial for yourself here: https://rmoff.dev/kafka-elasticsearch

ℹ️Contents:
00:00:00 Introduction
00:07:17 Updating documents in Elasticsearch
00:11:38 Deleting documents in Elasticsearch with the sink connector
00:13:25 Funny when live demos don't work
00:16:14 Schemas and Elasticsearch Document Mappings
00:25:57 Sending data to Elasticsearch without a declared schema
00:28:57 Getting Timestamps from Kafka to Elasticsearch
00:33:14 Using dynamic field mapping to guess at date field types
00:36:21 Using Single Message Transform to set timestamp schema type explicitly
00:38:49 Dynamic Templates for data type mapping from Kafka in Elasticsearch
00:47:20 Changing the target index name in Elasticsearch
00:48:01 Using RegEx to change the target index name
00:49:35 Time-based partitioning of indices in Elasticsearch
00:51:57 Combining Single Message Transform in Kafka Connect
00:53:00 Error handling
00:57:13 Configure Kafka Connect to not fail on deserialisation errors
00:59:31 Setting up a Dead Letter Queue
01:02:23 Connector error handling outside of the Kafka Connect framework

Try it out for yourself: https://rmoff.dev/kafka-elasticsearch

--

☁️ Confluent Cloud ☁️
Confluent Cloud is a managed Apache Kafka and Confluent Platform service. It scales to zero and lets you get started with Apache Kafka at the click of a mouse. You can signup at https://confluent.cloud/signup?utm_so... and use code 60DEVADV for $60 towards your bill (small print: https://www.confluent.io/confluent-cl...)