Building a Highly Efficient Search Engine for Logs In Rust | Francois, the Co-Founder of Quickwit

In today’s modern world, any systems/servers/applications produce an increasing tremendous amount of logs and you often have to choose between costly SaaS services or on-premise solutions that are hard to manage at a large scale.

That’s why we build Quickwit from the ground up to make it cloud-native. In this talk, we will see how we re-designed indexing and search to truly decouple storage and compute, and create stateless search instances while keeping a sub-second response time.

⚙️Thank you Rust London for organizing the event! Check out more of their events on their Youtube channel:
   / @rustlondon  

🔎 Quickwit Repo: https://github.com/quickwit-oss/quickwit

🏇 Tantivy Repo: https://github.com/quickwit-oss/tantivy

🚀Quickstart with Quickwit: https://quickwit.io/docs/get-started/...

_____

🕰️ TIMECODE :

0:00 Introduction
0:39 Speaker Bio
1:39 Overview of a Search Engine
3:35 Log Search Characteristics
6:44 Shared Nothing Architecture
13:35 Pros & Cons of Shared Nothing Architecture
15:15 Shared Disk Architecture
19:33 3 Problems to Solve
20:22 Problem 1: Staying truly stateless
22:31 Problem 2: Search TBs with a bad throughput
24:41 Problem 3: Search in a High-Latency World
27:30 Q&A Section
35:30 Outro
_____

⭐️ Psst...on GitHub there is a Star button...you know...only if you like our content : https://github.com/quickwit-oss/quickwit

🖤 Subscribe for more videos about software engineering & architecture :    / @quickwit8103  

💬 Join our community :

- Mind-blowing Discord :   / discord  
- Sky-breaking blogs : https://quickwit.io/blog
- Latest & greatest news on Twitter :   / quickwit_inc  
- Rust-mastery Twitch :   / quickwit_io  
- Cosmos-litting videos : https://www.youtube.com/channel/UCvZV...

Topics: indexing, search, data, query, architecture, distributed queues, messaging queues, metastore, metadata, indexer, scaling indexes, apache kafka, search engine internals, log management & analytics, shards

#softwareengineer #software #rust #developer #programming