How we built a DISTRIBUTED search engine in Rust

In a data-driven world, having to search through terabytes of logs is common. This simple task can turn out to be daunting and very expensive.
This talk will present how to implement such an engine with an efficient architecture strongly inspired by Snowflake which separated compute and storage.

🦀 Thank you RustLab for organizing the event! Check out more of their events on their Youtube channel:
‪@rustlabconference3671‬

🔎 Quickwit Repo: https://github.com/quickwit-oss/quickwit

🏇 Tantivy Repo: https://github.com/quickwit-oss/tantivy

🚀Quickstart with Quickwit: https://quickwit.io/docs/get-started/...
_____

🕰️ TIMECODE :

0:00 Introduction & agenda
2:45 Search architecture
3:50 Tantivy indexing & querying
5:54 Data structures
10:30 Incremental indexing
12:53 Shared-nothing architecture
17:15 Importance of log search
20:25 Shared-disk architecture
29:09 Problem 1: Staying truly stateless
31:58 Problem 2: Searching with a bad throughput
33:07 Problem 3: Searching in a high latency world
33:59 Demo time
42:50 What we learnt after 2 years
45:40 Choosing an actor framework
51:51 Questions
_____
⭐️ Psst...on GitHub there is a Star button...you know...only if you like our content : https://github.com/quickwit-oss/quickwit

🖤 Subscribe for more videos about software engineering & architecture : https://www.youtube.com/channel/UCvZV...

💬 Join our community :

- Mind-blowing Discord :   / discord  
- Sky-breaking blogs : https://quickwit.io/blog
- Latest & greatest news on Twitter :   / quickwit_inc  
- Cosmos-litting videos : ‪@quickwit8103‬

Topics: indexing, search, data, query, architecture, distributed queues, messaging queues, metastore, metadata, indexer, scaling indexes, apache kafka, search engine internals, log management & analytics, shards

#softwareengineer #software #rust