Real-Time Search and Recommendation at Scale Using Embeddings and Hopsworks

The dominant paradigm today for real-time personalized recommendations and personalized search is the retrieval and ranking architecture based on embeddings. It is a fan-out architecture where a single query produces a storm of requests on the backend. A single query will search through millions of items to retrieve hundreds of candidates that are then enriched by a feature store and ranked so only a few recommended items are presented to the user. A search should return in much less than 1 second. Retrieval and ranking architectures need significant infrastructure - an embeddings store and a feature store - to provide both the required scale and real-time performance.
In this talk, we will introduce an open-source, scalable retrieval and ranking serving architecture based on open-source technology: Hopsworks Feature Store, OpenSearch, and KServe. We will describe how to build and operate personalized search and recommendation systems using a retrieval model based on a two tower embedding model, and a ranking model gradient boosted trees. We will also show how you can train your embeddings and build your embeddings store index using Hopsworks and Apache Spark.

Attend this session to learn:

* how to to build a scalable, real-time retrieval and ranking recommender system using open-source platforms;
* how to train item/user embedding models and ranking models;
* how to put all these pieces together in an end-to-end solution for training and operating a scalable recommender/search engine.

Connect with us:
Website: https://databricks.com
Facebook:   / databricksinc  
Twitter:   / databricks  
LinkedIn:   / data.  .
Instagram:   / databricksinc