Apache: Big Data Europe 2016
Click here to Register or for more information 
View analytic
Monday, November 14 • 12:00 - 12:50
Building a Scalable Recommendation Engine with Apache Spark, Apache Kafka and Elasticsearch - Nick Pentreath, IBM

Sign up or log in to save this to your schedule and see who's attending!

There are many resources available for using Apache Spark to build collaborative filtering models. However, there are relatively few for how to build a large-scale, end-to-end recommender system.

This talk will show how to create such a system, using Apache Kafka, Spark Streaming and Elasticsearch for data ingestion, real-time analytics and data storage, Spark DataFrames and ML pipelines for data aggregation and model building, and Elasticsearch for model management, serving and data visualization. We will also explore techniques for scaling model serving, using Spark Streaming for real-time model updates, and how to incorporate state-of-the-art models into this framework.

The talk will be technical and developer-focused, highlighting experiences from building real-world recommender systems, and providing example code (which will be available as open source).

avatar for Nick Pentreath

Nick Pentreath

Principal Engineer, IBM
Nick is a Principal Engineer at IBM, working primarily on machine learning on Apache Spark. He is a member of the Apache Spark PMC and author of Machine Learning with Spark. Previously, he co-founded Graphflow, a machine learning startup focused on recommendations. He has also worked... Read More →

Monday November 14, 2016 12:00 - 12:50
Giralda I/II

Attendees (62)