Apache: Big Data Europe 2016
Click here to Register or for more information 
Back To Schedule
Monday, November 14 • 12:00 - 12:50
Building a Scalable Recommendation Engine with Apache Spark, Apache Kafka and Elasticsearch - Nick Pentreath, IBM

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

There are many resources available for using Apache Spark to build collaborative filtering models. However, there are relatively few for how to build a large-scale, end-to-end recommender system.

This talk will show how to create such a system, using Apache Kafka, Spark Streaming and Elasticsearch for data ingestion, real-time analytics and data storage, Spark DataFrames and ML pipelines for data aggregation and model building, and Elasticsearch for model management, serving and data visualization. We will also explore techniques for scaling model serving, using Spark Streaming for real-time model updates, and how to incorporate state-of-the-art models into this framework.

The talk will be technical and developer-focused, highlighting experiences from building real-world recommender systems, and providing example code (which will be available as open source).

avatar for Nick Pentreath

Nick Pentreath

Principal Engineer, IBM
Nick Pentreath is a principal engineer in IBM's Center for Open Source Data & AI Technologies (CODAIT), where he works on machine learning. Previously, he cofounded Graphflow, a machine learning startup focused on recommendations. He has also worked at Goldman Sachs, Cognitive Match... Read More →

Monday November 14, 2016 12:00 - 12:50 CET
Giralda I/II