Apache Big Data Europe 2016

Apache Spark is the hottest computing engine nowadays.

More people in big data industry start to use Spark in production systems,

such as machine learning and data ETL applications.

However, developers take quality of code seriously in production systems.

In the past experience, we find there is a gap between development and production.

The difficulties we meet when developing Spark applications in production systems are:

(1) hard to communicate with components,

(2) indirect management of application arguments,

(3) inadequate code maintainability.

To solve these problems and make development smoother, we propose

a dependency-injection-based programming framework on JVM systems.

It provides basic management, monitoring, and better communication mechanisms.

The huge flexibility can help developers writing Spark applications and integrating with components in a better manner.