Apache: Big Data Europe 2016
Click here to Register or for more information 
Back To Schedule
Wednesday, November 16 • 12:00 - 12:50
Smart Storage Management: Towards Higher HDFS Storage Efficiency - Wei Zhou, Intel

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

All kinds of data volume increases dramatically in recent years, new storage devices (NVMe SSD, flash SSD, etc.) can be utilized to improve data access performance. HDFS provides methodologies like HDFS Cache, Heterogeneous Storage Management (HSM) and Erasure Coding (EC) to provide such support, but it remains a big challenge to define and adjust different storage strategies for different data in a dynamic environment.

To overcome the challenge and improve the storage efficiency of HDFS, we will introduce a comprehensive solution, aka Smart Storage Management (SSM) in Apache Hadoop. HDFS operation data and system state information are collected from the cluster, based on the metrics collected SSM can extract some äóìdata access patternsäó and based on these patterns SSM will automatically make sophisticated usage of these methodologies to optimize HDFS storage efficiency.


Wei Zhou

Software engineer in Intel. Currently mainly focus on Apache Hadoop performance optimization. Co-speaker on HBase Developer Course in Strata+Hadoop world Beijing 2016.

Wednesday November 16, 2016 12:00 - 12:50 CET
Giralda V