A New Data Access Mechanism for HDFS

doi:10.1088/1742-6596/898/6/062018

IHEP OpenIR

	A New Data Access Mechanism for HDFS
	Li Q(李强); Sun ZY(孙震宇); Wei ZC(魏占辰); Sun GX(孙功星); Li, Qiang ; Sun, Zhenyu ; Wei, Zhanchen ; Sun, Gongxing
	2017
发表期刊	Journal of Physics: Conference Series
ISSN	1742-6588
EISSN	1742-6596
卷号	898 期号:6 页码:062018
文章类型	Proceedings Paper
摘要	With the era of big data emerging, Hadoop has become the de facto standard of big data processing platform. However, it is still difficult to get legacy applications, such as High Energy Physics (HEP) applications, to run efficiently on Hadoop platform. There are two reasons which lead to the difficulties mentioned above: firstly, random access is not supported on Hadoop File System (HDFS), secondly, it is difficult to make legacy applications adopt to HDFS streaming data processing mode. In order to address the two issues, a new read and write mechanism of HDFS is proposed. With this mechanism, data access is done on the local file system instead of through HDFS streaming interfaces. To enable files modified by users, three attributes including permissions, owner and group are imposed on Block objects. Blocks stored on Datanodes have the same attributes as the file they are owned by. Users can modify blocks when the Map task running locally, and HDFS is responsible to update the rest replicas later after the block modification finished. To further improve the performance of Hadoop system, a complete localization task execution mechanism is implemented for I/O intensive jobs. Test results show that average CPU utilization is improved by 10% with the new task selection strategy, data read and write performances are improved by about 10% and 30% separately. © Published under licence by IOP Publishing Ltd.
会议名称	22nd International Conference on Computing in High Energy and Nuclear Physics, CHEP 2016
会议地点	San Francisco, CA, United states
会议日期	October 10, 2016 - October 14, 2016
DOI	10.1088/1742-6596/898/6/062018
收录类别	EI ; SCOPUS ; ADS
语种	英语
EI入藏号	20175104567109
EI主题词	Data handling - File organization - High energy physics
EI分类号	723.2 Data Processing and Image Processing - 903.3 Information Retrieval and Use - 932.1 High Energy Physics
ADS Bibcode	2017JPhCS.898f2018L
ADS URL	https://ui.adsabs.harvard.edu/abs/2017JPhCS.898f2018L
ADS引文	https://ui.adsabs.harvard.edu/abs/2017JPhCS.898f2018L/citations
INSPIRE ID	1638538
引用统计	正在获取... 被引频次：0 [INSPIRE] 被引频次：0 [ADS]
文献类型	期刊论文
条目标识符	https://ir.ihep.ac.cn/handle/311005/284221
专题	中国科学院高能物理研究所
作者单位	1.Institute of High Energy Physics, Beijing, China; 2.University of Chinese, Academy of Sciences, Beijing, China
推荐引用方式 GB/T 7714	Li Q,Sun ZY,Wei ZC,et al. A New Data Access Mechanism for HDFS[J]. Journal of Physics: Conference Series,2017,898(6):062018.
APA	李强.,孙震宇.,魏占辰.,孙功星.,Li, Qiang.,...&Sun, Gongxing.(2017).A New Data Access Mechanism for HDFS.Journal of Physics: Conference Series,898(6),062018.
MLA	李强,et al."A New Data Access Mechanism for HDFS".Journal of Physics: Conference Series 898.6(2017):062018.