Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spark 1.6 下parquet vs orc #12

Open
cjuexuan opened this issue Mar 20, 2016 · 0 comments
Open

spark 1.6 下parquet vs orc #12

cjuexuan opened this issue Mar 20, 2016 · 0 comments
Labels

Comments

@cjuexuan
Copy link
Member

背景

这都是现在大数据下比较火热的两种存储格式,orc和hive的关系可能要密切一点,但spark 对parquet寄予了厚望,
最近我们在测一个有join场景下的多个dataset的读取情况,这里简单写一下测试的一个结果,测试环境是spark1.6,数据在hdfs上,spark是local模式,集群的测试下周进行

join 后group,最后count

user的数据量在4000w左右,字段为4个字段,play选择的是1月1号,记录是1084966

先对比存储:
存储中横向对比了orc格式和parquet格式:

datatype\format orc parquet
playinfo 7.7M 7.6M
userinfo 97M 87M

此时存储可以看出两者差别不大,parquet略小于orc,不过在海量数据下这个差距会被放大,此时的字段毕竟只是4个字段的user和3个字段的play

再对比性能:
执行类似:

user.joinWith(play,"$"userUid"===$"playUid","inner").groupBy($"_1.thirdPartyName").count().show()

执行结果:

left joinType right duration/ms foramt
user inner play 44213 orc
user outer play 82484 orc
play inner user 43229 orc
play outer user 81112 orc
play inner user 23288 parquet
play outer user 62150 parquet
user inner play 22549 parquet
user outer play 64828 parquet

解释下这张表,left表示在左边的,因为数据量是不完全对称的,而join方式也测试了inner和outer两种jointype,duration是指这个job执行的时间,而format是两种数据格式

不过网络也很重要,在测试时候我的下载大概是3M/s

join 后groupBy 最后avg

第二步我测试了

user.joinWith(play,"$"userUid"===$"playUid","inner").groupBy($"_1.thirdPartyName").toDf.agg(sum("_2.durationsum")).show()

测试结果:

playDate format duartion
2016/01/01 parquet 27967
2016/01 parquet 93854
2016/01/01 orc 49551
2016/01 orc 116043

初步看1.6用spark作为执行引擎使用parquet格式性能有一定的提升,后期会集群再测一次,而且我观察到两种格式的shuffle size好像相差挺大,这一个也是后期要看的一个指标

@cjuexuan cjuexuan added the spark label Mar 20, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant