MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2025-11-29 05:19:03 -05:00
Code Issues Packages Projects Releases Wiki Activity

Use concurrent bz2 decompression (#13360)

Browse Source 
Testing with `mc sql --compression BZIP2 --csv-input "rd=\n,fh=USE,fd=;" --query="select COUNT(*) from S3Object" local2/testbucket/nyc-taxi-data-10M.csv.bz2`

Before 96.98s, after 10.79s. Uses about 70% CPU while running.
This commit is contained in:
Klaus Post
2021-10-14 11:11:07 -07:00
committed by GitHub
parent 974073a2e5
commit 5e53f767c4
4 changed files with 31 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
go.mod
View File

@@ -16,6 +16,7 @@ require (
github.com/cheggaaa/pb v1.0.29
github.com/colinmarc/hdfs/v2 v2.2.0
github.com/coredns/coredns v1.4.0
github.com/cosnicolaou/pbzip2 v1.0.0
github.com/dchest/siphash v1.2.1
github.com/djherbis/atime v1.0.0
github.com/dswarbrick/smart v0.0.0-20190505152634-909a45200d6d