MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,455 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

18 Commits

Author SHA1 Message Date
Harshavardhana def04f01cf Update reedsolomon/highwayhash to start using ppc64le support (#7003) 
Thanks to @fwessels for the upstream work on reedsolomon and
highwayhash which has resulted in 10x performance improvement
on ppc64 architecture.
 2018-12-20 23:17:05 +05:30
Harshavardhana f162d7bd97 Performance improvements by re-using record buffer (#6622) 
Avoid unnecessary pointer reference allocations
when not needed, for example

- *SelectFuncs{}
- *Row{}
 2018-10-31 08:48:01 +05:30
Arjun Mishra 7c14cdb60e S3 Select API Support for CSV (#6127) 
Add support for trivial where clause cases
 2018-08-15 03:30:19 -07:00
Harshavardhana c0721164be Automatically set goroutines based on shardSize (#5346) 
Update reedsolomon library to enable feature to automatically
set number of go-routines based on the input shard size,
since shard size is sort of a constant in Minio for
objects > 10MiB (default blocksize)

klauspost reported around 15-20% improvement in performance
numbers on older systems such as AVX and SSE3

```
name                  old speed      new speed      delta
Encode10x2x10000-8    5.45GB/s ± 1%  6.22GB/s ± 1%  +14.20%    (p=0.000 n=9+9)
Encode100x20x10000-8  1.44GB/s ± 1%  1.64GB/s ± 1%  +13.77%  (p=0.000 n=10+10)
Encode17x3x1M-8       10.0GB/s ± 5%  12.0GB/s ± 1%  +19.88%  (p=0.000 n=10+10)
Encode10x4x16M-8      7.81GB/s ± 5%  8.56GB/s ± 5%   +9.58%   (p=0.000 n=10+9)
Encode5x2x1M-8        15.3GB/s ± 2%  19.6GB/s ± 2%  +28.57%   (p=0.000 n=9+10)
Encode10x2x1M-8       12.2GB/s ± 5%  15.0GB/s ± 5%  +22.45%  (p=0.000 n=10+10)
Encode10x4x1M-8       7.84GB/s ± 1%  9.03GB/s ± 1%  +15.19%    (p=0.000 n=9+9)
Encode50x20x1M-8      1.73GB/s ± 4%  2.09GB/s ± 4%  +20.59%   (p=0.000 n=10+9)
Encode17x3x16M-8      10.6GB/s ± 1%  11.7GB/s ± 4%  +10.12%   (p=0.000 n=8+10)
```
 2018-01-03 13:47:22 -08:00
Frank Wessels 6e6aeb6a9e Updated version of klauspost/reedsolomon using proper AVX2 instructions as well a providing support for Cauchy matrices. (#5215) 2017-11-24 14:23:20 -08:00
Andreas Auernhammer 02af37a394 optimize memory allocs during reconstruct (#4964) 
The reedsolomon library now avoids allocations during reconstruction.
This change exploits that to reduce memory allocs and GC preasure during
healing and reading.
 2017-09-27 10:29:42 -07:00
Frank Wessels 93f126364e Updated version of klauspost/reedsolomon with NEON support for ARM (#4865) 2017-08-30 09:49:00 -07:00
Andreas Auernhammer 85fcee1919 erasure: simplify XL backend operations (#4649) (#4758) 
This change provides new implementations of the XL backend operations:
 - create file
 - read   file
 - heal   file
Further this change adds table based tests for all three operations.

This affects also the bitrot algorithm integration. Algorithms are now
integrated in an idiomatic way (like crypto.Hash).
Fixes #4696
Fixes #4649
Fixes #4359
 2017-08-14 18:08:42 -07:00
Frank Wessels fffe4ac7e6 Prevent unnecessary verification of parity blocks while reading (#4683) 
* Prevent unnecessary verification of parity blocks while reading erasure
  coded file.
* Update klauspost/reedsolomon and just only reconstruct data blocks while
  reading (prevent unnecessary parity block reconstruction)
* Remove Verification of (all) reconstructed Data and Parity blocks since
  in our case we are protected by bit rot protection. And even if the
  verification would fail (essentially impossible) there is no way to
  definitively say whether the data is still correct or not, so this call
  make no sense for our use case.
 2017-08-11 18:25:46 -07:00
Harshavardhana faa6b1e925 vendorize deps for snappy, blake2b and sha256 (#3476) 
Bring in new optimization and portability changes.

Fixes https://github.com/minio/minio-go/issues/578
 2016-12-19 19:32:55 -08:00
Aditya Manthramurthy 8e6e9301ce Add support for Kafka as a notifications target (#2869) (#3439) 2016-12-15 08:23:48 -08:00
Harshavardhana ed676667d0 vendor: update reedsolomon package with new changes. (#2870) 
- Cached inverse matrices for better reconstruct performance.
- New error reconstruction required is returned, helpful in
  initiating healing.
 2016-10-06 21:57:42 -07:00
Harshavardhana 35d438e0ff vendorize: update all vendorized packages. (#2206) 
Bring in new changes from upstream for all the packages.

Important ones include
   - gorilla/mux
   - logrus
   - jwt
 2016-07-14 14:59:20 -07:00
Harshavardhana 7f38f46e20 vendor: update klauspost/reedsomon package with upstream changes. (#1912) 2016-06-17 15:16:26 +05:30
Harshavardhana ba7a55c321 xl: ReedSolomon code fix small file erasure bug. (#1431) 
For files less than 'dataBlocks', erasure encoding would fail
with short data due to a bug in the implementation itself.

Relax the error return, even a single byte can be properly
erasure coded without issues.

Fixes #1413
 2016-05-01 15:30:40 -07:00
Harshavardhana a9935f886c vendor: update reedsolomon package with new perm improvements. (#1422) 2016-04-29 14:59:03 -07:00
Harshavardhana a98a7fb1ad Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
Harshavardhana 9e10ee7e47 cpu: Remove pkg/cpu in favor of better klauspost/cpuid. 
Fixes #1128
 2016-02-15 13:50:33 -08:00