Commit Graph

33 Commits

Author SHA1 Message Date
Harshavardhana f503ac3db8 XL/Erasure: Make bit-rot verification based on xl.json algo. (#2299)
Currently `xl.json` saves algorithm information for bit-rot
verification. Since the bit-rot algo's can change in the
future make sure the erasureReadFile doesn't default to
a particular algo. Instead use the checkSumInfo.
2016-07-28 02:20:34 -07:00
Krishna Srinivas 043ddbd834 optimize memory allocation during erasure-read by using temporary buffer pool. (#2259)
* XL/erasure-read: optimize memory allocation during erasure-read by using temporary buffer pool.

With the change the buffer needed during GetObject by erasureReadFile is allocated only once.
2016-07-25 14:17:01 -07:00
Krishna Srinivas 18728a0b59 XL/erasure-read: refactor erasure read and add tests (#2232) 2016-07-20 01:30:30 -07:00
Krishna Srinivas 897d78d113 erasureReadFile and erasureCreateFile testcases. (#2229)
* unit-tests: Unit tests for erasureCreateFile and erasureReadFile.

* appendFile() should return errXLWriteQuorum.

* TestErasureReadFileOffsetLength() tests erasureReadFile() for different offset and lengths.

* Fix for the failure seen in the erasure read unit test case. Issue #2227

* Move common erasure setup code to newErasureTestSetup()

* Review fixes. Add few more test cases for erasureReadFile.
2016-07-18 23:56:16 -07:00
Krishna Srinivas 8cc163e51a Refactor xl.GetObject and erasureReadFile. (#2211)
* XL: Refactor xl.GetObject and erasureReadFile. erasureReadFile() responsible for just erasure coding, it takes ordered disks and checkSum slice.

* move getOrderedPartsMetadata and getOrderedDisks to xl-v1-utils.go

* Review fixes.
2016-07-16 08:35:30 -07:00
Krishna Srinivas b090c7112e Refactor of xl.PutObjectPart and erasureCreateFile. (#2193)
* XL: Refactor of xl.PutObjectPart and erasureCreateFile.

* GetCheckSum and AddCheckSum methods for xlMetaV1

* Simple unit test case for erasureCreateFile()
2016-07-14 14:59:01 -07:00
Harshavardhana ddf3245677 xl/fs: offset and length cannot be negative. (#2121)
Fixes #2119
2016-07-07 01:30:34 -07:00
Krishna Srinivas 5291db60c6 XL/erasure: refactor erasureReadFile. Move parallelRead into a separate function. (#2008) 2016-06-27 13:24:55 -07:00
karthic rao ce7d5eddbc Misspell warnings fix (#2001) 2016-06-26 22:05:48 -07:00
Harshavardhana 9add048f3c erasure: Add erasure encode/decode unit tests. (#1911)
Fixes #1819
2016-06-26 03:32:49 -07:00
Krishna Srinivas cb1200a66d XL/erasure-read: read disks in parallel. (#1975)
On read failure, fallback to reads from other
disks also happen in parallel.
2016-06-24 18:02:10 -07:00
Harshavardhana e8990e42c2 XL: Make allocations simpler avoid redundant allocs. (#1961)
- Reduce 10MiB buffers for loopy calls to use 128KiB.
- start using 128KiB buffer where needed.
2016-06-24 02:06:23 -07:00
Harshavardhana 9b82e64a11 XL/erasure-read: Avoid memory copy, write to writer directly all the dataBlocks. 2016-06-23 02:06:57 +05:30
Krishna Srinivas d4bea5fbf8 XL/erasure-read: Add Comments and enable bitrot detection. 2016-06-23 00:34:47 +05:30
Krishna Srinivas 17efaaa902 XL/erasure-read: Support parallel reads from disks. 2016-06-23 00:34:47 +05:30
Bala FA 7d757033f2 erasure-readfile: Use chunk size to read from each disk for a block. (#1949)
A block of data is split into data chunk and each data chunk is
written to each disk.  Previously block size was used to read data
chunk which returns corrupted data.

This patch fixes the issue by reading chunk sized data from each disk
and assembles a block.

Fixes #1939
2016-06-20 21:40:10 -07:00
Bala FA 1ea1dba528 erasure-readfile: write to given Writer than returning buffer. (#1910)
Fixes #1889
2016-06-19 13:35:26 -07:00
Harshavardhana c6ac3fa6db erasure: Fix block index matching.
This patch fixes an important issue of block reconstruction upon block corruption.
2016-06-05 06:19:58 -07:00
Harshavardhana fb95c1fad3 XL: Bring in some modularity into format verification and healing. (#1832) 2016-06-02 16:34:15 -07:00
Harshavardhana de21126f7e XL: Re-align the code again. 2016-06-02 01:54:06 -07:00
Harshavardhana ae311aa53b XL: Cleanup, comments and all the updated functions. (#1830) 2016-06-01 16:43:31 -07:00
Harshavardhana c493ab5d0d XL: Bring in sha512 checksum support. (#1797) 2016-05-31 20:23:31 -07:00
Bala FA db2fdbf38d erasure: allocate buffer only for non-nil disk (#1811) 2016-05-31 11:55:50 -07:00
Krishnan Parthasarathi 967c2b2940 Handled possible short writes to httpResponseWriter (#1804)
* XL: Handled possible short writes to httpResponseWriter

* Added tests for Range Header combinations
2016-05-30 11:27:15 -07:00
Harshavardhana a4a0ea605b XL: Fix GetObject erasure decode issues. (#1793) 2016-05-29 15:38:14 -07:00
Harshavardhana feb337098d XL: bring in new storage API. (#1780)
Fixes #1771
2016-05-28 16:12:51 -07:00
Harshavardhana 27cc8a6529 erasure: read only dataBlocks if we have enough. (#1776)
Reconstruct with parity blocks if we don't have enough data blocks.
2016-05-28 15:15:53 -07:00
Harshavardhana d65101a8c8 XL: Implement strided erasure distribution. (#1772)
Strided erasure distribution uses a new randomized
block distribution for each Put operation. This
information is captured inside `xl.json` for subsequent
Get operations.
2016-05-28 15:15:53 -07:00
Harshavardhana 553fdb9211 XL: Bring in support for object versions written during writeQuorum. (#1762)
Erasure is initialized as needed depending on the quorum and onlineDisks.
This way we can manage the quorum at the object layer.
2016-05-28 15:15:53 -07:00
Harshavardhana a97230dd56 XL/erasure: Reset dataBlocks to reduce the memory usage. (#1749)
Fixes #1748
2016-05-28 15:15:53 -07:00
Harshavardhana 1e393c6c5b XL: Add new metadata for checksum. (#1743) 2016-05-28 15:15:53 -07:00
Krishna Srinivas b38b9fea79 XL/erasure: fix for skipping 0 padding. (#1737)
Fixes #1736
2016-05-28 15:15:53 -07:00
Harshavardhana 293d246f95 XL/FS: Rewrite in new format. 2016-05-28 15:15:53 -07:00