Commit Graph

3316 Commits

Author SHA1 Message Date
Harshavardhana
1e3d80552f XL: format.json healing should cater for mismatching order. (#2285)
Fresh disks can be provided in any order, we need to make sure
to preserve existing disk order and populate the fresh disks
in new positions.

Thanks for Anis Elleuch <vadmeste@gmail.com> for finding this issue.
2016-07-26 03:18:47 -07:00
Harshavardhana
f253dfc922 docs: Fix erasure code image embedding issue. 2016-07-26 00:09:57 -07:00
Harshavardhana
1f9e38e3cd api: Add bucket notification util tests. (#2289) 2016-07-26 00:01:35 -07:00
karthic rao
530ed67b59 Adding leak test framework (#2254) 2016-07-25 20:39:14 -07:00
Harshavardhana
a2b6f0524d XL/erasure: Remove deprecated copyN function. (#2288) 2016-07-25 20:36:56 -07:00
karthic rao
091d80666a Enhancement for Erasure encode test. (#2287) 2016-07-25 20:36:41 -07:00
Harshavardhana
efbf7dbc0f api: Bucket notification add filter rules check and validate. (#2272)
These filtering techniques are used to validate
object names for their prefix and suffix.
2016-07-25 17:53:55 -07:00
Krishna Srinivas
043ddbd834 optimize memory allocation during erasure-read by using temporary buffer pool. (#2259)
* XL/erasure-read: optimize memory allocation during erasure-read by using temporary buffer pool.

With the change the buffer needed during GetObject by erasureReadFile is allocated only once.
2016-07-25 14:17:01 -07:00
Dee Koder
04f90bd463 doc: Broken links fixed in the Explore further section. (#2281) 2016-07-24 22:53:35 -07:00
Harshavardhana
9212e11b90 XL/GetObject: When disk is not available, checksum should be empty. (#2276) 2016-07-24 22:49:27 -07:00
Harshavardhana
79bab6b561 XL: Operations on uploads.json should cater for disk being unavailable. (#2277) 2016-07-24 18:08:15 -07:00
Krishnan Parthasarathi
7e5a78985d tests: Using listObjects clean up remaining tree walk go routines. (#2278)
* fs: Set nextMarker independent of it having a slash or not.
* tests: Using listObjects clean up remaining tree walk go routines.
* tests: Use slices to hold data instead of enumerating test cases by hand

... also fixed numbering of test cases.
2016-07-24 15:52:12 -07:00
Anis Elleuch
b0b919a1d6 Server http and https on the same port using a customized server (#2247) 2016-07-24 12:30:57 -07:00
Harshavardhana
6c2fb19ed7 docs: Removed and purged uneeded docs. (#2273) 2016-07-24 03:32:45 -07:00
Harshavardhana
f248089523 api: Implement bucket notification. (#2271)
* Implement basic S3 notifications through queues

Supports multiple queues and three basic queue types:

1. NilQueue -- messages don't get sent anywhere
2. LogQueue -- messages get logged
3. AmqpQueue -- messages are sent to an AMQP queue

* api: Implement bucket notification.

Supports two different queue types

- AMQP
- ElasticSearch.

* Add support for redis
2016-07-23 22:51:12 -07:00
Harshavardhana
f85d94288d api: extract http headers with some supported header list. (#2268) 2016-07-22 20:31:45 -07:00
Harshavardhana
55cb55675c api/multipart: Send S3 compatible error message, missing second sentence. (#2270) 2016-07-22 17:05:40 -07:00
koolhead17
7e076577de Update Minio-erasure-code-quickStart-guide.md (#2269)
Minor update to link the URL.
2016-07-22 13:35:27 -07:00
Harshavardhana
5d118141cd XL: Remove deadcode unionChecksumInfo. (#2261) 2016-07-21 19:07:00 -07:00
karthic rao
646ff2c64d Get Object disk not found test (#2264)
Test: GetObject disk not found test
2016-07-21 19:06:50 -07:00
Harshavardhana
0add96f655 fs: Save metadata for objects in minioMetaBucket directory. (#2251) 2016-07-21 17:31:14 -07:00
Krishna Srinivas
303f216150 tests: xl-v1-metadata.go, xl-v1-multipart-common.go - remove unused methods, add enhance tests to improve code coverage. (#2260) 2016-07-21 15:00:11 -07:00
koolhead17
a7b5b8e63f Doc: Modified the contents for Doctor. (#2262) 2016-07-21 14:58:16 -07:00
Krishnan Parthasarathi
5730d40478 tests: Added GetObject, DeleteObject and PutObject unit-tests (#2222) 2016-07-21 13:15:54 -07:00
karthic rao
0eaf684777 Remove consuming benchmarks, clean up closures, correct Get and PutObject Parallel benchmarks (#2258) 2016-07-21 11:17:28 -07:00
Harshavardhana
a0635dcdd9 XL: Do not rely on getLoadBalancedQuorumDisks for NS consistency. (#2243)
The reason is any function relying on `getLoadBalancedQuorumDisks`
cannot possibly have an idempotent behavior.

The problem comes from given a set of N disks returning just a
shuffled N/2 disks.  In case of a scenario where we have N/2
number of failed disks, the returned value of `getLoadBalancedQuorumDisks`
is not equal to the same failed disks so essentially calls using such
disks might succeed or fail randomly at different intervals in time.

This proposal change is we move to `getLoadBalancedDisks()`
and use the shuffled N disks as a whole. Since most of the time we might
hit a good disk since we are not reducing our solution space. This
also provides consistent behavior for all the functions which rely
on shuffled disks.

Fixes #2242
2016-07-21 00:27:08 -07:00
Dee Koder
41f4f2806d screenshots: update with the latest optimized image. (#2249) 2016-07-20 16:15:26 -07:00
Dee Koder
2a972ef1fd images: Move screenshot for docs inside docs/screenshots directory. (#2248)
* images: Move screenshot for docs inside docs/screenshots directory. Use optimized images.

* images: This fix optimizes the images for the Erasure Code Quick Start Guide
2016-07-20 13:52:30 -07:00
Harshavardhana
c1e953b368 api: Set content-encoding properly if set. (#2245)
Additionally don't set content-type if not present, golang http
server automaticaly handles this and sets it automatically.
2016-07-20 12:40:20 -07:00
Krishna Srinivas
18728a0b59 XL/erasure-read: refactor erasure read and add tests (#2232) 2016-07-20 01:30:30 -07:00
Harshavardhana
cef26fd6ea XL: Refactor usage of reduceErrs and consistent behavior. (#2240)
This refactor is also needed in lieu of our quorum
requirement change for the newly understood logic behind
klauspost/reedsolom implementation.
2016-07-19 19:24:32 -07:00
Dee Koder
f67c930731 doc: Fixed spacing with respect to code blocks. (#2241) 2016-07-19 19:08:43 -07:00
GarimaKapoor
3589a58179 Update Minio-erasure-code-quickStart-guide.md 2016-07-19 17:59:05 -07:00
Dee Koder
e8155abc18 screenshot: Use the full path to the screenshot when embedding images (#2239) 2016-07-19 17:48:18 -07:00
Dee Koder
2e8360120d headings: We need to add a consistent heading for all docs. Adding Minio FreeBSD QuickStart Guide in the title. (#2233) 2016-07-19 14:56:34 -07:00
Dee Koder
02b191222c headings: Added standardized heading for this document. (#2234) 2016-07-19 14:56:20 -07:00
Dee Koder
b699795901 docs: Remove additional headings. Added standard heading. Include numbering. (#2235) 2016-07-19 14:30:32 -07:00
Harshavardhana
86d31e99d5 api: use checkAuth now at PutBucket, DeleteBucket handlers. (#2225)
Additionally add a unit test for isReqAuthenticated function.
2016-07-18 23:56:27 -07:00
Krishna Srinivas
897d78d113 erasureReadFile and erasureCreateFile testcases. (#2229)
* unit-tests: Unit tests for erasureCreateFile and erasureReadFile.

* appendFile() should return errXLWriteQuorum.

* TestErasureReadFileOffsetLength() tests erasureReadFile() for different offset and lengths.

* Fix for the failure seen in the erasure read unit test case. Issue #2227

* Move common erasure setup code to newErasureTestSetup()

* Review fixes. Add few more test cases for erasureReadFile.
2016-07-18 23:56:16 -07:00
Harshavardhana
1f706e067d api: xmlDecoder should honor contentLength. (#2226)
This is needed so that we avoid reading large amounts
of data from compromised clients.
2016-07-18 21:20:17 -07:00
Krishnan Parthasarathi
5cc9e4e214 fs/XL: Return IncompleteBody{} error for short writes (#2228) 2016-07-18 19:06:48 -07:00
Krishna Srinivas
27a5b61f40 tree-walk: optimize tree walk such that leaf detection of entries is delayed till the entry is sent on the treeWalkResult channel. (#2220) 2016-07-17 15:16:52 -07:00
Harshavardhana
aeac902747 API: ListBuckets doesn't have a body, we should never read the body. (#2218)
ListBuckets was incorrectly reading the body of the request, fix it.
2016-07-17 13:23:15 -07:00
Harshavardhana
aaf7803831 api: Requests should be differentiated if possible based on http router. (#2219)
In current master ListObjectsV2 was merged into ListObjectsHandler
which also implements V1 API as well.

Move the detection of ListObject types to its rightful place
in http router.
2016-07-17 12:32:05 -07:00
Krishna Srinivas
8cc163e51a Refactor xl.GetObject and erasureReadFile. (#2211)
* XL: Refactor xl.GetObject and erasureReadFile. erasureReadFile() responsible for just erasure coding, it takes ordered disks and checkSum slice.

* move getOrderedPartsMetadata and getOrderedDisks to xl-v1-utils.go

* Review fixes.
2016-07-16 08:35:30 -07:00
Harshavardhana
2d38046a5a utils: BucketNames with double periods and ip address should be rejected. (#2213)
Fixes #2212
2016-07-15 17:30:37 -07:00
Harshavardhana
cbb6b48b94 doc: update README.md 2016-07-15 16:12:54 -07:00
Harshavardhana
d0636d633d doc: Move FreeBSD.md to docs. 2016-07-15 16:09:01 -07:00
Harshavardhana
41187fc2ef docker: Fix docker edge build 2016-07-15 15:10:38 -07:00
koolhead17
204ec2c6c0 doc:README.md/Updated to sync with docs.minio.io (#2210)
* doc:README.md/Updated to sync with docs.minio.io

* doc:README.me/Modified the minio server output terminal to reflect new release changes.

* docs:README.md/Modified and changed location of other markdown files.
2016-07-15 15:03:59 -07:00