Commit Graph

6358 Commits

Author SHA1 Message Date
Harshavardhana
d7dc9aaf52
fix: remove response header timeout (#8919)
Adding respone header timeout seems to have
premature timeout like consequences which
leads to potential disconnections.
2020-02-01 08:31:55 +05:30
Harshavardhana
bfe8a9bccc
jwt: Simplify JWT parsing (#8802)
JWT parsing is simplified by using a custom claim
data structure such as MapClaims{}, also writes
a custom Unmarshaller for faster unmarshalling.

- Avoid as much reflections as possible
- Provide the right types for functions as much
  as possible
- Avoid strings.Join, strings.Split to reduce
  allocations, rely on indexes directly.
2020-01-31 08:29:22 +05:30
Klaus Post
9990464cd5
Fix recursive deep scan of buckets (#8900) 2020-01-30 17:20:07 +05:30
poornas
881e983ed9
Fix Retention, ObjectLock, LegalHold struct namespaces correctly. (#8909)
Reverts #8903 to allow structs to be unmarshalled 
even if the namespace is missing.
2020-01-30 09:58:05 +05:30
Nesty92
f6a7d4d29b
Add CreateBucketError, ListMultipartUploads functional test (#8886) 2020-01-29 07:10:03 -08:00
Harshavardhana
f98616dce7
heal: Optimize heal listing by avoiding batches (#8901)
Also limit the heal per object if there is incoming
requests by suspending heal for longer periods of time.
2020-01-29 12:05:44 +05:30
Ashish Kumar Sinha
5bd0e95eef
Set default namespace for necessary structs (#8903) 2020-01-29 10:19:38 +05:30
Li Yi
0414f01b6e doc: match code on sysctl -w vm.swappiness=1 (#8894) 2020-01-27 16:19:15 -08:00
Harshavardhana
0cbebf0f57 Rename pkg/{tagging,lifecycle} to pkg/bucket sub-directory (#8892)
Rename to allow for more such features to come in a more
proper hierarchical manner.
2020-01-27 14:12:34 -08:00
Harshavardhana
4cb6ebcfa2 test: print more relevant info in healing failure (#8895) 2020-01-27 14:56:36 +05:30
poornas
2232e095d5 Make admin permissions more granular for admin handlers. (#8888) 2020-01-26 20:47:52 -06:00
Minio Trusted
aae337f5b5 Update yaml files to latest version RELEASE.2020-01-25T02-50-51Z 2020-01-25 02:57:23 +00:00
poornas
a78e5d4763 Add missing error check in cache GetObjectNInfo (#8889) 2020-01-24 15:49:16 -08:00
Harshavardhana
cf37c7997e Heal bucket only on missing drives in quorum (#8883)
MakeVol shouldn't be called in heal bucket
when bucket doesn't really exist in quorum.
2020-01-24 15:38:07 -08:00
Harshavardhana
1ffbb5c24c fix racy tests when editing xl.getDisks (#8879) 2020-01-23 11:50:09 -08:00
Harshavardhana
b9c48e0ab0 fix return appropriate error for MakeBucket in federation (#8878) 2020-01-22 08:25:28 -08:00
Harshavardhana
fe5d599802 fix: STS creds without "aud" should be honored with STS checks (#8868)
Fixes #8865
2020-01-22 15:09:46 +05:30
Aditya Manthramurthy
55063906b5 Fix group add/remove membership bug (#8877) 2020-01-21 19:00:41 -08:00
Klaus Post
c7178d2066 Profiling: Add base, fix memory profiling (#8850)
For 'snapshot' type profiles, record a 'before' profile that can be used 
as `go tool pprof -base=before ...` to compare before and after.

"Before" profiles are included in the zipped package.

[`runtime.MemProfileRate`](https://golang.org/pkg/runtime/#pkg-variables) 
should not be updated while the application is running, so we set it at startup.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-01-21 15:49:25 -08:00
Harshavardhana
f14f60a487 fix: Avoid double usage calculation on every restart (#8856)
On every restart of the server, usage was being
calculated which is not useful instead wait for
sufficient time to start the crawling routine.

This PR also avoids lots of double allocations
through strings, optimizes usage of string builders
and also avoids crawling through symbolic links.

Fixes #8844
2020-01-21 14:07:49 -08:00
Harshavardhana
e2b3c083aa
fix: close and drain the response body always (#8847) 2020-01-21 02:46:58 -08:00
Anis Elleuch
3011692d93 doc: Add missing ``` in compression documentation (#8872) 2020-01-21 15:50:54 +05:30
Harshavardhana
86252ec7e1
fix: document _ENABLE for all notification targets (#8864)
Fixes #8863
2020-01-20 16:48:19 -08:00
Harshavardhana
ef1aa870c5 cleanup unneeded files, update credits (#8858)
additionally add code of conduct
2020-01-20 10:38:58 -08:00
bijialin
c260182412 zh_CN docs: add Expanding existing distributed setup (#8862) 2020-01-20 08:51:09 -08:00
Nitish Tiwari
61c17c8933 Add ObjectTagging Support (#8754)
This PR adds support for AWS S3 ObjectTagging API as explained here
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
2020-01-20 08:45:59 -08:00
Forest Lovewood
dd93eee1e3 Implement bucket caching for b2 gateway (#8820)
fixes #8739 #6806
2020-01-20 22:13:38 +05:30
Harshavardhana
7a400542ae Remove usage of GORPOXY let 'go' decide the defaults (#8855) 2020-01-19 09:17:42 -08:00
Harshavardhana
88286cf8d0 fix: support pre-sign signature for STS tokens (#8826)
Fixes #8391
2020-01-18 17:04:50 -08:00
Klaus Post
8cb6184f1d Fix erasure block allocation (#8851)
Small blocks are undersized when file size isn't divisible by the 
shard could leading to allocation in *reedsolomon.Split()*
2020-01-18 14:21:58 -08:00
Zhongpeng Lin
b849fd7a75 remove gorilla/rpc replacement (#8852) 2020-01-18 14:21:13 -08:00
Harshavardhana
09ee145e9c gw/hdfs: indicate hdfs gateway is production ready (#8848) 2020-01-18 07:25:03 -08:00
Harshavardhana
04dfcbfdd7
update minio-go to latest master (#8849) 2020-01-18 04:05:17 -08:00
Harshavardhana
23e46f9dba
log formatting only the first time (#8846) 2020-01-17 15:39:07 -08:00
Harshavardhana
fc5213258e
posix: Do not take disk offline on I/O errors (#8836)
Choosing maxAllowedIOError is arbitrary and
prone to errors, when drives might be perfectly
capable of taking I/O with only few locations
return I/O error. This is a hindrance of sort
where backend filesystems like ZFS can automatically
fix and handle these scenarios.

The added problem with current approach that we
take the drive offline, making it virtually impossible
to bring it online without restart the server which
is not desirable on a busy cluster. Remove this state
such that let the backend return error appropriately
to caller and let the caller decide what to do with
the error.
2020-01-17 13:34:43 -08:00
kannappanr
005ebbb9b2 Typo: Add missing '<' character (#8845)
Fixes #8833
2020-01-17 13:31:46 -08:00
Anis Elleuch
017067e11f data-usage: Avoid crawling duplicated call (#8843)
This fix will also picks 3 and not 4 disks from a single erasure set.
2020-01-17 09:59:37 -08:00
Harshavardhana
2bb69033e5 http: fail appropriately and return standard Go error (#8837)
return http.ErrServerClosed with proper body when
server is shutting down, allowing more context instead
of just returning '503' which doesn't mean the same
thing.
2020-01-17 05:48:39 -08:00
Harshavardhana
fca4ee84c9
gw/hdfs: listing should list directories properly (#8827)
Fixes #8822
2020-01-16 17:11:25 -08:00
poornas
60e60f68dd Add support for object locking with legal hold. (#8634) 2020-01-16 15:41:56 -08:00
Minio Trusted
ba758361b3 Update yaml files to latest version RELEASE.2020-01-16T22-40-29Z 2020-01-16 22:47:41 +00:00
Harshavardhana
c6b218e5df
fix: readiness should return 200 OK with first zone online (#8834) 2020-01-16 13:49:25 -08:00
Anis Elleuch
c18fbdb29a posix: Remove a non needed nil check in DiskInfo() (#8830)
posix.DiskInfo() returns errFaultyDisk when posix is nil,
but there is no way that this would happen any time, therefore
removing un-needed code.
2020-01-16 11:27:50 -08:00
Harshavardhana
b1ad99edbf
fix: avoid crash copy map before reading (#8825)
code of this form is always racy, when the
map itself is being written to as well

```
func (r Map) retMap() map[string]string {
     .. lock ..
     return r.internalMap
}

func (r Map) addMap(k, v string) {
     .. lock ..
     r.internalMap[k] = v
}
```

Anyone reading from `retMap()` is not protected
because of locking and we need to make sure
to avoid code in this manner. Always safe to
copy the map and return.
2020-01-16 01:35:30 -08:00
Minio Trusted
080e0c2323 Update yaml files to latest version RELEASE.2020-01-16T03-05-44Z 2020-01-16 03:11:40 +00:00
Anis Elleuch
935546d5ca xl: Implement MRF healing (#8470) 2020-01-15 18:30:32 -08:00
Harshavardhana
64fde1ab95
xl/zones: return errNoHealRequired when no heal is required (#8821)
Zone abstraction of object layer was returning `nil`
incorrectly under situations where disk healing is
not required. Returning `nil` is considered as healing
successful, which leads to unexpected ReloadFormat()
peer notification calls during startup.

This PR fixes this behavior properly for zones.
2020-01-15 17:19:13 -08:00
Anis Elleuch
169e8742fc mint: Use https for access to maven reposity (#8824)
Maven repository requires HTTPS now. This lead to issues
building mint image in aws-sdk-java & minio-java.

The PR fixes the issue and also bump aws sdk version in
aws-sdk-java to the latest.
2020-01-15 15:48:49 -08:00
Anis Elleuch
069876e262 xl: All nodes create meta volumes in its local disks (#8786)
Meta volumes directories, tmp/, background-ops/, etc..
undr .minio.sys are created when disks are formatted
but also when the cluster is started.

However using MakeVolBulk() is not appropriate in the
case of a user migrating from a version which does not
have .minio.sys/background-ops/. The reason is that
MakeVolBulk() exits early when an error is occured:
errVolumeExists in this case, which is expected since
some directories such as tmp/ already exist.

This commit will avoid use MakeVolBulk and use MakeVol
instead.

Also the PR will make each node creates meta volumes
in its local disks and stop relying on the first disk
since the first node could be offline.
2020-01-15 12:36:52 -08:00
Harshavardhana
442e1698cb
heal: Avoid spinning up object healing during startup (#8819)
auto-heal disks, metadata and buckets in background but
not objects, let the auto heal kick in for objects after
the cluster has been up for a while.
2020-01-15 01:08:39 -08:00