7445 Commits

Author SHA1 Message Date
Anis Elleuch
47e54fb823 Send bucket name to peers when bucket notification is enabled (#11351) (#13035)
Co-authored-by: massintha azamoum <40169489+massintha-riaktr@users.noreply.github.com>
2022-01-03 01:16:20 -08:00
Harshavardhana
ec42715219 configure audit queue_size 2022-01-03 01:15:21 -08:00
Anis Elleuch
7e8767c1c1 s3: Put bucket tagging to return an error when bucket is not found (#13489)
Co-authored-by: Harshavardhana <harsha@minio.io>
2022-01-03 01:15:21 -08:00
sinhaashish
4267706e7c Update upload_part error message 2022-01-03 01:15:21 -08:00
Harshavardhana
ba1989fe84 do not panic if DNS_WEBHOOK_ENDPOINT is not reachable 2022-01-03 01:15:21 -08:00
Anis Elleuch
ba0fc46d7a Update dependencies to fix some security warnings (#12976)
* Increase alpine version to fix security issue with apk-tools

* Upgrade jwt dependency to fix a security issue

* Update miekg/dns dependency to the latest one

* Upgrade some librairies to get rid of 'github.com/gogo/protobuf@v1.3.1'

For that, upgrade the following dependencies:

  github.com/nats-io/nats-server/v2
  github.com/nats-io/nats.go
  github.com/nats-io/stan.go
2022-01-03 01:15:21 -08:00
Harshavardhana
23f11cec58 move from 503 -> 499 for disconnected clients 2022-01-03 01:15:21 -08:00
Anis Elleuch
5aff310b80
Always deduplicate folders in listing (#12909)
When a prefix with a trailing slash is found, always deduplicate it.

Co-authored-by: Anis Elleuch <anis@min.io>
2021-08-09 06:58:34 -07:00
Anis Elleuch
851f3c5f0c Listing: Do not include marker in CommonPrefixes (#12021) (#12897)
Walk() and Merge code can return dir__XLDIR__ as the last element 
on a page list and dir__XLDIR__ as the first element in the next list page.

dir__XLDIR__ is shown in the second-page list because the marker set to
dir/ is meant to skip dir/ and not dir__XLDIR__

To fix this, the code will avoid adding the marker itself to the listing
result.
2021-08-06 13:11:31 -07:00
Anis Elleuch
5df7bbf9f9 [10-28.sets branch] Use refresh locking mechanism in locks (#12862)
* locking: Add Refresh for better locking cleanup
* locking: Add timeout in unlock calls
2021-08-06 13:11:29 -07:00
Harshavardhana
a6e8f4aa1c Revert "Node should clear a lock internally that the lock owner don't recognize (#12782)"
This reverts commit 83676a0bc2ad9ff6fd4d2a52701ed20d1a10c169.
2021-08-02 00:18:30 -07:00
Harshavardhana
0a276a25cf fix: extend parentDirIsObject for all parents 2021-07-30 09:20:40 -07:00
Harshavardhana
83ed1f361b fix: make sure parentDirIsObject is used at set level (#11280)
parentDirIsObject is not using set level understanding
to check for parent objects, without this it can lead to
objects that can actually reside on a separate set as
objects and would conflict.
2021-07-29 09:02:52 -07:00
Anis Elleuch
83676a0bc2 Node should clear a lock internally that the lock owner don't recognize (#12782)
If a lock owner says the lock is expired, all nodes should remove the
lock internally immediately.
2021-07-26 11:44:26 -07:00
Harshavardhana
b1c731c448 Revert "remove healObjects() from scanner"
This reverts commit 5258a68b456330e32d08290e21679cfc0ade5068.
2021-07-22 16:23:41 -07:00
Harshavardhana
38709c84b7 optimize multipart upload
cherry-pick 33cee9f38a3e662ad68973ab48f595f0435d423f from master
branch for improving multipart upload and lock handling
2021-07-14 23:47:17 -07:00
Harshavardhana
5258a68b45 remove healObjects() from scanner 2021-07-08 09:36:41 -07:00
Harshavardhana
963f3ee047 fix: crash in healObjects() 2021-07-08 02:55:47 -07:00
Anis Elleuch
89db553204
Support rolling update for storage API (#12634) 2021-07-07 08:18:01 -07:00
Harshavardhana
1c2f82938f change healObjects to heal one set at time 2021-07-06 12:46:01 -07:00
Harshavardhana
51ad1d983d allow for rolling upgrades 2021-07-06 12:19:57 -07:00
Anis Elleuch
ed264449b1 Add force unlock support (#12274)
Add admin API to force unlock some stale locks

Signed-off-by: Anis Elleuch <anis@min.io>
2021-06-09 22:16:06 -07:00
Harshavardhana
0e1dce37ad add throttling delays for HealObjects() based on crawler delay 2021-05-21 13:20:58 -07:00
Andreas Auernhammer
472d78604b s3v4: read and verify S3 signature v4 chunks separately
This commit fixes a security issue in the signature v4 chunked
reader. Before, the reader returned unverified data to the caller
and would only verify the chunk signature once it has encountered
the end of the chunk payload.

Now, the chunk reader reads the entire chunk into an in-memory buffer,
verifies the signature and then returns data to the caller.

In general, this is a common security problem. We verifying data
streams, the verifier MUST NOT return data to the upper layers / its
callers as long as it has not verified the current data chunk / data
segment:
```
func (r *Reader) Read(buffer []byte) {
   if err := r.readNext(r.internalBuffer); err != nil {
      return err
   }
   if err := r.verify(r.internalBuffer); err != nil {
      return err
   }
   copy(buffer, r.internalBuffer)
}
```
2021-05-17 21:33:37 -07:00
Harshavardhana
d04201e2a6 fix: missing user policy enforcement in PostPolicyHandler (#11682) 2021-05-17 21:30:54 -07:00
Harshavardhana
d629ca0a47 fix: reject duplicate keys in PostPolicyJSON document (#11902)
fixes #11894
2021-05-17 21:28:59 -07:00
Harshavardhana
65a5e2c6d3 fix: merge duplicate keys in post policy (#11843)
some SDKs might incorrectly send duplicate
entries for keys such as "conditions", Go
stdlib unmarshal for JSON does not support
duplicate keys - instead skips the first
duplicate and only preserves the last entry.

This can lead to issues where a policy JSON
while being valid might not properly apply
the required conditions, allowing situations
where POST policy JSON would end up allowing
uploads to unauthorized buckets and paths.

This PR fixes this properly.
2021-05-17 21:26:48 -07:00
Harshavardhana
65864bc76a fix: for deleteBucket delete on dnsStore first
attempt a delete on remote DNS store first before
attempting locally, because removing at DNS store
is cheaper than deleting locally, in case of
errors locally we can cheaply recreate the
bucket on dnsStore instead of.
2021-05-14 11:27:50 -07:00
Harshavardhana
4e6e8c47b5 add root_disk threshold detection (#12259)
as there is no automatic way to detect if there
is a root disk mounted on / or /var for the container
environments due to how the root disk information
is masked inside overlay root inside container.

this PR brings an environment variable to set
root disk size threshold manually to detect the
root disks in such situations.
2021-05-09 23:06:14 -07:00
Harshavardhana
ab6f0c0831 handle dns.ErrBucketConflict as BucketAlreadyExists 2021-04-07 22:04:03 -07:00
Harshavardhana
ef1ea96044 move erasure blockSize to 1MiB 2021-03-18 21:53:56 -07:00
Harshavardhana
29e7058ebf background delete operations and delete serially every 10mins
addtionally introduce MINIO_DELETE_CLEANUP_INTERVAL environment
value to control this interval, choose a lesser value if higher
speed is necessary. Supports time.Duration format

expory MINIO_DELETE_CLEANUP_INTERVAL=1m

Would let MinIO run delete cleanup interval every minute, once
the previous cycle completes.
2021-03-09 16:49:39 -08:00
Harshavardhana
f864931ab4 delete dangling objects automatically 2021-03-07 00:08:30 -08:00
Harshavardhana
96b1377863 add additional logs 2021-03-07 00:04:11 -08:00
Harshavardhana
bff2f9c733 set http2 for KES communication 2021-02-18 21:43:26 -08:00
Klaus Post
5f41f6043d Avoid synchronizing usage writes (#11560)
If the periodic `case <-t.C:` save gets held up for a long time it will end up
synchronize all disk writes for saving the caches.

We add jitter to per set writes so they don't sync up and don't hold a
lock for the write, since it isn't needed anyway.

If an outage prevents writes for a long while we also add individual
waits for each disk in case there was a queue.

Furthermore limit the number of buffers kept to 2GiB, since this could get
huge in large clusters. This will not act as a hard limit but should be enough
for normal operation.
2021-02-18 21:34:18 -08:00
Ritesh H Shukla
21718705b8
turn off http2 for TLS setups for now (#11523) (#11569)
due to lots of issues with x/net/http2, as
well as the bundled h2_bundle.go in the go
runtime should be avoided for now.

https://github.com/golang/go/issues/23559
https://github.com/golang/go/issues/42534
https://github.com/golang/go/issues/43989
https://github.com/golang/go/issues/33425
https://github.com/golang/go/issues/29246

With collection of such issues present, it
make sense to remove HTTP2 support for now
2021-02-17 19:06:26 -08:00
Harshavardhana
53e0c16976 add bucket name to the log 2021-02-08 23:00:48 -08:00
Harshavardhana
fb78283c0a add GOMAXPROCS back 2021-02-08 22:29:10 -08:00
Harshavardhana
f07c9c58e7 fix: handle setIndexes properly 2021-02-08 22:25:06 -08:00
Harshavardhana
bc89e47066 remove GOMAXPROCS requirement 2021-02-08 21:54:00 -08:00
Harshavardhana
0615d85384 heal sets with optional prefix input 2021-02-05 11:15:42 -08:00
Harshavardhana
42157eb218 listing also match sets index for proper quorum 2021-02-01 22:48:08 -08:00
Harshavardhana
fa1cd6dcce heal multiple buckets in parallel 2021-02-01 22:45:34 -08:00
Harshavardhana
745a4b31ba add support for concurrent heals 2021-01-29 21:59:49 -08:00
Harshavardhana
5151c429e4 fix: add api level throttler for LIST calls 2021-01-28 22:59:15 -08:00
Klaus Post
dc1a46e5d2 crawler: Stream bucket usage cache data (#11068)
Stream bucket caches to storage and through RPC calls.
2021-01-25 21:27:28 -08:00
Harshavardhana
8724d49116 implement Heal sets API to heal erasure sets independently 2021-01-24 19:05:56 -08:00
Anis Elleuch
006c69f716 make: Add hotfix target to generate hotfix binaries (#11053)
hotfix target will fetch the release tag prior to the latest commit and create a binary
with the same release tag plus '.hotfix' suffix

e.g.   RELEASE.2020-12-03T05-49-24Z.hotfix
2021-01-22 11:01:46 -08:00
Harshavardhana
28974fb5da fix: release locks if the client timedout (#11030)
situations where client indeed timedout there was
a potential to falsely think that lock is still
active.
2020-12-05 00:01:56 -08:00