Commit Graph

7639 Commits

Author SHA1 Message Date
Klaus Post
e7d3b49a20
metacache: Make very small requests transient (#11109) 2020-12-15 11:25:36 -08:00
Harshavardhana
5df61ab96b
fix: remove gorilla/rpc/ deps fully after our fork (#11108) 2020-12-15 11:18:06 -08:00
Poorna Krishnamoorthy
3456b03b12
Ignore ObjectNotFound errors in delete api while enforcing locking (#11114)
AWS does not report this or version not found as errors in the response.
2020-12-15 11:15:49 -08:00
Klaus Post
f6fb27e8f0
Don't copy interesting ids, clean up logging (#11102)
When searching the caches don't copy the ids, instead inline the loop.

```
Benchmark_bucketMetacache_findCache-32    	   19200	     63490 ns/op	    8303 B/op	       5 allocs/op
Benchmark_bucketMetacache_findCache-32    	   20338	     58609 ns/op	     111 B/op	       4 allocs/op
```

Add a reasonable, but still the simplistic benchmark.

Bonus - make nicer zero alloc logging
2020-12-14 13:13:33 -08:00
Harshavardhana
8368ab76aa
fix: remove the requirement for healing buckets in ListBucketsHeal (#11098)
With new refactor of bucket healing, healing bucket happens
automatically including its metadata, there is no need to
redundant heal buckets also in ListBucketsHeal remove
it.
2020-12-14 12:07:07 -08:00
Harshavardhana
3e83643320
lifecycle improvements and additional debug logging (#11096)
Bonus change fix browser assets
2020-12-13 12:05:54 -08:00
Harshavardhana
2eb52ca5f4
fix: heal bucket metadata right before healing bucket (#11097)
optimization mainly to avoid listing the entire
`.minio.sys/buckets/.minio.sys` directory, this
can get really huge and comes in the way of startup
routines, contents inside `.minio.sys/buckets/.minio.sys`
are rather transient and not necessary to be healed.
2020-12-13 11:57:08 -08:00
Harshavardhana
705e196b6c
upgrade event notification dependencies (#11095) 2020-12-12 20:31:28 -08:00
Andreas Auernhammer
7b5223d83d
add vulnerability report policy (#11084) 2020-12-12 17:38:37 -08:00
Anis Elleuch
f164085227
xl: Always set root disk to true in test environment (#11094)
Tests environments (go test or manual testing) should always consider
the passed disks are root disks and should not rely on disk.IsRootDisk()
function. The reason is that this latter can return a false negative
when called in a busy system. However, returning a false negative will
only occur in a testing environment and not in a production, so we can
accept this trade-off for now.
2020-12-12 16:10:07 -08:00
Minio Trusted
31bf6f0c25 Update yaml files to latest version RELEASE.2020-12-12T08-39-07Z 2020-12-12 08:56:30 +00:00
Harshavardhana
48191dd748
return NoSuchVersion if invalid version-id is specified (#11091) 2020-12-11 20:44:08 -08:00
Anis Elleuch
c4f29d24da
metacache: Ask all disks when drive count is 4 (#11087) 2020-12-11 17:54:31 -08:00
Harshavardhana
db7890660e
fix: a crash when disk is nil, safe access on erasureDisks (#11089)
fixes #11088
2020-12-11 16:58:36 -08:00
Poorna Krishnamoorthy
9adc33efbb
Return version-id header in DeleteObject response (#11090)
even when the object version is non-existent

To make this consistent with aws behavior.

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
2020-12-11 16:58:15 -08:00
Poorna Krishnamoorthy
8f65aba04b
ignore NoSuchVersion error in DeleteObjects API (#11086)
Currently, the error response reports NoSuchVersion
for a non-existent version-id, whereas AWS ignores it.
2020-12-11 12:39:09 -08:00
Harshavardhana
3a0082f0f1
fix: TTFB prometheus metrics calculation (#11082)
until now metrics was reporting entire call
duration instead of ttfb's this PR fixes it
2020-12-10 23:02:25 -08:00
Harshavardhana
14792cdbc6
docs: fix the metrics formatting (#11081) 2020-12-10 18:15:47 -08:00
Harshavardhana
4939987eb8
update deps to latest for some vulnerable deps (#11080)
- github.com/miekg/dns
- update elasticsearch library deps
  to circumvent some aws-sdk-go deps
2020-12-10 13:23:06 -08:00
Klaus Post
4bca62a0bd
crawler: Stream bucket usage cache data (#11068)
Stream bucket caches to storage and through RPC calls.
2020-12-10 13:03:22 -08:00
Klaus Post
82e2be4239
metacache: Speed up cleanup operation (#11078)
Perform cleanup operations on copied data. Avoids read locking
data while determining which caches to keep.

Also, reduce the log(N*N) operation to log(N*M) where M caches 
with the same root or below when checking potential replacements.
2020-12-10 12:30:28 -08:00
Harshavardhana
4550ac6fff
fix: refactor locks to apply them uniquely per node (#11052)
This refactor is done for few reasons below

- to avoid deadlocks in scenarios when number
  of nodes are smaller < actual erasure stripe
  count where in N participating local lockers
  can lead to deadlocks across systems.

- avoids expiry routines to run 1000 of separate
  network operations and routes per disk where
  as each of them are still accessing one single
  local entity.

- it is ideal to have since globalLockServer
  per instance.

- In a 32node deployment however, each server
  group is still concentrated towards the
  same set of lockers that partipicate during
  the write/read phase, unlike previous minio/dsync
  implementation - this potentially avoids send
  32 requests instead we will still send at max
  requests of unique nodes participating in a
  write/read phase.

- reduces overall chattiness on smaller setups.
2020-12-10 07:28:37 -08:00
Harshavardhana
97856bfebf
fix: grafana double counting for bucket usage, histrogram and objects (#11070) 2020-12-09 20:30:37 -08:00
Harshavardhana
a60a0e52bb ubi-minimal doesn't support arm32, remove from build manifest 2020-12-09 18:56:11 -08:00
Minio Trusted
83a67a1d21 Update yaml files to latest version RELEASE.2020-12-10T01-54-29Z 2020-12-10 02:12:01 +00:00
Harshavardhana
12391ec4ba
update nats.io/jwt dependency (#11066)
primarily for a vulnerability report https://nvd.nist.gov/vuln/detail/CVE-2020-26521
2020-12-09 14:30:35 -08:00
Klaus Post
e65ed2e44f
listcache: Add path index (#11063)
Add a root path index.

```
Before:
Benchmark_bucketMetacache_findCache-32    	   10000	    730737 ns/op

With excluded prints:
Benchmark_bucketMetacache_findCache-32    	   10000	    207100 ns/op

With the root path:
Benchmark_bucketMetacache_findCache-32    	  705765	      1943 ns/op
```

Benchmark used (not linear):

```Go
func Benchmark_bucketMetacache_findCache(b *testing.B) {
	bm := newBucketMetacache("", false)
	for i := 0; i < b.N; i++ {
		bm.findCache(listPathOptions{
			ID:           mustGetUUID(),
			Bucket:       "",
			BaseDir:      "prefix/" + mustGetUUID(),
			Prefix:       "",
			FilterPrefix: "",
			Marker:       "",
			Limit:        0,
			AskDisks:     0,
			Recursive:    false,
			Separator:    slashSeparator,
			Create:       true,
			CurrentCycle: 0,
			OldestCycle:  0,
		})
	}
}
```

Replaces #11058
2020-12-09 08:37:43 -08:00
Anis Elleuch
d90044b847
federation: Redirect Lifecycle PUT request by bucket name (#11062)
The bucket forwarder handler considers MakeBucket to be always local but
it mistakenly thinks that PUT bucket lifecycle to be a MakeBucket call.

Fix the check of the MakeBucket call by ensuring that the query is empty
in the PUT url.
2020-12-09 07:25:26 -08:00
Harshavardhana
d8c1f93de6
reject mixed drive situations with drives on root disks (#11057)
till now we used to match the inode number of the root
drive and the drive path minio would use, if they match
we knew that its a root disk.

this may not be true in all situations such as running
inside a container environment where the container might
be mounted from a different partition altogether, root
disk detection might fail.
2020-12-09 00:27:02 -08:00
Nitish Tiwari
54d243cd98
fix: grafana dashboard calculating online nodes (#11041)
Also use a generic name instead of diff names per revision
2020-12-09 00:26:42 -08:00
Harshavardhana
d74e4642e3
avoid updating nsswitch.conf for redhat UBI images (#11056) 2020-12-08 14:28:12 -08:00
Anis Elleuch
a51488cbaa
s3: Fix reading GET with partNumber specified (#11032)
partNumber was miscalculting the start and end of parts when partNumber
query is specified in the GET request. This commit fixes it and also
fixes the ContentRange header in that case.
2020-12-08 13:12:42 -08:00
Ritesh H Shukla
04848dfa1c
Add documentation for bucket replication related metrics (#11055) 2020-12-08 12:48:10 -08:00
Nitish Tiwari
78d18d8fc8
Remove alpine based image in favour or RedHat UBI (#11006) 2020-12-08 11:14:06 -08:00
Harshavardhana
dc819afa44 fix: auto update crawler meta version
PR 038bcd9079 introduced
version '3', we need to make sure that we do not
print an unexpected error instead log a message to
indicate we will auto update the version.
2020-12-08 10:40:51 -08:00
Harshavardhana
4a564336fe Revert "Add metrics for nodes online and offline (#11050)"
This reverts commit f60bbdf86b.
2020-12-08 09:23:35 -08:00
Anis Elleuch
6b7ced80fe
make: Add hotfix target to generate hotfix binaries (#11053)
hotfix target will fetch the release tag prior to the latest commit and create a binary
with the same release tag plus '.hotfix' suffix

e.g.   RELEASE.2020-12-03T05-49-24Z.hotfix
2020-12-08 08:12:13 -08:00
Ritesh H Shukla
f60bbdf86b
Add metrics for nodes online and offline (#11050) 2020-12-08 01:06:27 -08:00
Harshavardhana
8c79f87f02
add dynamic config docs (#11048)
Co-authored-by: Eco <41090896+eco-minio@users.noreply.github.com>
2020-12-07 19:02:20 -08:00
Poorna Krishnamoorthy
f3beb1236a
Add cache usage, total capacity to prometheus metrics (#11026) 2020-12-07 16:35:11 -08:00
Poorna Krishnamoorthy
934bed47fa
Add transition event notification (#11047)
This is a MinIO specific extension to allow monitoring of transition events.
2020-12-07 13:53:28 -08:00
Ritesh H Shukla
038bcd9079
Add replication capacity metrics support in crawler (#10786) 2020-12-07 13:47:48 -08:00
Justin Page
6d70f6a4ac
fix: README with missing word (#11035)
Include "to" for clarity.
2020-12-07 11:11:58 -08:00
Harshavardhana
ce93b2681b
fix: re-use er.getDisks() properly in certain calls (#11043) 2020-12-07 10:04:07 -08:00
Harshavardhana
8d036ed6d8
fix: allow sub-admin to modify password for other users (#11039)
fixes #11037
2020-12-06 20:36:34 -08:00
Harshavardhana
9c53cc1b83
fix: heal multiple buckets in bulk (#11029)
makes server startup, orders of magnitude
faster with large number of buckets
2020-12-05 13:00:44 -08:00
Harshavardhana
3514e89eb3
support envs as well for new crawler sub-system (#11033) 2020-12-04 21:54:24 -08:00
Nitish Tiwari
6ff12f5f01
Add the dashboard json file (#11028)
This will allow users to contribute to the dashboard as needed.
2020-12-04 16:27:41 -08:00
Harshavardhana
ee2a436a5b
fix: release locks if the client timedout (#11030)
situations where client indeed timedout there was
a potential to falsely think that lock is still
active.
2020-12-04 11:33:56 -08:00
Klaus Post
a896125490
Add crawler delay config + dynamic config values (#11018) 2020-12-04 09:32:35 -08:00