88d719689c
Synchronize bucket cycle numbers ( #13058 )
...
Synchronize bucket cycles so it is much more
likely that the same prefixes will be picked up
for scanning.
Use the global bloom filter cycle for that.
Bump bloom filter versions to clear those.
2021-08-25 08:25:26 -07:00
200eb8dc0e
fix: remove any internal metadata keys from notification ( #13062 )
2021-08-24 21:13:37 -07:00
082650bea3
update helm version to v1.0.4
2021-08-24 19:12:51 -07:00
abf079135e
Update yaml files to latest version RELEASE.2021-08-25T00-41-18Z
2021-08-25 02:01:05 +00:00
f00e8bc107
update console v0.9.4
2021-08-24 17:41:18 -07:00
ce05e67a0c
Add admin api to return sys config info ( #12988 )
...
The intention is to list values of sys config that can potentially
impact the performance of minio.
At present, it will return max value configured for rlimit
Signed-off-by: Shireesh Anjal <shireesh@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-08-24 17:09:37 -07:00
fecb1b0489
fix: look to exact return returned from initServer() in tests
2021-08-24 15:27:02 -07:00
6a7e22386e
Use part sizes correctly in multipart replication ( #13061 )
...
fixes #13057
2021-08-24 14:41:05 -07:00
85dfb4351c
fix: allow an entire set to be dropped ( #13060 )
...
proceed to heal the cluster when all the
drives in a set have failed, this is extremely
rare occurrence but even if it happens we allow
the cluster to be functional.
2021-08-24 12:43:57 -07:00
addf15f61f
update console version to v0.9.3
2021-08-24 12:40:16 -07:00
bbf3576f70
remove unecessary metadata structs in applyTransitionAction() ( #13059 )
2021-08-24 12:24:00 -07:00
da3f4bd452
update helm release to v1.0.3
2021-08-24 12:04:07 -07:00
5eb6f903f2
fix: helm missing console ingress apiVersion
...
fixes #13049
2021-08-24 11:58:05 -07:00
60394ddf83
Add support for changing job name in Grafana dashboard ( #13050 )
2021-08-24 09:51:09 -07:00
293d261cf9
use available memory to restrict API calls ( #13047 )
...
also choose 90% of the available memory
to calculate maximum API calls.
2021-08-24 09:14:46 -07:00
f1cab828ee
fix: New disks healing should pick unformatted disks as well ( #13054 )
...
A recent regression caused new disks not being re-formatted. In the old
code, a disk needed be 'online' to be chosen to be formatted but the
disk has to be already formatted for XL storage IsOnline() function to
return true.
It is enough to check if XL storage is nil or not if we want to avoid
formatting root disks.
Co-authored-by: Anis Elleuch <anis@min.io>
2021-08-24 07:40:56 -07:00
6a8d0fb955
fix(Router): typo: completemutipartupload to completemultipartupload ( #13051 )
2021-08-24 07:14:34 -07:00
c8ca055935
Fix concurrent map read/write ( #13052 )
...
Clones were not independent.
Fixes race:
```
WARNING: DATA RACE
Read at 0x00c002040cc0 by goroutine 50:
runtime.mapiterinit()
c:/go/src/runtime/map.go:802 +0x0
github.com/minio/minio/cmd.(*dataUsageCache).flatten()
d:/minio/minio/cmd/data-usage-cache.go:551 +0xad
github.com/minio/minio/cmd.(*dataUsageCache).dui()
d:/minio/minio/cmd/data-usage-cache.go:352 +0x144
github.com/minio/minio/cmd.(*erasureServerPools).NSScanner.func3.1()
d:/minio/minio/cmd/erasure-server-pool.go:542 +0x2a4
github.com/minio/minio/cmd.(*erasureServerPools).NSScanner.func3()
d:/minio/minio/cmd/erasure-server-pool.go:561 +0x24b
Previous write at 0x00c002040cc0 by goroutine 1391:
runtime.mapassign_faststr()
c:/go/src/runtime/map_faststr.go:202 +0x0
github.com/minio/minio/cmd.(*dataUsageEntry).addChild()
d:/minio/minio/cmd/data-usage-cache.go:231 +0x313
github.com/minio/minio/cmd.(*dataUsageCache).replace()
d:/minio/minio/cmd/data-usage-cache.go:383 +0x293
github.com/minio/minio/cmd.erasureObjects.nsScanner.func1()
d:/minio/minio/cmd/erasure.go:428 +0x3a6
```
2021-08-24 07:11:38 -07:00
e0abb46616
fix: go mod tidy
2021-08-23 17:32:29 -07:00
170b89f468
Update minio-go library ( #13045 )
...
This fixes some issues in replication & S3 gateway:
- https://github.com/minio/minio-go/pull/1531
- https://github.com/minio/minio-go/pull/1533
2021-08-23 17:31:53 -07:00
674c6f7a7b
fix: resync of replication of delete markers ( #12932 )
...
Fixes #12919
2021-08-23 14:48:22 -07:00
db35bcf2ce
heal: Remove transitioned objects' parts from outdated disks ( #13018 )
...
Bonus: check equality for replication and other metadata
2021-08-23 13:14:55 -07:00
901d1314af
Fix formatting disks in a test environment ( #13043 )
...
markRootDisksAsDown() relies on disk info even if the
disk is unformatted. Therefore, we should always return
DiskInfo data even when DiskInfo storage API returns
errUnformattedDisk
2021-08-23 12:53:54 -07:00
9e9bfd0255
fix: missing index.yaml for helm charts
2021-08-23 11:32:11 -07:00
1080609c86
Reuse buffers when writing metadata ( #13040 )
...
Simplify returning buffers.
Tested using `warp mixed --duration=1m --obj.size=100K`:
```
Operation: DELETE
Operations: 7148 -> 7642
* Average: +6.77% (+8.1) obj/s
-------------------
Operation: GET
Operations: 32200 -> 34403
* Average: +6.74% (+3.5 MiB/s) throughput, +6.74% (+36.2) obj/s
* First Byte: Average: -105.403µs (-3%), Median: -309µs (-11%), Best: -2.7µs (-0%), Worst: +3.5637ms (+3%)
-------------------
Operation: PUT
Operations: 10741 -> 11475
* Average: +6.78% (+1.2 MiB/s) throughput, +6.78% (+12.1) obj/s
-------------------
Operation: STAT
Operations: 21465 -> 22927
* Average: +6.71% (+24.0) obj/s
```
2021-08-23 11:17:27 -07:00
8315bcd0d8
Fix TrafficMeter data race ( #13041 )
...
When reading `TrafficMeter` values, there was a value receiver.
This means that receivers are copied unsafely when invoked.
Fixes race seen with `-race` build.
2021-08-23 09:19:14 -07:00
7fb9301c03
heal: Return parity for storage classes in heal info API ( #13038 )
...
`mc admin heal` command will show servers/disks tolerance, for that
purpose, you need to know the number of parity disks for each storage
class.
Parity is always the same in all pools.
2021-08-23 08:50:35 -07:00
63f3e5c3fc
replication: Lock object while replicating ( #13014 )
...
Introduce a replication lock that will ensure that only one replication
operation will run for any given object at any time.
Fixes #13013
2021-08-23 08:16:18 -07:00
a75412440f
remove flaky healing test
2021-08-23 08:01:36 -07:00
47de1d2e0e
Fix diskinfo race ( #12857 )
...
Fixes share info struct.
```
WARNING: DATA RACE
Read at 0x00c011780618 by goroutine 419:
github.com/minio/minio/cmd.(*DiskMetrics).DecodeMsg()
c:/gopath/src/github.com/minio/minio/cmd/storage-datatypes_gen.go:331 +0x247
github.com/minio/minio/cmd.(*DiskInfo).DecodeMsg()
c:/gopath/src/github.com/minio/minio/cmd/storage-datatypes_gen.go:76 +0x5ec
github.com/tinylib/msgp/msgp.Decode()
c:/gopath/pkg/mod/github.com/tinylib/msgp@v1.1.6-0.20210521143832-0becd170c402/msgp/read.go:105 +0x70
github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo.func1.1()
c:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:288 +0x235
github.com/minio/minio/cmd.(*timedValue).Get()
c:/gopath/src/github.com/minio/minio/cmd/utils.go:886 +0x77
github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo()
c:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:297 +0xf9
github.com/minio/minio/cmd.getDiskInfos()
c:/gopath/src/github.com/minio/minio/cmd/object-api-utils.go:962 +0x1a8
github.com/minio/minio/cmd.(*erasureServerPools).getServerPoolsAvailableSpace.func1()
c:/gopath/src/github.com/minio/minio/cmd/erasure-server-pool.go:241 +0x27c
github.com/minio/minio/internal/sync/errgroup.(*Group).Go.func1()
c:/gopath/src/github.com/minio/minio/internal/sync/errgroup/errgroup.go:123 +0xd7
Previous write at 0x00c011780618 by goroutine 423:
github.com/minio/minio/cmd.(*DiskMetrics).DecodeMsg()
c:/gopath/src/github.com/minio/minio/cmd/storage-datatypes_gen.go:332 +0x6e4
github.com/minio/minio/cmd.(*DiskInfo).DecodeMsg()
c:/gopath/src/github.com/minio/minio/cmd/storage-datatypes_gen.go:76 +0x5ec
github.com/tinylib/msgp/msgp.Decode()
c:/gopath/pkg/mod/github.com/tinylib/msgp@v1.1.6-0.20210521143832-0becd170c402/msgp/read.go:105 +0x70
github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo.func1.1()
c:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:288 +0x235
github.com/minio/minio/cmd.(*timedValue).Get()
c:/gopath/src/github.com/minio/minio/cmd/utils.go:886 +0x77
github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo()
c:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:297 +0xf9
github.com/minio/minio/cmd.getDiskInfos()
c:/gopath/src/github.com/minio/minio/cmd/object-api-utils.go:962 +0x1a8
github.com/minio/minio/cmd.(*erasureServerPools).getServerPoolsAvailableSpace.func1()
c:/gopath/src/github.com/minio/minio/cmd/erasure-server-pool.go:241 +0x27c
github.com/minio/minio/internal/sync/errgroup.(*Group).Go.func1()
c:/gopath/src/github.com/minio/minio/internal/sync/errgroup/errgroup.go:123 +0xd7
```
2021-08-23 01:13:47 -07:00
14fe8ecb58
fix: decodeDirObject in prefix usage function ( #13026 )
...
prefixes at top level create such as
```
~ mc mb alias/bucket/prefix
```
The prefix/ incorrect appears as prefix__XL_DIR__/
in the accountInfo output, make sure to trim '__XL_DIR__'
2021-08-22 16:46:45 -07:00
0f01e7ef0f
fix: check for xl.meta as directory fallback ( #13023 )
...
Objects uploaded in this format for example
```
mc cp /etc/hosts alias/bucket/foo/bar/xl.meta
mc ls -r alias/bucket/foo/bar
```
Won't list the object, handle this scenario.
2021-08-21 00:12:29 -07:00
16f7c64a9f
update helm to v1.0.1
2021-08-20 15:32:29 -07:00
00aa9841b7
Add MinIO server helm chart ( #12509 )
2021-08-20 15:30:54 -07:00
7802088e71
Update yaml files to latest version RELEASE.2021-08-20T18-32-01Z
2021-08-20 20:42:00 +00:00
6d04c9c585
populate additional claims for prometheus endpoint ( #13011 )
...
service accounts and STS provide additional claims for
policy authorization which needs to be verified along
with Prometheus issuer claim.
2021-08-20 11:32:01 -07:00
e210cb3670
fix: use transition/replication fields in FileInfo quorum calculation ( #13010 )
2021-08-19 14:55:42 -07:00
47b577fcc0
Lock while creating buckets ( #12999 )
...
Ensure that one call will succeed and others will serialize
Example failure without code in place:
```
bucket-policy-handlers_test.go:120: unexpected error: cmd.InsufficientWriteQuorum: Storage resources are insufficient for the write operation doz2wjqaovp5kvlrv11fyacowgcvoziszmkmzzz9nk9au946qwhci4zkane5-1/
bucket-policy-handlers_test.go:120: unexpected error: cmd.InsufficientWriteQuorum: Storage resources are insufficient for the write operation doz2wjqaovp5kvlrv11fyacowgcvoziszmkmzzz9nk9au946qwhci4zkane5-1/
bucket-policy-handlers_test.go:135: want 1 ok, got 0
```
2021-08-19 13:21:02 -07:00
e9d970154d
use renameAll instead of deleteAll for metacache-manager ( #13005 )
...
renameAll is cheaper, rely on background deletes instead.
2021-08-19 09:16:14 -07:00
202d0b64eb
fix: enable go1.17 github ci/cd ( #12997 )
2021-08-18 18:35:22 -07:00
c25816eabc
xl walk: Limit walk concurrent IO ( #12885 )
...
We are observing heavy system loads, potentially
locking the system up for periods when concurrent
listing operations are performed.
We place a per-disk lock on walk IO operations.
This will minimize the impact of concurrent listing
operations on the entire system and de-prioritize
them compared to other operations.
Single list operations should remain largely unaffected.
2021-08-18 18:10:36 -07:00
ee028a4693
listObjects optimized to handle max-keys=1 when prefix is object ( #13000 )
...
Some applications albeit poorly written rather than using headObject
rely on listObjects to check for existence of object, this unusual
request always has prefix=(to actual object) and max-keys=1
handle this situation specially such that we can avoid readdir()
on the top level parent to avoid sorting and skipping, ensuring
that such type of listObjects() always behaves similar to a
headObject() call.
2021-08-18 18:05:05 -07:00
9c65168312
fix: all levels deep flat key match ( #12996 )
...
this addresses a regression from #12984
which only addresses flat key from single
level deep at bucket level.
added extra tests as well to cover all
these scenarios.
2021-08-18 07:40:53 -07:00
16aeb68c28
Update yaml files to latest version RELEASE.2021-08-17T20-53-08Z
2021-08-17 22:34:33 +00:00
6167104644
update console to v0.9.1
2021-08-17 13:53:08 -07:00
30b77f59b1
doc: Add ilm prometheus metrics information ( #12994 )
2021-08-17 12:19:36 -07:00
a690772cc5
add support to set subnet license for embedded console ( #12993 )
2021-08-17 11:56:01 -07:00
cf8abd8888
Add prometheus metrics for ILM tasks ( #12933 )
2021-08-17 10:21:19 -07:00
17e0de1e87
fix: typo in CONTRIBUTING.md ( #12907 )
2021-08-17 09:20:10 -07:00
b7e3651d3c
Set free-version id in case of version/version-suspended buckets ( #12982 )
...
This free-version id may be used to track tiered object contents of the
object (version) being deleted.
2021-08-17 08:59:48 -07:00
Klaus Post
Harshavardhana
Minio Trusted
Shireesh Anjal
Poorna Krishnamoorthy
Nitish Tiwari
Anis Elleuch
MoonJustry
Krishnan Parthasarathi
johnnyaug