Commit Graph

3958 Commits

Author SHA1 Message Date
Andreas Auernhammer
e438dccf19
sts: add support for certificate-based authentication (#12748)
This commit adds a new STS API for X.509 certificate
authentication.

A client can make an HTTP POST request over a TLS connection
and MinIO will verify the provided client certificate, map it to an 
S3 policy and return temp. S3 credentials to the client.

So, this STS API allows clients to authenticate with X.509
certificates over TLS and obtain temp. S3 credentials.

For more details and examples refer to the docs/sts/tls.md
documentation.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-09-07 19:03:48 -07:00
Anis Elleuch
42b1d92b2a
azure: Fix crash after using ParseForm() for query param lookups (#13147)
Azure storage SDK uses http.Request feature which panics when the
request contains r.Form popuplated.

Azure gateway code creates a new request, however it modifies the
transport to add our metrics code which sets Request.Form during
shouldMeterRequest() call.

This commit simplifies shouldMeterRequest() to avoid setting
request.Form and avoid the crash.
2021-09-04 09:07:24 -07:00
Harshavardhana
1250312287
fail ready/liveness if etcd is unhealthy in gateway mode (#13146) 2021-09-03 17:05:41 -07:00
Klaus Post
308371b434
Clean up ToFileInfo and avoid copy (#13144)
Simplify code and remove an iteration of all versions. Remove unneded copy.
2021-09-03 12:31:32 -07:00
Anis Elleuch
88e6c11746
iam: Retry connection to etcd during initialization (#13143)
Retry connecting to etcd during IAM initialization when etcd is enabled.
2021-09-03 09:10:48 -07:00
Harshavardhana
a19e3bc9d9
add more dangling heal related tests (#13140)
also make sure that HealObject() never returns
'ObjectNotFound' or 'VersionNotFound' errors,
as those are meaningless and not useful for the
caller.
2021-09-02 20:56:13 -07:00
Harshavardhana
495c55e6a5
fix: make sure to delete dangling objects during heal (#13138)
heal with --remove was not removing dangling versions
on versioned buckets, this PR fixes this properly.

this is a regression introduced in PR #12617
2021-09-02 17:45:30 -07:00
Poorna Krishnamoorthy
a366143c5b
Remove replication permission check (#13135)
Fixes #13105
2021-09-02 09:31:13 -07:00
Harshavardhana
f486cfae86 point all docs for containers to quay.io 2021-09-01 18:48:26 -07:00
Harshavardhana
27c9f8be7a ignore MINIO_CERT_PASSWD to be same on all instances 2021-09-01 16:02:35 -07:00
Harshavardhana
67b6c945e2 whitelist certain ENVs from bootstrap check 2021-09-01 15:10:17 -07:00
Harshavardhana
c89aee37b9
fix: log errors for incorrect environment inputs (#13121)
Invalid MINIO_ARGS, MINIO_ENDPOINTS would be silently
ignored when using remoteEnv style, make sure to log
errors to indicate invalid configuration.
2021-09-01 11:34:07 -07:00
Harshavardhana
03b7bebc96
fix: invalid quorum calculation in TransitionObject (#13125)
Quorum calculation should be based on the
existing metadata, custom quorum calculation
can lead to unreadable content.
2021-09-01 08:57:42 -07:00
Harshavardhana
f89d0f68d0
fix: missing cleanup of tmp folders in NAS gateway setup (#13124)
console service should be shutdown last once all shutdown
sequences are complete, this is to ensure that we do not
prematurely kill the server before it cleans up the

`.minio.sys/tmp/uuid` folder.

NOTE: this only applies to NAS gateway setup.
2021-08-31 18:52:48 -07:00
Harshavardhana
72a288f73f
fix: add x-amz-server-side-encryption header in ListObjectsM (#13122)
fixes #13046
2021-08-31 17:18:13 -07:00
Harshavardhana
0f7a51f461
fix: speedTest between peers keep the connection alive (#13120)
for longer durations keep the speedTest alive instead
of timing them out based on ResponseHeaderTimeout.
2021-08-31 14:08:23 -07:00
Klaus Post
556552340a
listing: Don't log errFileNotFound and friends (#13119) 2021-08-31 09:46:42 -07:00
Harshavardhana
2077d27053 init() globalAPIConfig only once 2021-08-30 22:46:54 -07:00
Klaus Post
76b3d3c559
fix: missing close on error for keepAlive connections (#13109)
Add missing close when error is reported
before body is done.
2021-08-30 08:46:46 -07:00
Dominik Hassler
514b2d6f12
fix: build on illumos (Solaris) (#13097) 2021-08-30 08:40:16 -07:00
Klaus Post
470553ff5d
Tweak readall allocation and renameData buffer reuse (#13108)
Use a single allocation for reading the file, not the growing buffer of `io.ReadAll`.

Reuse the write buffer if we can when writing metadata in RenameData.
2021-08-30 08:38:11 -07:00
Harshavardhana
88e0aa1cb2
verify all nodes have same ENVs in bootstrap (#13096) 2021-08-30 08:27:39 -07:00
Harshavardhana
35f2552fc5
reduce extra getObjectInfo() calls during ILM transition (#13091)
* reduce extra getObjectInfo() calls during ILM transition

This PR also changes expiration logic to be non-blocking,
scanner is now free from additional costs incurred due
to slower object layer calls and hitting the drives.

* move verifying expiration inside locks
2021-08-27 17:06:47 -07:00
Anis Elleuch
e05886561d
lock: Fix Refresh logic with multi resources lock (#13092)
A multi resources lock is a single lock UID with multiple associated
resources. This is created for example by multi objects delete
operation. This commit changes the behavior of Refresh() to iterate over
all locks having the same UID and refresh them.

Bonus: Fix showing top locks for multi delete objects
2021-08-27 13:07:55 -07:00
Klaus Post
2451b9a75a
fix: hanging operations on PUT with slow IO (#13087)
#11878 added "keepHTTPResponseAlive" to CreateFile requests. 
The problem is that it will begin writing to the response before the 
body is read after 10 seconds. This will abort the writes on the 
client-side, since it assumes the server has received what it wants.

The proposed solution here is to monitor the completion of the body 
before beginning to send keepalive pings.

Fixes observed high number of goroutines stuck in `io.Copy` in 
`github.com/minio/minio/cmd.(*xlStorage).CreateFile` and 
`(*storageRESTClient).CreateFile` stuck in `http.DrainBody`.
2021-08-27 09:16:36 -07:00
Anis Elleuch
06b71c99ee
locks: Ensure local lock removal after a failed refresh (#12979)
In the event when a lock is not refreshed in the cluster, this latter
will be automatically removed in the subsequent cleanup of non 
refreshed locks routine, but it forgot to clean the local server, 
hence having the same weird stale locks present.

This commit will remove the lock locally also in remote nodes, if
removing a lock from a remote node will fail, it will be anyway 
removed later in the locks cleanup routine.
2021-08-27 08:59:36 -07:00
Harshavardhana
ae8f7f11d5
fix: svc accounts cannot have same name as parent/targetUser (#13082)
Currently in master this can cause existing
parent users to stop working and lead to
credentials getting overwritten.

```
~ mc admin user add alias/ minio123 minio123456
```

```
~ mc admin user svcacct add alias/ minio123 \
    --access-key minio123 --secret-key minio123456
```

This PR rejects all such scenarios.
2021-08-26 21:57:30 -07:00
Harshavardhana
ed16ce9b73
add healing workers support to parallelize healing (#13081)
Faster healing as well as making healing more
responsive for faster scanner times.

also fixes a bug introduced in #13079, newly replaced
disks were not healing automatically.
2021-08-26 20:32:58 -07:00
Poorna Krishnamoorthy
27f895cf2c
Check pathlength before reading metadata (#13080)
fixes bug where the server returns 503 instead of 400 if 
objectName is longer than 255 characters

Fixes regression introduced in #12942
2021-08-26 16:23:12 -07:00
Harshavardhana
c11a2ac396
refactor healing to remove certain structs (#13079)
- remove sourceCh usage from healing
  we already have tasks and resp channel

- use read locks to lookup globalHealConfig

- fix healing resolver to pick candidates quickly
  that need healing, without this resolver was
  unexpectedly skipping.
2021-08-26 14:06:04 -07:00
Harshavardhana
0559f46bbb
fix: make healObject() make non-blocking (#13071)
healObject() should be non-blocking to ensure
that scanner is not blocked for a long time,
this adversely affects performance of the scanner
and also affects the way usage is updated
subsequently.

This PR allows for a non-blocking behavior for
healing, dropping operations that cannot be queued
anymore.
2021-08-25 17:46:20 -07:00
Klaus Post
e1b0582859
fsOpenFile: Close on error (#13064)
Close files on error.
2021-08-25 09:43:01 -07:00
Klaus Post
88d719689c
Synchronize bucket cycle numbers (#13058)
Synchronize bucket cycles so it is much more
likely that the same prefixes will be picked up
for scanning.

Use the global bloom filter cycle for that. 
Bump bloom filter versions to clear those.
2021-08-25 08:25:26 -07:00
Harshavardhana
200eb8dc0e
fix: remove any internal metadata keys from notification (#13062) 2021-08-24 21:13:37 -07:00
Shireesh Anjal
ce05e67a0c
Add admin api to return sys config info (#12988)
The intention is to list values of sys config that can potentially
impact the performance of minio.

At present, it will return max value configured for rlimit

Signed-off-by: Shireesh Anjal <shireesh@minio.io>

Co-authored-by: Harshavardhana <harsha@minio.io>
2021-08-24 17:09:37 -07:00
Poorna Krishnamoorthy
6a7e22386e
Use part sizes correctly in multipart replication (#13061)
fixes #13057
2021-08-24 14:41:05 -07:00
Harshavardhana
85dfb4351c
fix: allow an entire set to be dropped (#13060)
proceed to heal the cluster when all the
drives in a set have failed, this is extremely
rare occurrence but even if it happens we allow
the cluster to be functional.
2021-08-24 12:43:57 -07:00
Harshavardhana
bbf3576f70
remove unecessary metadata structs in applyTransitionAction() (#13059) 2021-08-24 12:24:00 -07:00
Harshavardhana
293d261cf9
use available memory to restrict API calls (#13047)
also choose 90% of the available memory
to calculate maximum API calls.
2021-08-24 09:14:46 -07:00
Anis Elleuch
f1cab828ee
fix: New disks healing should pick unformatted disks as well (#13054)
A recent regression caused new disks not being re-formatted. In the old
code, a disk needed be 'online' to be chosen to be formatted but the
disk has to be already formatted for XL storage IsOnline() function to
return true.

It is enough to check if XL storage is nil or not if we want to avoid
formatting root disks.

Co-authored-by: Anis Elleuch <anis@min.io>
2021-08-24 07:40:56 -07:00
MoonJustry
6a8d0fb955
fix(Router): typo: completemutipartupload to completemultipartupload (#13051) 2021-08-24 07:14:34 -07:00
Klaus Post
c8ca055935
Fix concurrent map read/write (#13052)
Clones were not independent.

Fixes race:

```
WARNING: DATA RACE
Read at 0x00c002040cc0 by goroutine 50:
  runtime.mapiterinit()
      c:/go/src/runtime/map.go:802 +0x0
  github.com/minio/minio/cmd.(*dataUsageCache).flatten()
      d:/minio/minio/cmd/data-usage-cache.go:551 +0xad
  github.com/minio/minio/cmd.(*dataUsageCache).dui()
      d:/minio/minio/cmd/data-usage-cache.go:352 +0x144
  github.com/minio/minio/cmd.(*erasureServerPools).NSScanner.func3.1()
      d:/minio/minio/cmd/erasure-server-pool.go:542 +0x2a4
  github.com/minio/minio/cmd.(*erasureServerPools).NSScanner.func3()
      d:/minio/minio/cmd/erasure-server-pool.go:561 +0x24b

Previous write at 0x00c002040cc0 by goroutine 1391:
  runtime.mapassign_faststr()
      c:/go/src/runtime/map_faststr.go:202 +0x0
  github.com/minio/minio/cmd.(*dataUsageEntry).addChild()
      d:/minio/minio/cmd/data-usage-cache.go:231 +0x313
  github.com/minio/minio/cmd.(*dataUsageCache).replace()
      d:/minio/minio/cmd/data-usage-cache.go:383 +0x293
  github.com/minio/minio/cmd.erasureObjects.nsScanner.func1()
      d:/minio/minio/cmd/erasure.go:428 +0x3a6
```
2021-08-24 07:11:38 -07:00
Poorna Krishnamoorthy
674c6f7a7b
fix: resync of replication of delete markers (#12932)
Fixes #12919
2021-08-23 14:48:22 -07:00
Krishnan Parthasarathi
db35bcf2ce
heal: Remove transitioned objects' parts from outdated disks (#13018)
Bonus: check equality for replication and other metadata
2021-08-23 13:14:55 -07:00
Anis Elleuch
901d1314af
Fix formatting disks in a test environment (#13043)
markRootDisksAsDown() relies on disk info even if the 
disk is unformatted. Therefore, we should always return 
DiskInfo data even when DiskInfo storage API returns 
errUnformattedDisk
2021-08-23 12:53:54 -07:00
Klaus Post
1080609c86
Reuse buffers when writing metadata (#13040)
Simplify returning buffers.

Tested using `warp mixed --duration=1m --obj.size=100K`:

```
Operation: DELETE
Operations: 7148 -> 7642
* Average: +6.77% (+8.1) obj/s
-------------------
Operation: GET
Operations: 32200 -> 34403
* Average: +6.74% (+3.5 MiB/s) throughput, +6.74% (+36.2) obj/s
* First Byte: Average: -105.403µs (-3%), Median: -309µs (-11%), Best: -2.7µs (-0%), Worst: +3.5637ms (+3%)
-------------------
Operation: PUT
Operations: 10741 -> 11475
* Average: +6.78% (+1.2 MiB/s) throughput, +6.78% (+12.1) obj/s
-------------------
Operation: STAT
Operations: 21465 -> 22927
* Average: +6.71% (+24.0) obj/s
```
2021-08-23 11:17:27 -07:00
Anis Elleuch
7fb9301c03
heal: Return parity for storage classes in heal info API (#13038)
`mc admin heal` command will show servers/disks tolerance, for that
purpose, you need to know the number of parity disks for each storage
class.

Parity is always the same in all pools.
2021-08-23 08:50:35 -07:00
Klaus Post
63f3e5c3fc
replication: Lock object while replicating (#13014)
Introduce a replication lock that will ensure that only one replication 
operation will run for any given object at any time.

Fixes #13013
2021-08-23 08:16:18 -07:00
Klaus Post
47de1d2e0e
Fix diskinfo race (#12857)
Fixes share info struct.

```
WARNING: DATA RACE
Read at 0x00c011780618 by goroutine 419:
  github.com/minio/minio/cmd.(*DiskMetrics).DecodeMsg()
      c:/gopath/src/github.com/minio/minio/cmd/storage-datatypes_gen.go:331 +0x247
  github.com/minio/minio/cmd.(*DiskInfo).DecodeMsg()
      c:/gopath/src/github.com/minio/minio/cmd/storage-datatypes_gen.go:76 +0x5ec
  github.com/tinylib/msgp/msgp.Decode()
      c:/gopath/pkg/mod/github.com/tinylib/msgp@v1.1.6-0.20210521143832-0becd170c402/msgp/read.go:105 +0x70
  github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo.func1.1()
      c:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:288 +0x235
  github.com/minio/minio/cmd.(*timedValue).Get()
      c:/gopath/src/github.com/minio/minio/cmd/utils.go:886 +0x77
  github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo()
      c:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:297 +0xf9
  github.com/minio/minio/cmd.getDiskInfos()
      c:/gopath/src/github.com/minio/minio/cmd/object-api-utils.go:962 +0x1a8
  github.com/minio/minio/cmd.(*erasureServerPools).getServerPoolsAvailableSpace.func1()
      c:/gopath/src/github.com/minio/minio/cmd/erasure-server-pool.go:241 +0x27c
  github.com/minio/minio/internal/sync/errgroup.(*Group).Go.func1()
      c:/gopath/src/github.com/minio/minio/internal/sync/errgroup/errgroup.go:123 +0xd7

Previous write at 0x00c011780618 by goroutine 423:
  github.com/minio/minio/cmd.(*DiskMetrics).DecodeMsg()
      c:/gopath/src/github.com/minio/minio/cmd/storage-datatypes_gen.go:332 +0x6e4
  github.com/minio/minio/cmd.(*DiskInfo).DecodeMsg()
      c:/gopath/src/github.com/minio/minio/cmd/storage-datatypes_gen.go:76 +0x5ec
  github.com/tinylib/msgp/msgp.Decode()
      c:/gopath/pkg/mod/github.com/tinylib/msgp@v1.1.6-0.20210521143832-0becd170c402/msgp/read.go:105 +0x70
  github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo.func1.1()
      c:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:288 +0x235
  github.com/minio/minio/cmd.(*timedValue).Get()
      c:/gopath/src/github.com/minio/minio/cmd/utils.go:886 +0x77
  github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo()
      c:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:297 +0xf9
  github.com/minio/minio/cmd.getDiskInfos()
      c:/gopath/src/github.com/minio/minio/cmd/object-api-utils.go:962 +0x1a8
  github.com/minio/minio/cmd.(*erasureServerPools).getServerPoolsAvailableSpace.func1()
      c:/gopath/src/github.com/minio/minio/cmd/erasure-server-pool.go:241 +0x27c
  github.com/minio/minio/internal/sync/errgroup.(*Group).Go.func1()
      c:/gopath/src/github.com/minio/minio/internal/sync/errgroup/errgroup.go:123 +0xd7
```
2021-08-23 01:13:47 -07:00
Harshavardhana
14fe8ecb58
fix: decodeDirObject in prefix usage function (#13026)
prefixes at top level create such as

```
~ mc mb alias/bucket/prefix
```

The prefix/ incorrect appears as prefix__XL_DIR__/
in the accountInfo output, make sure to trim '__XL_DIR__'
2021-08-22 16:46:45 -07:00
Harshavardhana
0f01e7ef0f
fix: check for xl.meta as directory fallback (#13023)
Objects uploaded in this format for example

```
mc cp /etc/hosts alias/bucket/foo/bar/xl.meta
mc ls -r alias/bucket/foo/bar
```

Won't list the object, handle this scenario.
2021-08-21 00:12:29 -07:00
Harshavardhana
6d04c9c585
populate additional claims for prometheus endpoint (#13011)
service accounts and STS provide additional claims for
policy authorization which needs to be verified along
with Prometheus issuer claim.
2021-08-20 11:32:01 -07:00
Krishnan Parthasarathi
e210cb3670
fix: use transition/replication fields in FileInfo quorum calculation (#13010) 2021-08-19 14:55:42 -07:00
Klaus Post
47b577fcc0
Lock while creating buckets (#12999)
Ensure that one call will succeed and others will serialize

Example failure without code in place:
```
    bucket-policy-handlers_test.go:120: unexpected error: cmd.InsufficientWriteQuorum: Storage resources are insufficient for the write operation doz2wjqaovp5kvlrv11fyacowgcvoziszmkmzzz9nk9au946qwhci4zkane5-1/
    bucket-policy-handlers_test.go:120: unexpected error: cmd.InsufficientWriteQuorum: Storage resources are insufficient for the write operation doz2wjqaovp5kvlrv11fyacowgcvoziszmkmzzz9nk9au946qwhci4zkane5-1/
    bucket-policy-handlers_test.go:135: want 1 ok, got 0
```
2021-08-19 13:21:02 -07:00
Harshavardhana
e9d970154d
use renameAll instead of deleteAll for metacache-manager (#13005)
renameAll is cheaper, rely on background deletes instead.
2021-08-19 09:16:14 -07:00
Harshavardhana
202d0b64eb
fix: enable go1.17 github ci/cd (#12997) 2021-08-18 18:35:22 -07:00
Klaus Post
c25816eabc
xl walk: Limit walk concurrent IO (#12885)
We are observing heavy system loads, potentially
locking the system up for periods when concurrent
listing operations are performed.

We place a per-disk lock on walk IO operations.
This will minimize the impact of concurrent listing
operations on the entire system and de-prioritize
them compared to other operations.

Single list operations should remain largely unaffected.
2021-08-18 18:10:36 -07:00
Harshavardhana
ee028a4693
listObjects optimized to handle max-keys=1 when prefix is object (#13000)
Some applications albeit poorly written rather than using headObject
rely on listObjects to check for existence of object, this unusual
request always has prefix=(to actual object) and max-keys=1

handle this situation specially such that we can avoid readdir()
on the top level parent to avoid sorting and skipping, ensuring
that such type of listObjects() always behaves similar to a
headObject() call.
2021-08-18 18:05:05 -07:00
Harshavardhana
9c65168312
fix: all levels deep flat key match (#12996)
this addresses a regression from #12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
2021-08-18 07:40:53 -07:00
Harshavardhana
a690772cc5
add support to set subnet license for embedded console (#12993) 2021-08-17 11:56:01 -07:00
Krishnan Parthasarathi
cf8abd8888
Add prometheus metrics for ILM tasks (#12933) 2021-08-17 10:21:19 -07:00
Krishnan Parthasarathi
b7e3651d3c
Set free-version id in case of version/version-suspended buckets (#12982)
This free-version id may be used to track tiered object contents of the
object (version) being deleted.
2021-08-17 08:59:48 -07:00
Harshavardhana
ef4d023c85
fix: various performance improvements to tiering (#12965)
- deletes should always Sweep() for tiering at the
  end and does not need an extra getObjectInfo() call
- puts, copy and multipart writes should conditionally
  do getObjectInfo() when tiering targets are configured
- introduce 'TransitionedObject' struct for ease of usage
  and understanding.
- multiple-pools optimization deletes don't need to hold
  read locks verifying objects across namespace and pools.
2021-08-17 07:50:00 -07:00
Harshavardhana
654a6e9871
always set the filter to skip navigating baseDir (#12984)
baseDir is empty if the top level prefix does not
end with `/` this causes large recursive listings
without any filtering, to fix this filtering make
sure to set the filter prefix appropriately.

also do not navigate folders at top level that do
not match the filter prefix, entries don't need
to match prefix since they are never prefixed
with the prefix anyways.
2021-08-17 07:43:24 -07:00
Klaus Post
ad928f0078
Return list request when canceled (#12977)
* Return list request when canceled
* Cancel list if abandoned
2021-08-16 11:59:16 -07:00
Klaus Post
92bb2928e4
Compress better on amd64 (#12974)
Since S2 has amd64 assembly, it now operates at a reasonable 
speed to use by default.

Here are some examples of stream compression speed, 16 cores:
```
nyc-taxi-data-10M.csv	s2	1	3325605752	-> 1095998837	312ms	10139.07MB/s		67.04% reduction
nyc-taxi-data-10M.csv	s2	2	3325605752	-> 917905514	428ms	7393.74MB/s		72.40%

github-june-2days-2019.json	s2	1	6273951764	-> 1043196283	391ms	15301.99 MB/s		83.37%
github-june-2days-2019.json	s2	2	6273951764	-> 955924506	519ms	11510.81MB/s		84.76%

github-ranks-backup.bin	s2	1	1862623243	-> 623911363	146ms	12133MB/s		66.50%
github-ranks-backup.bin	s2	2	1862623243	-> 563752759	230ms	7705.26MB/s		69.73%
```

We keep non-assembly platforms on the faster, but less efficient mode.
2021-08-16 11:55:07 -07:00
Anis Elleuch
47dfc1b1b0
ldap: Reevalute filter when searching for non eligible users (#12953)
The previous code removes SVC/STS accounts for ldap users that do not
exist anymore in LDAP server. This commit will actually re-evaluate
filter as well if it is changed and remove all local SVC/STS accounts
beloning to the ldap user if the latter is not eligible for the
search filter anymore.

For example: the filter selects enabled users among other criteras in
the LDAP database, if one ldap user changes his status to disabled
later, then associated SVC/STS accounts will be removed because that user
does not meet the filter search anymore.
2021-08-13 11:40:04 -07:00
Klaus Post
7d8413a589
Reuse more metadata buffers (#12955)
Reuse metadata buffers when no longer referenced.

Takes care of most of the happy paths.
2021-08-13 11:39:27 -07:00
Klaus Post
24722ddd02
Remove inline data hack (#12946)
move the code down to the storage layer,
this logic decouples the inline data from the 
size parameter making it flexible and future
proof.
2021-08-13 08:25:54 -07:00
Klaus Post
f31a00de01
fix: http stats race in traffic metering (#12956)
Traffic metering was not protected against concurrent updates.

```
WARNING: DATA RACE
Read at 0x00c02b0dace8 by goroutine 235:
  github.com/minio/minio/cmd.setHTTPStatsHandler.func1()
      d:/minio/minio/cmd/generic-handlers.go:360 +0x27d
  net/http.HandlerFunc.ServeHTTP()
...

Previous write at 0x00c02b0dace8 by goroutine 994:
  github.com/minio/minio/internal/http/stats.(*IncomingTrafficMeter).Read()
      d:/minio/minio/internal/http/stats/http-traffic-recorder.go:34 +0xd2

```
2021-08-13 07:30:03 -07:00
Shireesh Anjal
d44e4399e6
Add admin api to return sys services info (#12939)
The intention is to provide status of any sys services that can
potentially impact the performance of minio.

At present, it will return information about the `selinux` service
(not-installed/disabled/permissive/enforcing)

Signed-off-by: Shireesh Anjal <shireesh@minio.io>
2021-08-12 18:58:40 -07:00
Harshavardhana
f9ae71fd17
fix: deleteMultiObjects performance regression (#12951)
fixes performance regression found in deleteObjects(),
putObject(), copyObject and completeMultipart calls.
2021-08-12 18:57:37 -07:00
Harshavardhana
ce28e904c9 pass the current credentials for claims 2021-08-12 18:24:04 -07:00
Harshavardhana
8f2a3efa85
disallow sub-credentials based on root credentials to gain priviledges (#12947)
This happens because of a change added where any sub-credential
with parentUser == rootCredential i.e (MINIO_ROOT_USER) will
always be an owner, you cannot generate credentials with lower
session policy to restrict their access.

This doesn't affect user service accounts created with regular
users, LDAP or OpenID
2021-08-12 18:07:08 -07:00
Klaus Post
89febdb3d6
Reuse small buffers (#12948)
When reading metadata allow reuse of buffers 
in certain cases. Take the low-hanging fruit.

Reduce GC overhead when listing.
2021-08-12 14:27:22 -07:00
Klaus Post
3eac02f676
Use metadata reader in ReadVersion (#12942)
Use `readMetadata` when reading version 
information without data requested. 

Reduces IO on inlined data.

Bonus: Inline compressed data as well when 
compression is enabled.
2021-08-12 10:05:24 -07:00
Krishnan Parthasarathi
65b6f4aa31
Add dynamic reconfiguration of number of transition workers (#12926) 2021-08-11 22:23:56 -07:00
Harshavardhana
9e88941515
fix: skip disks that are offline when healing the drives (#12931) 2021-08-11 12:57:18 -07:00
Harshavardhana
40a2fa8e81
fix: add more optimizations to putMetacacheObject() (#12916)
- avoid extra lookup for 'xl.meta' since we are
  definitely sure that it doesn't exist.

- use this in newMultipartUpload() as well

- also additionally do not write with O_DSYNC
  to avoid loading the drives, instead create
  'xl.meta' for listing operations without
  O_DSYNC since these are ephemeral objects.

- do the same with newMultipartUpload() since
  it gets synced when the PutObjectPart() is
  attempted, we do not need to tax newMultipartUpload()
  instead.
2021-08-10 11:12:22 -07:00
Aditya Manthramurthy
59bb54ed6a
Use common function for authenticating admin requests (#12915) 2021-08-09 18:14:38 -07:00
Klaus Post
9ab5e0312d
Simplify gzhttp wrapper (#12912)
The wrapper now accepts interfaces so we don't need a wrapper for that any more.
2021-08-09 12:45:59 -07:00
Harshavardhana
54ab3a1d5b
implement putMetacacheObject() optimizing List operations (#12903)
removes unexpected features from regular putObject() such as

- increasing parity when disks are down, avoids
  a lot of DiskInfo() calls.

- triggering MRF for metacache objects
  if disks are offline

- avoiding renames from temporary location
  to actual namespace, not needed since
  metacache files are unique.
2021-08-09 06:58:54 -07:00
Klaus Post
92c94011f1
Skip downed interfaces on Windows (#12910)
Disregard interfaces that are down when selecting bind addresses

Windows often has a number of disabled NICs used for VPN and other services.

This often causes minio to select an address for contacting the console that is on a disabled (virtual) NIC.

This checks if the interface is up before adding it to the pool on Windows.
2021-08-09 06:57:54 -07:00
Anis Elleuch
35cbe43b6d
Start gateway when KMS is enabled and encryption is unsupported (#12808)
Before, the gateway will complain that it found KMS configured in the
environment but the gateway mode does not support encryption. This
commit will allow starting of the gateway but ensure that S3 operations
with encryption headers will fail when the gateway doesn't support
encryption. That way, the user can use etcd + KMS and have IAM data
encrypted in the etcd store.

Co-authored-by: Anis Elleuch <anis@min.io>
2021-08-08 12:51:48 -07:00
Harshavardhana
a2cd3c9a1d
use ParseForm() to allow query param lookups once (#12900)
```
cpu: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
BenchmarkURLQueryForm
BenchmarkURLQueryForm-4         247099363                4.809 ns/op           0 B/op          0 allocs/op
BenchmarkURLQuery
BenchmarkURLQuery-4              2517624               462.1 ns/op           432 B/op          4 allocs/op
PASS
ok      github.com/minio/minio/cmd      3.848s
```
2021-08-07 22:43:01 -07:00
Harshavardhana
6c0757eea6
fix: simplify APIEndpoints() usage (#12893)
improvements include

- skip IPv6 correctly
- do not set default value for
  MINIO_SERVER_URL, let it be
  configured if not use local IPs

Bonus:

- In healing return error from listPathRaw()
- update console to v0.8.3
2021-08-05 15:01:19 -07:00
Harshavardhana
b10f823907
allow diskFillFraction to be 99% (#12879)
larger 4-8TiB sized disks would return
error prematurely even with sufficient
amount of disk space left, increase
diskFillFraction to 1%
2021-08-04 15:23:39 -07:00
Harshavardhana
4d8f81a992
remove replicate set fallback behavior (#12865)
replica set deployment style is an incorrect
style of distributed MinIO deployment, since
statefulsets have been around for many releases
in k8s now.
2021-08-04 15:23:18 -07:00
Klaus Post
cc60d66909
Fix incremental usage accounting (#12871)
Remote caches were not returned correctly, so they would not get updated on save.

Furthermore make some tweaks for more reliable updates.

Invalidate bloom filter to ensure rescan.
2021-08-04 09:14:14 -07:00
Krishnan Parthasarathi
4197870287
Storage-class headers should depend only on ObjectInfo state (#12864) 2021-08-03 17:35:52 -07:00
Harshavardhana
035882d292
fix: remove parentIsObject() check (#12851)
we will allow situations such as

```
a/b/1.txt
a/b
```

and

```
a/b
a/b/1.txt
```

we are going to document that this usecase is
not supported and we will never support it, if
any application does this users have to delete
the top level parent to make sure namespace is
accessible at lower level.

rest of the situations where the prefixes get
created across sets are supported as is.
2021-08-03 13:26:57 -07:00
Harshavardhana
9371852c7d
fix: getAPIEndpoints() should return public_url (#12852)
fixes #12850
2021-08-02 21:50:20 -07:00
Krishnan Parthasarathi
0a62ae4e61
Revert ignoring inlined objects for transition (#12843) 2021-07-30 16:45:17 -07:00
Harshavardhana
a51799d9f0
feat: Add support for audit notifications for transition (#12842)
This PR adds audit notifications for transitioning objects,
similar to audit logging for expiration and replication
traffic.
2021-07-30 12:45:25 -07:00
Harshavardhana
e7baf2d7d2 stop all console logging 2021-07-29 23:06:05 -07:00
Shireesh Anjal
56097dfca8
Include system errors in health report (#12776)
Some examples of system errors:

- audit is enabled
- updatedb is installed
2021-07-29 23:05:34 -07:00
Harshavardhana
d004828b20 fix: speedtest Endpoint formatting 2021-07-29 17:51:30 -07:00
Harshavardhana
3c34e18a4e
allow multipart uploads for single part multipart (#12821)
its possible that some multipart uploads would have
uploaded only single parts so relying on `len(o.Parts)`
alone is not sufficient, we need to look for ETag
pattern to be absolutely sure.
2021-07-28 22:11:55 -07:00
Poorna Krishnamoorthy
b6cd54779c
Increase context timeout for bandwidth throttled reader (#12820)
increase default timeout up to one hour for toy setups.

fixes #12812
2021-07-28 15:20:01 -07:00
Harshavardhana
bfbdb8f0a8
fix: incorrect O_DIRECT behavior for reads (#12811)
O_DIRECT behavior was broken and it was still
caching all the reads, this change properly fixes
this behavior.
2021-07-28 11:20:16 -07:00