Commit Graph

8962 Commits

Author SHA1 Message Date
Harshavardhana b9aae1aaae
fix: speedtest should exit upon errors cleanly (#13851)
- deleteBucket() should be called for cleanup
  if client abruptly disconnects

- out of disk errors should be sent to client
  properly and also cancel the calls

- limit concurrency to available MAXPROCS not
  32 for auto-tuned setup, if procs are beyond
  32 then continue normally. this is to handle
  smaller setups.

fixes #13834
2021-12-06 16:36:14 -08:00
Harshavardhana 7d70afc937
fix: potential crash in diskCache when fileScorer is empty (#13850)
```
goroutine 115 [running]:
github.com/minio/minio/cmd.(*diskCache).purge.func3({0xc007a10a40, 0x40}, 0x40)
   github.com/minio/minio/cmd/disk-cache-backend.go:430 +0x90d
```
2021-12-06 15:55:29 -08:00
Aditya Manthramurthy 12b63061c2
Fix LDAP service account creation (#13849)
- when a user has only group permissions
- fixes regression from ac74237f0 (#13657)
- fixes https://github.com/minio/console/issues/1291
2021-12-06 15:55:11 -08:00
Klaus Post 038fdeea83
snowball: return errors on failures (#13836)
Return errors when untar fails at once.

Current error handling was quite a mess. Errors are written 
to the stream, but processing continues.

Instead, return errors when they occur and transform 
internal errors to bad request errors, since it is likely a 
problem with the input.

Fixes #13832
2021-12-06 09:45:23 -08:00
Anis Elleuch 0b6225bcc3
Better error msg when version mismatch of internode API (#13845)
Sometimes, we see an error message like "Server expects 'storage' API
version 'v41', instead found 'v41'" shows a more generic error message
with the path of the REST call.
2021-12-06 09:44:48 -08:00
Anis Elleuch f286ef8e17
isMultipart to test on parts sizes only if object is encrypted (#13839)
ObjectInfo.isMultipart() is testing if parts sizes are compatible with
encrypted parts but this only can be done if the object is encrypted.
2021-12-06 09:43:43 -08:00
Harshavardhana b120bcb60a
validate if cached value is empty before use (#13830)
fixes a crash reproduced while running hadoop tests

```
goroutine 201564 [running]:
github.com/minio/minio/cmd.metaCacheEntries.resolve({0xc0206ab7a0, 0x4, 0xc0015b1908}, 0xc0212a7040)
	github.com/minio/minio/cmd/metacache-entries.go:352 +0x58a
```

Bonus: HeadBucket() should always provide content-type
2021-12-06 02:59:51 -08:00
Harshavardhana be34fc9134
fix: kms-id header should have arn:aws:kms: prefix (#13833)
arn:aws:kms: is a must for KMS keyID.
2021-12-06 00:39:32 -08:00
Harshavardhana 8591d17d82
return appropriate errors upon parseErrors (#13831) 2021-12-05 11:36:26 -08:00
Harshavardhana f6190d6751
Add single drive support for directory prefixes in Listing (#13829)
This fixes the compatibility issue with Hadoop 3.3.1

fixes #13710
2021-12-03 18:08:40 -08:00
Harshavardhana f0fc77fded update CREDITS file with new deps 2021-12-03 13:24:49 -08:00
Klaus Post f56cac6381
jwt: Parse standard claims faster (#13821)
* Use structless/allocationless decoding for header (note "typ" isn't used)
* Create custom unmarshal code using jsonparser for StandardClaims.

Before/After:

```
BenchmarkParseJWTStandardClaims-32    	 4270724	       294.0 ns/op	     706 B/op	      16 allocs/op
BenchmarkParseJWTStandardClaims-32    	 5634847	       214.7 ns/op	     544 B/op	       9 allocs/op

BenchmarkParseJWTMapClaims-32    	 2763045	       428.6 ns/op	    1251 B/op	      29 allocs/op
BenchmarkParseJWTMapClaims-32    	 2839455	       410.9 ns/op	    1219 B/op	      26 allocs/op
```
2021-12-03 13:19:38 -08:00
Aditya Manthramurthy 4f35054d29
Ensure that role ARNs don't collide (#13817)
This is to prepare for multiple providers enhancement.
2021-12-03 13:15:56 -08:00
Shireesh Anjal d29df6714a
Introduce new config `subnet api_key` (#13793)
The earlier approach of using a license token for 
communicating with SUBNET is being replaced 
with a simpler mechanism of API keys. Unlike the 
license which is a JWT token, these API keys will 
be simple UUID tokens and don't have any embedded 
information in them. SUBNET would generate the 
API key on cluster registration, and then it would 
be saved in this config, to be used for subsequent 
communication with SUBNET.
2021-12-03 09:32:11 -08:00
jiangfucheng 7460fb8349
fix padding error and compatible with uploaded objects (#13803) 2021-12-03 09:26:30 -08:00
Harshavardhana a7c430355a
fix: throw appropriate errors when all disks fail (#13820)
when all disks fail with same error, fail server
startup anyways - we cannot proceed.

fixes #13818
2021-12-03 09:25:17 -08:00
Harshavardhana 1df1517449 Add missing Dockerfile.dev 2021-12-03 00:56:40 -08:00
Harshavardhana f7c357ebad update console to v0.12.6 2021-12-02 17:54:29 -08:00
Harshavardhana 20c60aae68 Update hotfix documentation and container building 2021-12-02 17:52:46 -08:00
Aditya Manthramurthy b14527b7af
If role policy is configured, require that role ARN be set in STS (#13814) 2021-12-02 15:43:39 -08:00
Harshavardhana f840080e5b
cleanup site-replication docs (#13812) 2021-12-02 13:27:01 -08:00
Harshavardhana 2c6983a2f1
fix: use consistent ports in verify-healing (#13813)
also use unique directories in setup testing.
2021-12-02 12:40:48 -08:00
Harshavardhana acfb83ec5e update to v3.3.1 - helm chart docs Minio -> MinIO
fixes #13775
2021-12-02 12:09:18 -08:00
Klaus Post 3db931dc0e
Improve listing consistency with version merging (#13723) 2021-12-02 11:29:16 -08:00
Klaus Post 8309ddd486
Fix panic (not fatal) on connection drops (#13811)
Fix more regressions from #13597 with double closed channels.

```
panic: "POST /minio/storage/data/distxl-plain/s1/d2/v42/createfile?disk-id=c789f7e1-2b52-442a-b518-aa2dac03f3a1&file-path=f6161668-b939-4543-9873-91b9da4cdff6%2F5eafa986-a3bf-4b1c-8bc0-03a37de390a3%2Fpart.1&length=2621760&volume=.minio.sys%2Ftmp": send on closed channel
goroutine 1977 [running]:
runtime/debug.Stack()
        c:/go/src/runtime/debug/stack.go:24 +0x65
github.com/minio/minio/cmd.setCriticalErrorHandler.func1.1()
        d:/minio/minio/cmd/generic-handlers.go:468 +0x8e
panic({0x2928860, 0x4fb17e0})
        c:/go/src/runtime/panic.go:1038 +0x215
github.com/minio/minio/cmd.keepHTTPReqResponseAlive.func2({0x4fe4ea0, 0xc02737d8a0})
        d:/minio/minio/cmd/storage-rest-server.go:818 +0x48
github.com/minio/minio/cmd.(*storageRESTServer).CreateFileHandler(0xc0015a8510, {0x50073e0, 0xc0273ec460}, 0xc029b9a400)
        d:/minio/minio/cmd/storage-rest-server.go:334 +0x1d2
net/http.HandlerFunc.ServeHTTP(...)
        c:/go/src/net/http/server.go:2046
github.com/minio/minio/cmd.httpTraceHdrs.func1({0x50073e0, 0xc0273ec460}, 0x0)
        d:/minio/minio/cmd/handler-utils.go:372 +0x53
net/http.HandlerFunc.ServeHTTP(0x5007380, {0x50073e0, 0xc0273ec460}, 0x10)
        c:/go/src/net/http/server.go:2046 +0x2f
github.com/minio/minio/cmd.addCustomHeaders.func1({0x5007380, 0xc0273dcf00}, 0xc0273f7340)
```

Reverts but adds write checks.
2021-12-02 11:22:32 -08:00
Harshavardhana 21c868a646
fix: do not ignore delete-marker directories in ListObjects() (#13804)
Following scenario such as objects that exist inside a
prefix say `folder/` must be included in the listObjects()
response.

```
2aa16073-387e-492c-9d59-b4b0b7b6997a v2 DEL folder/
a5b9ce68-7239-4921-90ab-20aed402c7a2 v1 PUT folder/
f2211798-0eeb-4d9e-9184-fcfeae27d069 v1 PUT folder/1.txt
```

Current master does not handle this scenario, because it
ignores the top level delete-marker on folders. This is
however unexpected. It is expected that list-objects returns
the top level prefix in this situation.

```
aws s3api list-objects --bucket harshavardhana --prefix unique/ \
     --delimiter / --profile minio --endpoint-url http://localhost:9000
{
    "CommonPrefixes": [
        {
            "Prefix": "unique/folder/"
        }
    ]
}
```

There are applications in the wild such as Hadoop s3a connector
that exploit this behavior and expect the folder to be present
in the response.

This also makes the behavior consistent with AWS S3.
2021-12-02 08:46:33 -08:00
Harshavardhana ffe9acfe4a docs: Add a markdown documentation on hotfix branches and process 2021-12-01 01:00:18 -08:00
Harshavardhana 24d904d194
reload certs from disk upon SIGHUP (#13792) 2021-12-01 00:38:32 -08:00
Harshavardhana b280a37c4d
add delete-marker proactively in DeleteObject() (#13795)
single object delete was not working properly
on a bucket when versioning was suspended,
current version 'null' object was never removed.

added unit tests to cover the behavior

fixes #13783
2021-11-30 18:30:06 -08:00
vinzenzs 906548d0ba
Added the service account to deloyments and statefulset (#13790) 2021-11-30 15:25:30 -08:00
Poorna K 1485a5bf3b
fix: dockerfile.dev to create /opt/bin first (#13794) 2021-11-30 15:24:39 -08:00
Poorna K 9ec197f2e8
Add support for adding new site(s) to site replication (#13696)
Currently, the new site is expected to be empty
2021-11-30 13:16:37 -08:00
Poorna K d21466f595
cache: in writeback mode skip etag verification (#13781)
if the commit is still in pending or failed status

This PR also does some minor code cleanup
2021-11-30 10:22:42 -08:00
Harshavardhana 4f3290309e Revert "disable CI/CD for draft PRs (#13784)"
This reverts commit 5a22f2cf0b.
2021-11-30 09:22:17 -08:00
Klaus Post d6fe0f61a9
do not panic when input cannot be parsed (#13791)
Fix cases where `s3Select.Open` fails and doesn't set the recordReader.

Fixes #13786
2021-11-30 08:42:42 -08:00
Krishnan Parthasarathi 5a22f2cf0b
disable CI/CD for draft PRs (#13784) 2021-11-29 23:35:07 -08:00
Aditya Manthramurthy 42d11d9e7d
Move IAM notifications into IAM system functions (#13780) 2021-11-29 14:38:57 -08:00
Harshavardhana e49c184595
add configurable 'shutdown-timeout' for HTTP server (#13771)
fixes #12317
2021-11-29 09:06:56 -08:00
Harshavardhana 99d87c5ca2
fix: totalDrives reported in speedTest for multiple-pools (#13770)
totalDrives reported in speedTest result were wrong
for multiple pools, this PR fixes this.

Bonus: add support for configurable storage-class, this
allows us to test REDUCED_REDUNDANCY to see further
maximum throughputs across the cluster.
2021-11-29 09:05:46 -08:00
Aditya Manthramurthy 4c0f48c548
Add role ARN support for OIDC identity provider (#13651)
- Allows setting a role policy parameter when configuring OIDC provider

- When role policy is set, the server prints a role ARN usable in STS API requests

- The given role policy is applied to STS API requests when the roleARN parameter is provided.

- Service accounts for role policy are also possible and work as expected.
2021-11-26 19:22:40 -08:00
Aditya Manthramurthy 4ce6d35e30
Add new `site` config sub-system intended to replace `region` (#13672)
- New sub-system has "region" and "name" fields.

- `region` subsystem is marked as deprecated, however still works, unless the
new region parameter under `site` is set - in this case, the region subsystem is
ignored. `region` subsystem is hidden from top-level help (i.e. from `mc admin
config set myminio`), but appears when specifically requested (i.e. with `mc
admin config set myminio region`).

- MINIO_REGION, MINIO_REGION_NAME are supported as legacy environment variables for server region.

- Adds MINIO_SITE_REGION as the current environment variable to configure the
server region and MINIO_SITE_NAME for the site name.
2021-11-25 13:06:25 -08:00
Harshavardhana 81bf0c66c6 update helm chart to 3.3.0 2021-11-25 09:33:26 -08:00
Klaus Post 34dc725d26
fix: s3zip in fs mode (#13758)
The index was converted directly from bytes to binary. This would fail a roundtrip through json.

This would result in `Error: invalid input: magic number mismatch` when reading back.

On non-erasure backends store index as base64.
2021-11-25 09:11:25 -08:00
Minio Trusted a5db4ca092 Update yaml files to latest version RELEASE.2021-11-24T23-19-33Z 2021-11-25 07:39:00 +00:00
Aditya Manthramurthy 61029fe20b
fix: returning invalid account-not-exists error for LDAP svc acc (#13756) 2021-11-24 15:19:33 -08:00
Harshavardhana fee3f88cb5
use acceptedResponseStatusCode everywhere in HTTP logger (#13755) 2021-11-24 13:53:11 -08:00
Harshavardhana 932500e43d update console to v0.12.5 2021-11-24 12:46:53 -08:00
Anis Elleuch 55d4cdd464
multi-delete: Avoid empty Delete tag in the response (#13725)
When removing an object fails, such as when it is WORM protected, a
wrong <Delete> will still be in the response. This commit fixes it.
2021-11-24 10:01:07 -08:00
Klaus Post fe3e47b1e8
Fix "send on closed channel" panic (#13745)
The httpStreamResponse should not return until CloseWithError has been called.

Instead keep track of write state and skip writing/flushing if an error has occurred.

Fixes #13743

Regression from #13597 (not released)
2021-11-24 09:42:42 -08:00
Harshavardhana 9ca25bd48f
fix: atomic.Value should be a concrete type to avoid panics (#13740)
Go's atomic.Value does not support `nil` type,
concrete type is necessary to avoid any panics with
the current implementation.

Also remove boolean to turn-off tracking of freezeCount.
2021-11-23 16:09:28 -08:00