MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,987 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

9987 Commits

Author SHA1 Message Date
Harshavardhana 8144a125ce
check for update in background (#13889) 2021-12-13 09:43:03 -08:00
Minio Trusted 3e34e41a5a update helm to 3.3.4 2021-12-11 09:59:34 -08:00
Minio Trusted 2270887d43 update helm to 3.3.3 2021-12-11 09:28:02 -08:00
Minio Trusted c6431f9a04 update helm to 3.3.2 
resolves #13887
 2021-12-11 09:26:01 -08:00
jiangfucheng 88c0d0120c
update heal object unit test (#13886) 2021-12-11 09:04:07 -08:00
Aditya Manthramurthy 44fefe5b9f
Add option to policy info API to return create/mod timestamps (#13796) 
- This introduces a new admin API with a query parameter (v=2) to return a
response with the timestamps

- Older API still works for compatibility/smooth transition in console
 2021-12-11 09:03:39 -08:00
Minio Trusted 878d368cea Update yaml files to latest version RELEASE.2021-12-10T23-03-39Z 2021-12-11 00:02:59 -08:00
Aditya Manthramurthy f2bd026d0e
Allow OIDC user to query user info if policies permit (#13882) 2021-12-10 15:03:39 -08:00
Klaus Post 518612492c
xl-meta: Add header titles (#13880) 
Add type for headers and create custom marshal to make 
it easier to read. Group headers and metadata.

Restore functionality that will read `xl.meta` in the current dir with no params.

Before:
```
{
  "Headers": [
    [
      "8M04bTiYRDmEMQGeAsk1yg==",
      1639150471630100400,
      "rLD1Rw==",
      1,
      6
    ],
  ],
    "Versions": [
    {
      "Type": 1,
      "V2Obj": {
        "CSumAlgo": 1,
        "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
        "EcAlgo": 1,
        "EcBSize": 1048576,
        "EcDist": [
          7,
          8,
          1,
          2,
          3,
          4,
          5,
          6
        ],
        "EcIndex": 1,
        "EcM": 4,
        "EcN": 4,
        "ID": "8M04bTiYRDmEMQGeAsk1yg==",
        "MTime": 1639150471630100400,
        "MetaSys": {
          "x-minio-internal-inline-data": "dHJ1ZQ=="
        },
        "MetaUsr": {
          "content-type": "application/octet-stream",
          "etag": "b8252c86fad2d8937300aa92b467a3aa"
        },
        "PartASizes": [
          1000
        ],
        "PartETags": null,
        "PartNums": [
          1
        ],
        "PartSizes": [
          1000
        ],
        "Size": 1000
      }
    }
  ]
}
```

After:
```
{
  "Versions": [
    {
      "Header": {
        "Flags": 6,
        "ModTime": "2021-12-10T16:34:31.6301004+01:00",
        "Signature": "acb0f547",
        "Type": 1,
        "VersionID": "f0cd386d389844398431019e02c935ca"
      },
      "Idx": 0,
      "Metadata": {
        "Type": 1,
        "V2Obj": {
          "CSumAlgo": 1,
          "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
          "EcAlgo": 1,
          "EcBSize": 1048576,
          "EcDist": [
            7,
            8,
            1,
            2,
            3,
            4,
            5,
            6
          ],
          "EcIndex": 1,
          "EcM": 4,
          "EcN": 4,
          "ID": "8M04bTiYRDmEMQGeAsk1yg==",
          "MTime": 1639150471630100400,
          "MetaSys": {
            "x-minio-internal-inline-data": "dHJ1ZQ=="
          },
          "MetaUsr": {
            "content-type": "application/octet-stream",
            "etag": "b8252c86fad2d8937300aa92b467a3aa"
          },
          "PartASizes": [
            1000
          ],
          "PartETags": null,
          "PartNums": [
            1
          ],
          "PartSizes": [
            1000
          ],
          "Size": 1000
        }
      }
    }
  ]
}
```
 2021-12-10 15:03:25 -08:00
Klaus Post 81e43b87c2
Don't zero buffer if big enough (#13877) 
Only append zeroed bytes when we don't have enough space anyway.
 2021-12-10 13:08:10 -08:00
Aditya Manthramurthy a02e17f15c
Add tests to ensure that OIDC user can create IAM users (#13881) 2021-12-10 13:04:21 -08:00
Ravind Kumar c76f86fdbd
Clarify example for Standalone Docker instructions (#13879) 
Closes #13868 

Also points users to the web documentation for docker installation.
 2021-12-10 09:43:02 -08:00
Harshavardhana 5b7c00ff52
add more tests to cover areas for weird object names (#13873) 
continuation of #13858 to add more tests and also validate the 
written object data.
 2021-12-09 17:52:53 -08:00
Aditya Manthramurthy b9f0046ee7
Allow STS credentials to create users (#13874) 
- allow any regular user to change their own password
- allow STS credentials to create users if permissions allow

Bonus: do not allow changes to sts/service account credentials (via add user API)
 2021-12-09 17:48:51 -08:00
Harshavardhana 3b79f7e4ae
ignore if volume exists in MakeVolBulk, return other errors (#13866) 2021-12-09 15:55:42 -08:00
Aditya Manthramurthy 85d2df02b9
fix: user listing with LDAP (#13872) 
Users listing was showing just a weird policy 
mapping output which does not make sense here.
 2021-12-09 15:55:28 -08:00
Harshavardhana 2f1e8ba612
add more directory marker tests and fix a bug (#13871) 
ListObjects() should never list a delete-marked folder
if latest is delete marker and delimiter is not provided.

ListObjectVersions() should list a delete-marked folder
even if latest is delete marker and delimiter is not
provided.

Enhance further versioning listing on the buckets
 2021-12-09 14:59:23 -08:00
Anis Elleuch 84c690cb07
storage: Use request.Form and avoid mux matching (#13858) 
request.Form uses less memory allocation and avoids gorilla mux matching
with weird characters in parameters such as '\n'

- Remove Queries() to avoid matching
- Ensure r.ParseForm is called to populate fields
- Add a unit test for object names with '\n'
 2021-12-09 08:38:46 -08:00
Harshavardhana 239bbad7ab
add test to expect prefix without a directory object (#13865) 
Motivation is to cover more areas
 2021-12-09 08:36:54 -08:00
Minio Trusted 4be8023408 Update yaml files to latest version RELEASE.2021-12-09T06-19-41Z 2021-12-09 08:40:46 +00:00
Harshavardhana 83e8da57b8 update console to release v0.12.8 2021-12-08 22:19:41 -08:00
Harshavardhana dcff6c996d
fix: do not list delete-marked objects (#13864) 
delete marked objects should not be considered
for listing when listing is delimited, this issue
as introduced in PR #13804 which was mainly to
address listing of directories in listing when
delimited.

This PR fixes this properly and adds tests to
ensure that we behave in accordance with how
an S3 API behaves for ListObjects() without
versions.
 2021-12-08 17:34:52 -08:00
Poorna K 0a66a6f1e5
Avoid cache GC of writebacks before commit syncs (#13860) 
Save part.1 for writebacks in a separate folder
and move it to cache dir atomically while saving
the cache metadata. This is to avoid GC mistaking
part.1 as orphaned cache entries and purging them.

This PR also fixes object size being overwritten during
retries for write-back mode.
 2021-12-08 14:52:31 -08:00
Harshavardhana e82a5c5c54
fix: site replication issues and add tests (#13861) 
- deleting policies was deleting all LDAP
  user mapping, this was a regression introduced
  in #13567

- deleting of policies is properly sent across
  all sites.

- remove unexpected errors instead embed the real
  errors as part of the 500 error response.
 2021-12-08 11:50:15 -08:00
Harshavardhana 92fdcafb66
add verification tests for ETag on replicated content (#13857) 2021-12-07 10:08:26 -08:00
Harshavardhana b9aae1aaae
fix: speedtest should exit upon errors cleanly (#13851) 
- deleteBucket() should be called for cleanup
  if client abruptly disconnects

- out of disk errors should be sent to client
  properly and also cancel the calls

- limit concurrency to available MAXPROCS not
  32 for auto-tuned setup, if procs are beyond
  32 then continue normally. this is to handle
  smaller setups.

fixes #13834
 2021-12-06 16:36:14 -08:00
Harshavardhana 7d70afc937
fix: potential crash in diskCache when fileScorer is empty (#13850) 
```
goroutine 115 [running]:
github.com/minio/minio/cmd.(*diskCache).purge.func3({0xc007a10a40, 0x40}, 0x40)
   github.com/minio/minio/cmd/disk-cache-backend.go:430 +0x90d
```
 2021-12-06 15:55:29 -08:00
Aditya Manthramurthy 12b63061c2
Fix LDAP service account creation (#13849) 
- when a user has only group permissions
- fixes regression from ac74237f0 (#13657)
- fixes https://github.com/minio/console/issues/1291
 2021-12-06 15:55:11 -08:00
Klaus Post 038fdeea83
snowball: return errors on failures (#13836) 
Return errors when untar fails at once.

Current error handling was quite a mess. Errors are written 
to the stream, but processing continues.

Instead, return errors when they occur and transform 
internal errors to bad request errors, since it is likely a 
problem with the input.

Fixes #13832
 2021-12-06 09:45:23 -08:00
Anis Elleuch 0b6225bcc3
Better error msg when version mismatch of internode API (#13845) 
Sometimes, we see an error message like "Server expects 'storage' API
version 'v41', instead found 'v41'" shows a more generic error message
with the path of the REST call.
 2021-12-06 09:44:48 -08:00
Anis Elleuch f286ef8e17
isMultipart to test on parts sizes only if object is encrypted (#13839) 
ObjectInfo.isMultipart() is testing if parts sizes are compatible with
encrypted parts but this only can be done if the object is encrypted.
 2021-12-06 09:43:43 -08:00
Harshavardhana b120bcb60a
validate if cached value is empty before use (#13830) 
fixes a crash reproduced while running hadoop tests

```
goroutine 201564 [running]:
github.com/minio/minio/cmd.metaCacheEntries.resolve({0xc0206ab7a0, 0x4, 0xc0015b1908}, 0xc0212a7040)
	github.com/minio/minio/cmd/metacache-entries.go:352 +0x58a
```

Bonus: HeadBucket() should always provide content-type
 2021-12-06 02:59:51 -08:00
Harshavardhana be34fc9134
fix: kms-id header should have arn:aws:kms: prefix (#13833) 
arn:aws:kms: is a must for KMS keyID.
 2021-12-06 00:39:32 -08:00
Harshavardhana 8591d17d82
return appropriate errors upon parseErrors (#13831) 2021-12-05 11:36:26 -08:00
Harshavardhana f6190d6751
Add single drive support for directory prefixes in Listing (#13829) 
This fixes the compatibility issue with Hadoop 3.3.1

fixes #13710
 2021-12-03 18:08:40 -08:00
Harshavardhana f0fc77fded update CREDITS file with new deps 2021-12-03 13:24:49 -08:00
Klaus Post f56cac6381
jwt: Parse standard claims faster (#13821) 
* Use structless/allocationless decoding for header (note "typ" isn't used)
* Create custom unmarshal code using jsonparser for StandardClaims.

Before/After:

```
BenchmarkParseJWTStandardClaims-32    	 4270724	       294.0 ns/op	     706 B/op	      16 allocs/op
BenchmarkParseJWTStandardClaims-32    	 5634847	       214.7 ns/op	     544 B/op	       9 allocs/op

BenchmarkParseJWTMapClaims-32    	 2763045	       428.6 ns/op	    1251 B/op	      29 allocs/op
BenchmarkParseJWTMapClaims-32    	 2839455	       410.9 ns/op	    1219 B/op	      26 allocs/op
```
 2021-12-03 13:19:38 -08:00
Aditya Manthramurthy 4f35054d29
Ensure that role ARNs don't collide (#13817) 
This is to prepare for multiple providers enhancement.
 2021-12-03 13:15:56 -08:00
Shireesh Anjal d29df6714a
Introduce new config `subnet api_key` (#13793) 
The earlier approach of using a license token for 
communicating with SUBNET is being replaced 
with a simpler mechanism of API keys. Unlike the 
license which is a JWT token, these API keys will 
be simple UUID tokens and don't have any embedded 
information in them. SUBNET would generate the 
API key on cluster registration, and then it would 
be saved in this config, to be used for subsequent 
communication with SUBNET.
 2021-12-03 09:32:11 -08:00
jiangfucheng 7460fb8349
fix padding error and compatible with uploaded objects (#13803) 2021-12-03 09:26:30 -08:00
Harshavardhana a7c430355a
fix: throw appropriate errors when all disks fail (#13820) 
when all disks fail with same error, fail server
startup anyways - we cannot proceed.

fixes #13818
 2021-12-03 09:25:17 -08:00
Harshavardhana 1df1517449 Add missing Dockerfile.dev 2021-12-03 00:56:40 -08:00
Harshavardhana f7c357ebad update console to v0.12.6 2021-12-02 17:54:29 -08:00
Harshavardhana 20c60aae68 Update hotfix documentation and container building 2021-12-02 17:52:46 -08:00
Aditya Manthramurthy b14527b7af
If role policy is configured, require that role ARN be set in STS (#13814) 2021-12-02 15:43:39 -08:00
Harshavardhana f840080e5b
cleanup site-replication docs (#13812) 2021-12-02 13:27:01 -08:00
Harshavardhana 2c6983a2f1
fix: use consistent ports in verify-healing (#13813) 
also use unique directories in setup testing.
 2021-12-02 12:40:48 -08:00
Harshavardhana acfb83ec5e update to v3.3.1 - helm chart docs Minio -> MinIO 
fixes #13775
 2021-12-02 12:09:18 -08:00
Klaus Post 3db931dc0e
Improve listing consistency with version merging (#13723) 2021-12-02 11:29:16 -08:00
Klaus Post 8309ddd486
Fix panic (not fatal) on connection drops (#13811) 
Fix more regressions from #13597 with double closed channels.

```
panic: "POST /minio/storage/data/distxl-plain/s1/d2/v42/createfile?disk-id=c789f7e1-2b52-442a-b518-aa2dac03f3a1&file-path=f6161668-b939-4543-9873-91b9da4cdff6%2F5eafa986-a3bf-4b1c-8bc0-03a37de390a3%2Fpart.1&length=2621760&volume=.minio.sys%2Ftmp": send on closed channel
goroutine 1977 [running]:
runtime/debug.Stack()
        c:/go/src/runtime/debug/stack.go:24 +0x65
github.com/minio/minio/cmd.setCriticalErrorHandler.func1.1()
        d:/minio/minio/cmd/generic-handlers.go:468 +0x8e
panic({0x2928860, 0x4fb17e0})
        c:/go/src/runtime/panic.go:1038 +0x215
github.com/minio/minio/cmd.keepHTTPReqResponseAlive.func2({0x4fe4ea0, 0xc02737d8a0})
        d:/minio/minio/cmd/storage-rest-server.go:818 +0x48
github.com/minio/minio/cmd.(*storageRESTServer).CreateFileHandler(0xc0015a8510, {0x50073e0, 0xc0273ec460}, 0xc029b9a400)
        d:/minio/minio/cmd/storage-rest-server.go:334 +0x1d2
net/http.HandlerFunc.ServeHTTP(...)
        c:/go/src/net/http/server.go:2046
github.com/minio/minio/cmd.httpTraceHdrs.func1({0x50073e0, 0xc0273ec460}, 0x0)
        d:/minio/minio/cmd/handler-utils.go:372 +0x53
net/http.HandlerFunc.ServeHTTP(0x5007380, {0x50073e0, 0xc0273ec460}, 0x10)
        c:/go/src/net/http/server.go:2046 +0x2f
github.com/minio/minio/cmd.addCustomHeaders.func1({0x5007380, 0xc0273dcf00}, 0xc0273f7340)
```

Reverts but adds write checks.
 2021-12-02 11:22:32 -08:00