Commit Graph

7425 Commits

Author SHA1 Message Date
Harshavardhana 95411228db
add missing cleanupStaleMultipartUploads (#10325)
fixes #10319
2020-08-21 21:39:54 -07:00
ebozduman 23774353b7
get_object() returns NoSuchKey error when object is a prefix (#10315) 2020-08-21 13:08:01 -07:00
Tobias Nygren 052b5262ff
use statvfs(2) for disk.GetInfo on NetBSD (#10257) 2020-08-20 20:13:06 -07:00
poornas a2a5ec93d3
fix: use global context for filling cache in the background (#10308) 2020-08-20 14:23:24 -07:00
飞雪无情 331c517a5b
translate bucket versioning design document to zh_CN (#10300) 2020-08-20 13:18:51 -07:00
Harshavardhana 27a774cbe9
fix: FS mode should reject putBucketVersioning (#10307) 2020-08-20 13:18:06 -07:00
Klaus Post 8e6787a302
Fix TestDataUpdateTracker hanging (#10302)
Keep dataUpdateTracker while goroutine is starting.

This will ensure the object is updated one `start` returns

Tested with

```
λ go test -cpu=1,2,4,8 -test.run TestDataUpdateTracker -count=1000
PASS
ok      github.com/minio/minio/cmd      8.913s
```

Fixes #10295
2020-08-20 13:17:42 -07:00
Harshavardhana 59352d0ac2
load all blocking metadata in background (#10298)
most of this metadata already has fallbacks
and there is no good reason to load them
in blocking fashion
2020-08-20 10:38:53 -07:00
Harshavardhana 75d44b3bae
add disk for more context in bitrot errors (#10296) 2020-08-20 09:41:15 -07:00
Klaus Post 95ae6c4b49
Fix missing unlock in *healSequence.hasEnded() (#10305)
The background healing sequence would always hang when this function is called.
2020-08-20 08:48:09 -07:00
Harshavardhana 98ca770f81
mint fixes for go1.14 (#10294)
add go.mod to ensure that mint can be
built with go1.14.x
2020-08-19 20:07:37 -07:00
Krishnan Parthasarathi ccd967e3be
Add ExpiresAt to LicenseInfo (#10293) 2020-08-19 19:21:04 -07:00
KevinSmile 0ebb73ee2e
use const instead of literals (#10292) 2020-08-19 16:43:52 -07:00
Harshavardhana c8b84a0e9e
Add nancy vulnerability scanner (#10289) 2020-08-19 14:25:21 -07:00
Ritesh H Shukla 3acb5cff45
Update code comment (#10287) 2020-08-19 14:24:58 -07:00
Harshavardhana ab801ad3d4
build edge containers with go1.14 (#10279) 2020-08-19 12:08:11 +05:30
Harshavardhana 74116204ce
handle fresh setup with mixed drives (#10273)
fresh drive setups when one of the drive is
a root drive, we should ignore such a root
drive and not proceed to format.

This PR handles this properly by marking
the disks which are root disk and they are
taken offline.
2020-08-18 14:37:26 -07:00
飞雪无情 2eb5f934d8
add bucket versioning zh_CN document (#10281) 2020-08-18 13:10:26 -07:00
Minio Trusted b43d376a87 Update yaml files to latest version RELEASE.2020-08-18T19-41-00Z 2020-08-18 20:04:22 +00:00
Harshavardhana e4a44f6224
fix: commonPrefixes behavior in ListObjectVersions (#10286)
```
$ aws s3api --profile minio --endpoint-url http://localhost:9003 \
    list-object-versions --bucket testbucket \
    --delimiter / --prefix Veeam/Archive/

{
    "CommonPrefixes": [
        {
            "Prefix": "Veeam/Archive/003/"
        }
    ]
}
```

Also add coverage tests similar to ListObjects to
catch errors in future, skip these tests in FS mode
2020-08-18 12:19:44 -07:00
poornas 0272973175
Fix regression in web ui for retention (#10285)
Fixes: #10283 regression from PR #9259
2020-08-18 12:09:42 -07:00
Klaus Post adca28801d
feat: disable Parquet by default (breaking change) (#9920)
I have built a fuzz test and it crashes heavily in seconds and will OOM shortly after.
It seems like supporting Parquet is basically a completely open way to crash the 
server if you can upload a file and run s3 select on it.

Until Parquet is more hardened it is DISABLED by default since hostile 
crafted input can easily crash the server.

If you are in a controlled environment where it is safe to assume no hostile
content can be uploaded to your cluster you can safely enable Parquet.

To enable Parquet set the environment variable `MINIO_API_SELECT_PARQUET=on`
while starting the MinIO server.

Furthermore, we guard parquet by recover functions.
2020-08-18 10:23:28 -07:00
Harshavardhana d2a3f92452
fix: health handler for lockers (#10280) 2020-08-18 07:27:41 -07:00
Harshavardhana ede86845e5
docs: Add policy variables for resource and conditions (#10278)
Bonus fix adds LDAP policy variable and clarifies the
usage of policy variables for temporary credentials.

fixes #10197
2020-08-17 17:39:55 -07:00
Harshavardhana e57c742674
use single dynamic timeout for most locked API/heal ops (#10275)
newDynamicTimeout should be allocated once, in-case
of temporary locks in config and IAM we should
have allocated timeout once before the `for loop`

This PR doesn't fix any issue as such, but provides
enough dynamism for the timeout as per expectation.
2020-08-17 11:29:58 -07:00
Klaus Post bb5976d727
healbucket: Send object version ID (#10263)
Based on our previous conversations I assume we should send the version
 id when healing an object.

Maybe we should even list object versions and heal all?
2020-08-17 08:25:44 -07:00
Minio Trusted 670724184c Update yaml files to latest version RELEASE.2020-08-16T18-39-38Z 2020-08-16 18:56:41 +00:00
Harshavardhana f7c1a59de1
add validation logs for configured Logger/Audit HTTP targets (#10274)
extra logs in-case of misconfiguration of audit/logger targets
2020-08-16 10:25:00 -07:00
飞雪无情 01a2ccc52f
add bucket retention zh_CN document (#10264) 2020-08-14 22:12:03 -07:00
Anis Elleuch 51ba1dac49
listing: Fix result when prefix is an object with a slash (#10267)
In a non recursive mode, issuing a list request where prefix
is an existing object with a slash and delimiter is a slash will
return entries in the object directory (data dir IDs)

```
$ aws s3api --profile minioadmin --endpoint-url http://localhost:9000 \
        list-objects-v2 --bucket testbucket --prefix code_of_conduct.md/ --delimiter '/'
{
    "CommonPrefixes": [
        {
            "Prefix":
"code_of_conduct.md/ec750fe0-ea7e-4b87-bbec-1e32407e5e47/"
        }
    ]
}
```

This commit adds a fast exit track in Walk() in this specific case.
2020-08-14 20:13:24 -07:00
Harshavardhana a4463dd40f
fix: storageClass shouldn't set the value upon failure (#10271) 2020-08-14 19:48:04 -07:00
Harshavardhana 83a82d818e
allow lock tolerance to match storage-class drive tolerance (#10270) 2020-08-14 18:17:14 -07:00
Harshavardhana 1d1c4430b2
decrypt ETags in parallel around 500 at a time (#10261)
Listing speed-up gained from 10secs for
just 400 entries to 2secs for 400 entries
2020-08-14 11:56:35 -07:00
Krishnan Parthasarathi 4e00b47b52
licverifier: fail verify if accountId is missing in license metadata (#10258) 2020-08-13 17:05:24 -07:00
Harshavardhana 43e6d1ce2d
fix: missing proxy request by bucket for ListVersions (#10260) 2020-08-13 16:31:58 -07:00
Harshavardhana 30da442a85
rootDisk on containers can have different device Id (#10259)
use `/etc/hosts` instead of `/` to check for common
device id, if the device is same for `/etc/hosts`
and the --bind mount to detect root disks.

Bonus enhance healthcheck logging by adding maintenance
tags, for all messages.
2020-08-13 15:21:20 -07:00
Harshavardhana 038d91feaa
fix: add public certs automatically as part of global CAs (#10256) 2020-08-13 09:46:50 -07:00
Harshavardhana e7ba78beee
use GlobalContext instead of context.Background when possible (#10254) 2020-08-13 09:16:01 -07:00
Krishnan Parthasarathi ab43804efd
licverifier: Validate JWT token expiry (#10253)
With this change the expiry is validated for the license key JWT
2020-08-12 21:31:52 -07:00
Minio Trusted 1c865dd119 Update yaml files to latest version RELEASE.2020-08-13T02-39-50Z 2020-08-13 02:57:25 +00:00
Harshavardhana b32d0a5b60 use the correct endpoints for offline drives 2020-08-12 19:17:49 -07:00
poornas 79e21601b0
fix: web handlers to enforce replication (#10249)
This PR also preserves source ETag for replication
2020-08-12 17:32:24 -07:00
Harshavardhana 34253aa595
feat: cache env value in-case network is not reachable (#10251) 2020-08-12 16:53:15 -07:00
Harshavardhana 79ed7ce451
fs: listObjects shouldn't take FS locks while listing (#10248) 2020-08-12 15:23:14 +05:30
Harshavardhana 900eebb9a4
use jwt instead of basicAuth for webEnv (#10246) 2020-08-11 16:09:34 -07:00
飞雪无情 6914b2c99d
Add bucket replication zh_CN document (#10243) 2020-08-11 11:33:01 -07:00
Harshavardhana 0dd3a08169
move the certPool loader function into pkg/certs (#10239) 2020-08-11 08:29:50 -07:00
Klaus Post f8f290e848
security: Remove insecure custom headers (#10244)
Background: https://github.com/google/security-research/security/advisories/GHSA-76wf-9vgp-pj7w

Remove these custom headers from incoming and outgoing requests.
2020-08-11 08:29:29 -07:00
Harshavardhana 9179cdfc9d
update mint tests with new minio-py APIs (#10238) 2020-08-10 14:32:17 -07:00
Krishnan Parthasarathi 76b6dc0112
Add licverifier package (#10237)
license verification package implements a simple library to 
verify MinIO Subnet license keys.
2020-08-10 13:30:12 -07:00