MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,593 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

7593 Commits

Author SHA1 Message Date
Poorna Krishnamoorthy 39f3d5493b
Show Delete replication status header (#10946) 
X-Minio-Replication-Delete-Status header shows the
status of the replication of a permanent delete of a version.

All GETs are disallowed and return 405 on this object version.
In the case of replicating delete markers.

X-Minio-Replication-DeleteMarker-Status shows the status 
of replication, and would similarly return 405.

Additionally, this PR adds reporting of delete marker event completion
and updates documentation
 2020-11-21 23:48:50 -08:00
Shireesh Anjal 14a7ae8586
Remove platform specific structure definitions (#10935) 
Instead of having less/more fields inside a structure depending on the
platform (non-linux/linux), it would be better to have the same standard
definition in all platforms, and certain fields of the structure to be
populated or left unpopulated depending on the platform.
 2020-11-21 09:41:33 -08:00
Klaus Post 692ff41ef7
Unwrap network errors (#10934) 
Alternative to #10927

Instead of having an upstream fix, do unwrap when checking network errors.

'As' will also work when destination is an interface as checked by the tests.
 2020-11-20 22:55:35 -08:00
Harshavardhana 86409fa93d
add audit/admin trace support for browser requests (#10947) 
To support this functionality we had to fork
the gorilla/rpc package with relevant changes
 2020-11-20 22:52:17 -08:00
Shireesh Anjal 7bc47a14cc
Rename OBD to Health (#10842) 
Also, Remove thread stats and openfds from the health report 
as we already have process stats and numfds
 2020-11-20 12:52:53 -08:00
Dominik Lessel 4a31b31ca6
feat(docker): add a CI/CD Dockerfile, which starts the server right away (#10933) 2020-11-20 11:27:43 -08:00
Harshavardhana 9263be8cca
docs: fix missing event types in notifications (#10944) 2020-11-20 11:27:27 -08:00
Harshavardhana 73e308079a
fix: handle errors appropriately as they are wrapped (#10917) 2020-11-20 10:43:07 -08:00
Poorna Krishnamoorthy 08b24620c0 Display storage-class of transitioned object in HEAD 2020-11-20 09:17:31 -08:00
Harshavardhana 95675b0c9a
fix: do not crash PutObjectTags when node is down (#10940) 
fixes #10939
 2020-11-20 09:10:48 -08:00
Poorna Krishnamoorthy 251c1ef6da Add support for replication of object tags, retention metadata (#10880) 2020-11-19 18:56:09 -08:00
Poorna Krishnamoorthy 0fa430c1da validate service type of target in replication/ilm transition config (#10928) 2020-11-19 18:47:33 -08:00
Poorna Krishnamoorthy f60b6eb82e fix validation for deletemarker replication on object locked bucket (#10892) 2020-11-19 18:47:19 -08:00
Poorna Krishnamoorthy 1ebf6f146a Add support for ILM transition (#10565) 
This PR adds transition support for ILM
to transition data to another MinIO target
represented by a storage class ARN. Subsequent
GET or HEAD for that object will be streamed from
the transition tier. If PostRestoreObject API is
invoked, the transitioned object can be restored for
duration specified to the source cluster.
 2020-11-19 18:47:17 -08:00
Harshavardhana 8f7fe0405e fix: delete marker replication should support directories (#10878) 
allow directories to be replicated as well, along with
their delete markers in replication.

Bonus fix to fix bloom filter updates for directories
to be preserved.
 2020-11-19 18:47:12 -08:00
Harshavardhana 9a34fd5c4a Revert "Revert "Add delete marker replication support (#10396)"" 
This reverts commit 267d7bf0a9.
 2020-11-19 18:43:58 -08:00
Minio Trusted b9e3a8b5ac Update yaml files to latest version RELEASE.2020-11-19T23-48-16Z 2020-11-20 00:06:03 +00:00
Harshavardhana f794fe79e3
fix: network shutdown was not handle properly (#10927) 
fixes a regression introduced in #10859, due
to the error returned by rest.Client being typed
i.e *rest.NetworkError - IsNetworkHostDown function
didn't work as expected to detect network issues.

This in-turn aggravated the situations when nodes
are disconnected leading to performance loss.
 2020-11-19 13:53:49 -08:00
Harshavardhana 0f9e125cf3
fix: check for gateway backend online without http request (#10924) 
fixes #10921
 2020-11-19 10:38:02 -08:00
Harshavardhana d778d9493f
remove MinIO release tag as part of HTTP Server string (#10929) 2020-11-19 09:16:02 -08:00
Harshavardhana 70d2c2ccc9
skip files that are not erasure objects or directories (#10926) 
without this change WalkDir reports errors while
trying to read `format.json/xl.meta` which is a
replicated file
 2020-11-19 09:15:09 -08:00
Harshavardhana 9dea7020f0
allow prefix filtering for WalkDir to be optional (#10923) 2020-11-18 12:03:16 -08:00
Klaus Post 990d074f7d
metacache: Allow prefix filtering (#10920) 
Do listings with prefix filter when bloom filter is dirty.

This will forward the prefix filter to the lister which will make it 
only scan the folders/objects with the specified prefix.

If we have a clean bloom filter we try to build a more generally 
useful cache so in that case, we will list all objects/folders.
 2020-11-18 10:44:18 -08:00
Klaus Post e413f05397
Save listing error async (#10922) 
Since the RPC call may have to time out save an error state async 
to not hold up the listing returning.

Fixes #10919
 2020-11-18 10:28:22 -08:00
Harshavardhana d1b1fee080
fix: save healing tracker right before healing (#10915) 
this change avoids a situation where accidentally
if the user deleted the healing tracker or drives
were replaced again within the 10sec window.
 2020-11-18 09:34:46 -08:00
Harshavardhana 9738d605e4
increase readdir per block memory to facilitate faster WalkDir (#10908) 2020-11-18 09:21:02 -08:00
Harshavardhana 7ff8128f15
update jstream to latest release v1.0.1 (#10909) 2020-11-17 09:13:28 -08:00
Klaus Post 10099357b6
listcache: Wrap returned errors (#10882) 
To give an indication of where they happen
 2020-11-17 09:11:59 -08:00
Harshavardhana 80b8ce89a4
remove context deadline from Delete calls (#10901) 2020-11-17 09:09:45 -08:00
Matthias Petermann 0745736e28
Use hw.physmem64 instead of hw.physmem for NetBSD in pkg/sys/getHwPhysmem (#10907) 
"hw.physmem" only reports reasonable results on 32 bit systems and is
kept to not break binary compatibility. On other systems, it reports
"-1". "hw.phymem64" reports reasonable results on all systems.

MinIO getHwPhysmem method currently uses the deprecated 
"hw.physmem" key which results in a parse error (due to the -1).

This pull request asks for changing this. The change was tested
successfully on NetBSD/amd64 9.1.
 2020-11-16 20:53:47 -08:00
Poorna Krishnamoorthy 0b766288ef
fix: send replication completed event notification (#10902) 2020-11-15 22:16:41 -08:00
Rafael Bodill 598ca0569c
fix: global in-place update boolean check (#10900) 2020-11-15 13:34:12 -08:00
Poorna Krishnamoorthy d295ce5708
Fix disk cache usage percent for prometheus (#10898) 
Fixes: #10895

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
 2020-11-14 19:18:00 -08:00
Klaus Post b5a3d79bce
listobjectversions: Add shortcut for Veeam blocks (#10893) 
Add shortcut for `APN/1.0 Veeam/1.0 Backup/10.0`

It requests unique blocks with a specific prefix. We skip 
scanning the parent directory for more objects matching the prefix.
 2020-11-13 16:58:20 -08:00
Harshavardhana 17a5ff51ff
fix: move context timeout closer to network for Delete calls (#10897) 
allowing for disconnects to be limited to the drive
themselves instead of disconnecting all drives.
 2020-11-13 16:56:45 -08:00
Minio Trusted 0784a0c33a Update yaml files to latest version RELEASE.2020-11-13T20-10-18Z 2020-11-13 20:27:01 +00:00
Minio Trusted 3595cb1267 Update yaml files to latest version RELEASE.2020-11-12T22-33-34Z 2020-11-12 22:51:11 +00:00
Harshavardhana 0bcb1b679d
fix: disallow update if dates are same (#10890) 
fixes #10889
 2020-11-12 14:18:59 -08:00
Klaus Post a3017c724e
Sort directory objects correctly (#10886) 
Decode dir objects when listing and sort them correctly.
 2020-11-12 13:09:34 -08:00
Omar Alvarez 07859ef48b
docs: clarify notifications support for gateways (#10729) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-11-12 12:19:04 -08:00
Harshavardhana 267d7bf0a9 Revert "Add delete marker replication support (#10396)" 
This reverts commit 50c10a5087.

PR is moved to origin/dev branch
 2020-11-12 11:43:14 -08:00
cksac be83dfc52a
fix: HDFS list bucket when subpath is provided (#10884) 2020-11-12 11:26:51 -08:00
Harshavardhana ca88ca753c
ignore typed errors correctly in list cache layer (#10879) 
bonus write bucket metadata cache with enough quorum

Possible fix for #10868
 2020-11-12 09:28:56 -08:00
Klaus Post f86d3538f6
Allow deeper sleep (#10883) 
Allow each crawler operation to sleep up to 10 seconds on very heavily loaded systems.

This will of course make minimum crawler speed less, but should be more effective at stopping.
 2020-11-12 09:17:56 -08:00
Klaus Post 1c3590078d
Skip 0 byte stream writes (#10875) 
Don't send a packet when receiving 0 bytes or there is an error recorded
 2020-11-11 18:07:40 -08:00
Harshavardhana aa158228f9
fix: simplify healing metadata objects per set (#10867) 2020-11-11 10:58:16 -08:00
Klaus Post 8747834c69
DeletedObjects: Return objects on lock failure (#10874) 
Return objects when locking fails.

<details>
<summary>Panic</summary>

```
: 2020/11/10 04:15:55 http: panic serving 10.10.62.153:44858: runtime error: index out of range [0] with length 0
: goroutine 363537270 [running]:
: net/http.(*conn).serve.func1(0xc019232780)
:         net/http/server.go:1801 +0x147
: panic(0x1cadd60, 0xc001719260)
:         runtime/panic.go:975 +0x47a
: github.com/minio/minio/cmd.criticalErrorHandler.ServeHTTP.func1(0xc0121d1200, 0x210cda0, 0xc0141940e0)
:         github.com/minio/minio/cmd/generic-handlers.go:781 +0x1a8
: panic(0x1cadd60, 0xc001719260)
:         runtime/panic.go:969 +0x1b9
: github.com/minio/minio/cmd.objectAPIHandlers.DeleteMultipleObjectsHandler(0x1e71ce8, 0x1e71cc8, 0x2108420, 0xc0192328c0, 0xc0121d1400)
:         github.com/minio/minio/cmd/bucket-handlers.go:465 +0x2490
: net/http.HandlerFunc.ServeHTTP(...)
:         net/http/server.go:2042
: github.com/minio/minio/cmd.httpTraceAll.func1(0x2108420, 0xc0192328c0, 0xc0121d1400)
:         github.com/minio/minio/cmd/handler-utils.go:353 +0x158
: net/http.HandlerFunc.ServeHTTP(...)
:         net/http/server.go:2042
: github.com/minio/minio/cmd.collectAPIStats.func1(0x2108420, 0xc019232820, 0xc0121d1400)
:         github.com/minio/minio/cmd/handler-utils.go:380 +0xed
: net/http.HandlerFunc.ServeHTTP(...)
:         net/http/server.go:2042
: github.com/minio/minio/cmd.maxClients.func1(0x2108420, 0xc019232820, 0xc0121d1400)
:         github.com/minio/minio/cmd/handler-api.go:132 +0x33b
: net/http.HandlerFunc.ServeHTTP(0xc00271d590, 0x2108420, 0xc019232820, 0xc0121d1400)
:         net/http/server.go:2042 +0x44
: github.com/minio/minio/cmd.redirectHandler.ServeHTTP(0x20e2180, 0xc00271d590, 0x2108420, 0xc019232820, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:192 +0x156
: github.com/minio/minio/cmd.customHeaderHandler.ServeHTTP(0x20e1060, 0xc0141a22b0, 0x21083e0, 0xc01814d2e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:751 +0x162
: github.com/minio/minio/cmd.securityHeaderHandler.ServeHTTP(0x20e0fc0, 0xc0141a22c0, 0x21083e0, 0xc01814d2e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:766 +0x1d6
: github.com/minio/minio/cmd.bucketForwardingHandler.ServeHTTP(0xc0121c7a40, 0x20e1120, 0xc0141a22d0, 0x21083e0, 0xc01814d2e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:624 +0xbf
: github.com/minio/minio/cmd.requestValidityHandler.ServeHTTP(0x20e0f20, 0xc01814d280, 0x21083e0, 0xc01814d2e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:608 +0x42a
: github.com/minio/minio/cmd.httpStatsHandler.ServeHTTP(0x20e10c0, 0xc0141a2300, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:536 +0xe4
: github.com/minio/minio/cmd.requestSizeLimitHandler.ServeHTTP(0x20e0fe0, 0xc0141a2310, 0x50004000000, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:68 +0xd4
: github.com/minio/minio/cmd.requestHeaderSizeLimitHandler.ServeHTTP(0x20e10a0, 0xc01814d2a0, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:93 +0x1b7
: github.com/minio/minio/cmd.crossDomainPolicy.ServeHTTP(0x20e1080, 0xc0141a2320, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/crossdomain-xml-handler.go:51 +0x82
: github.com/minio/minio/cmd.browserRedirectHandler.ServeHTTP(0x20e0fa0, 0xc0141a2330, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:276 +0x68
: github.com/minio/minio/cmd.minioReservedBucketHandler.ServeHTTP(0x20e0f00, 0xc0141a2340, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:344 +0xb8
: github.com/minio/minio/cmd.cacheControlHandler.ServeHTTP(0x20e1020, 0xc0141a2350, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:303 +0x1ce
: github.com/minio/minio/cmd.timeValidityHandler.ServeHTTP(0x20e0f40, 0xc0141a2360, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:414 +0x3ca
: github.com/minio/minio/cmd.resourceHandler.ServeHTTP(0x20e1160, 0xc0141a2370, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:516 +0xab
: github.com/minio/minio/cmd.authHandler.ServeHTTP(0x20e1100, 0xc0141a2380, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/auth-handler.go:502 +0x2e7
: github.com/minio/minio/cmd.sseTLSHandler.ServeHTTP(0x20e0ee0, 0xc0141a2390, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:802 +0x79
: github.com/minio/minio/cmd.reservedMetadataHandler.ServeHTTP(0x20e1140, 0xc0141a23a0, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:139 +0x1b7
: github.com/gorilla/mux.(*Router).ServeHTTP(0xc00073fb00, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
: github.com/rs/cors.(*Cors).Handler.func1(0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         github.com/rs/cors@v1.7.0/cors.go:219 +0x1b9
: net/http.HandlerFunc.ServeHTTP(0xc0009aece0, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         net/http/server.go:2042 +0x44
: github.com/minio/minio/cmd.criticalErrorHandler.ServeHTTP(0x20e2180, 0xc0009aece0, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         github.com/minio/minio/cmd/generic-handlers.go:784 +0x85
: github.com/minio/minio/cmd/http.(*Server).Start.func1(0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         github.com/minio/minio/cmd/http/server.go:101 +0x258
: net/http.HandlerFunc.ServeHTTP(0xc000dc4080, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         net/http/server.go:2042 +0x44
: net/http.serverHandler.ServeHTTP(0xc000764c60, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         net/http/server.go:2843 +0xa3
: net/http.(*conn).serve(0xc019232780, 0x2114720, 0xc03381f6c0)
:         net/http/server.go:1925 +0x8ad
: created by net/http.(*Server).Serve
:         net/http/server.go:2969 +0x36c
```
</details>
 2020-11-11 09:14:32 -08:00
Anton Melser 2c1e37197b
fix: bad example json for policy in replication docs (#10869) 2020-11-10 17:49:49 -08:00
Poorna Krishnamoorthy 50c10a5087
Add delete marker replication support (#10396) 
Delete marker replication is implemented for V2
configuration specified in AWS spec (though AWS
allows it only in the V1 configuration).

This PR also brings in a MinIO only extension of
replicating permanent deletes, i.e. deletes specifying
version id are replicated to target cluster.
 2020-11-10 15:24:14 -08:00
Minio Trusted 9f4ad873bc Update yaml files to latest version RELEASE.2020-11-10T21-02-24Z 2020-11-10 21:18:35 +00:00