MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,772 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

7772 Commits

Author SHA1 Message Date
Harshavardhana 9dea7020f0
allow prefix filtering for WalkDir to be optional (#10923) 2020-11-18 12:03:16 -08:00
Klaus Post 990d074f7d
metacache: Allow prefix filtering (#10920) 
Do listings with prefix filter when bloom filter is dirty.

This will forward the prefix filter to the lister which will make it 
only scan the folders/objects with the specified prefix.

If we have a clean bloom filter we try to build a more generally 
useful cache so in that case, we will list all objects/folders.
 2020-11-18 10:44:18 -08:00
Klaus Post e413f05397
Save listing error async (#10922) 
Since the RPC call may have to time out save an error state async 
to not hold up the listing returning.

Fixes #10919
 2020-11-18 10:28:22 -08:00
Harshavardhana d1b1fee080
fix: save healing tracker right before healing (#10915) 
this change avoids a situation where accidentally
if the user deleted the healing tracker or drives
were replaced again within the 10sec window.
 2020-11-18 09:34:46 -08:00
Harshavardhana 9738d605e4
increase readdir per block memory to facilitate faster WalkDir (#10908) 2020-11-18 09:21:02 -08:00
Harshavardhana 7ff8128f15
update jstream to latest release v1.0.1 (#10909) 2020-11-17 09:13:28 -08:00
Klaus Post 10099357b6
listcache: Wrap returned errors (#10882) 
To give an indication of where they happen
 2020-11-17 09:11:59 -08:00
Harshavardhana 80b8ce89a4
remove context deadline from Delete calls (#10901) 2020-11-17 09:09:45 -08:00
Matthias Petermann 0745736e28
Use hw.physmem64 instead of hw.physmem for NetBSD in pkg/sys/getHwPhysmem (#10907) 
"hw.physmem" only reports reasonable results on 32 bit systems and is
kept to not break binary compatibility. On other systems, it reports
"-1". "hw.phymem64" reports reasonable results on all systems.

MinIO getHwPhysmem method currently uses the deprecated 
"hw.physmem" key which results in a parse error (due to the -1).

This pull request asks for changing this. The change was tested
successfully on NetBSD/amd64 9.1.
 2020-11-16 20:53:47 -08:00
Poorna Krishnamoorthy 0b766288ef
fix: send replication completed event notification (#10902) 2020-11-15 22:16:41 -08:00
Rafael Bodill 598ca0569c
fix: global in-place update boolean check (#10900) 2020-11-15 13:34:12 -08:00
Poorna Krishnamoorthy d295ce5708
Fix disk cache usage percent for prometheus (#10898) 
Fixes: #10895

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
 2020-11-14 19:18:00 -08:00
Klaus Post b5a3d79bce
listobjectversions: Add shortcut for Veeam blocks (#10893) 
Add shortcut for `APN/1.0 Veeam/1.0 Backup/10.0`

It requests unique blocks with a specific prefix. We skip 
scanning the parent directory for more objects matching the prefix.
 2020-11-13 16:58:20 -08:00
Harshavardhana 17a5ff51ff
fix: move context timeout closer to network for Delete calls (#10897) 
allowing for disconnects to be limited to the drive
themselves instead of disconnecting all drives.
 2020-11-13 16:56:45 -08:00
Minio Trusted 0784a0c33a Update yaml files to latest version RELEASE.2020-11-13T20-10-18Z 2020-11-13 20:27:01 +00:00
Minio Trusted 3595cb1267 Update yaml files to latest version RELEASE.2020-11-12T22-33-34Z 2020-11-12 22:51:11 +00:00
Harshavardhana 0bcb1b679d
fix: disallow update if dates are same (#10890) 
fixes #10889
 2020-11-12 14:18:59 -08:00
Klaus Post a3017c724e
Sort directory objects correctly (#10886) 
Decode dir objects when listing and sort them correctly.
 2020-11-12 13:09:34 -08:00
Omar Alvarez 07859ef48b
docs: clarify notifications support for gateways (#10729) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-11-12 12:19:04 -08:00
Harshavardhana 267d7bf0a9 Revert "Add delete marker replication support (#10396)" 
This reverts commit 50c10a5087.

PR is moved to origin/dev branch
 2020-11-12 11:43:14 -08:00
cksac be83dfc52a
fix: HDFS list bucket when subpath is provided (#10884) 2020-11-12 11:26:51 -08:00
Harshavardhana ca88ca753c
ignore typed errors correctly in list cache layer (#10879) 
bonus write bucket metadata cache with enough quorum

Possible fix for #10868
 2020-11-12 09:28:56 -08:00
Klaus Post f86d3538f6
Allow deeper sleep (#10883) 
Allow each crawler operation to sleep up to 10 seconds on very heavily loaded systems.

This will of course make minimum crawler speed less, but should be more effective at stopping.
 2020-11-12 09:17:56 -08:00
Klaus Post 1c3590078d
Skip 0 byte stream writes (#10875) 
Don't send a packet when receiving 0 bytes or there is an error recorded
 2020-11-11 18:07:40 -08:00
Harshavardhana aa158228f9
fix: simplify healing metadata objects per set (#10867) 2020-11-11 10:58:16 -08:00
Klaus Post 8747834c69
DeletedObjects: Return objects on lock failure (#10874) 
Return objects when locking fails.

<details>
<summary>Panic</summary>

```
: 2020/11/10 04:15:55 http: panic serving 10.10.62.153:44858: runtime error: index out of range [0] with length 0
: goroutine 363537270 [running]:
: net/http.(*conn).serve.func1(0xc019232780)
:         net/http/server.go:1801 +0x147
: panic(0x1cadd60, 0xc001719260)
:         runtime/panic.go:975 +0x47a
: github.com/minio/minio/cmd.criticalErrorHandler.ServeHTTP.func1(0xc0121d1200, 0x210cda0, 0xc0141940e0)
:         github.com/minio/minio/cmd/generic-handlers.go:781 +0x1a8
: panic(0x1cadd60, 0xc001719260)
:         runtime/panic.go:969 +0x1b9
: github.com/minio/minio/cmd.objectAPIHandlers.DeleteMultipleObjectsHandler(0x1e71ce8, 0x1e71cc8, 0x2108420, 0xc0192328c0, 0xc0121d1400)
:         github.com/minio/minio/cmd/bucket-handlers.go:465 +0x2490
: net/http.HandlerFunc.ServeHTTP(...)
:         net/http/server.go:2042
: github.com/minio/minio/cmd.httpTraceAll.func1(0x2108420, 0xc0192328c0, 0xc0121d1400)
:         github.com/minio/minio/cmd/handler-utils.go:353 +0x158
: net/http.HandlerFunc.ServeHTTP(...)
:         net/http/server.go:2042
: github.com/minio/minio/cmd.collectAPIStats.func1(0x2108420, 0xc019232820, 0xc0121d1400)
:         github.com/minio/minio/cmd/handler-utils.go:380 +0xed
: net/http.HandlerFunc.ServeHTTP(...)
:         net/http/server.go:2042
: github.com/minio/minio/cmd.maxClients.func1(0x2108420, 0xc019232820, 0xc0121d1400)
:         github.com/minio/minio/cmd/handler-api.go:132 +0x33b
: net/http.HandlerFunc.ServeHTTP(0xc00271d590, 0x2108420, 0xc019232820, 0xc0121d1400)
:         net/http/server.go:2042 +0x44
: github.com/minio/minio/cmd.redirectHandler.ServeHTTP(0x20e2180, 0xc00271d590, 0x2108420, 0xc019232820, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:192 +0x156
: github.com/minio/minio/cmd.customHeaderHandler.ServeHTTP(0x20e1060, 0xc0141a22b0, 0x21083e0, 0xc01814d2e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:751 +0x162
: github.com/minio/minio/cmd.securityHeaderHandler.ServeHTTP(0x20e0fc0, 0xc0141a22c0, 0x21083e0, 0xc01814d2e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:766 +0x1d6
: github.com/minio/minio/cmd.bucketForwardingHandler.ServeHTTP(0xc0121c7a40, 0x20e1120, 0xc0141a22d0, 0x21083e0, 0xc01814d2e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:624 +0xbf
: github.com/minio/minio/cmd.requestValidityHandler.ServeHTTP(0x20e0f20, 0xc01814d280, 0x21083e0, 0xc01814d2e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:608 +0x42a
: github.com/minio/minio/cmd.httpStatsHandler.ServeHTTP(0x20e10c0, 0xc0141a2300, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:536 +0xe4
: github.com/minio/minio/cmd.requestSizeLimitHandler.ServeHTTP(0x20e0fe0, 0xc0141a2310, 0x50004000000, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:68 +0xd4
: github.com/minio/minio/cmd.requestHeaderSizeLimitHandler.ServeHTTP(0x20e10a0, 0xc01814d2a0, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:93 +0x1b7
: github.com/minio/minio/cmd.crossDomainPolicy.ServeHTTP(0x20e1080, 0xc0141a2320, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/crossdomain-xml-handler.go:51 +0x82
: github.com/minio/minio/cmd.browserRedirectHandler.ServeHTTP(0x20e0fa0, 0xc0141a2330, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:276 +0x68
: github.com/minio/minio/cmd.minioReservedBucketHandler.ServeHTTP(0x20e0f00, 0xc0141a2340, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:344 +0xb8
: github.com/minio/minio/cmd.cacheControlHandler.ServeHTTP(0x20e1020, 0xc0141a2350, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:303 +0x1ce
: github.com/minio/minio/cmd.timeValidityHandler.ServeHTTP(0x20e0f40, 0xc0141a2360, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:414 +0x3ca
: github.com/minio/minio/cmd.resourceHandler.ServeHTTP(0x20e1160, 0xc0141a2370, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:516 +0xab
: github.com/minio/minio/cmd.authHandler.ServeHTTP(0x20e1100, 0xc0141a2380, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/auth-handler.go:502 +0x2e7
: github.com/minio/minio/cmd.sseTLSHandler.ServeHTTP(0x20e0ee0, 0xc0141a2390, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:802 +0x79
: github.com/minio/minio/cmd.reservedMetadataHandler.ServeHTTP(0x20e1140, 0xc0141a23a0, 0x210cda0, 0xc0141940e0, 0xc0121d1400)
:         github.com/minio/minio/cmd/generic-handlers.go:139 +0x1b7
: github.com/gorilla/mux.(*Router).ServeHTTP(0xc00073fb00, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
: github.com/rs/cors.(*Cors).Handler.func1(0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         github.com/rs/cors@v1.7.0/cors.go:219 +0x1b9
: net/http.HandlerFunc.ServeHTTP(0xc0009aece0, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         net/http/server.go:2042 +0x44
: github.com/minio/minio/cmd.criticalErrorHandler.ServeHTTP(0x20e2180, 0xc0009aece0, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         github.com/minio/minio/cmd/generic-handlers.go:784 +0x85
: github.com/minio/minio/cmd/http.(*Server).Start.func1(0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         github.com/minio/minio/cmd/http/server.go:101 +0x258
: net/http.HandlerFunc.ServeHTTP(0xc000dc4080, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         net/http/server.go:2042 +0x44
: net/http.serverHandler.ServeHTTP(0xc000764c60, 0x210cda0, 0xc0141940e0, 0xc0121d1200)
:         net/http/server.go:2843 +0xa3
: net/http.(*conn).serve(0xc019232780, 0x2114720, 0xc03381f6c0)
:         net/http/server.go:1925 +0x8ad
: created by net/http.(*Server).Serve
:         net/http/server.go:2969 +0x36c
```
</details>
 2020-11-11 09:14:32 -08:00
Anton Melser 2c1e37197b
fix: bad example json for policy in replication docs (#10869) 2020-11-10 17:49:49 -08:00
Poorna Krishnamoorthy 50c10a5087
Add delete marker replication support (#10396) 
Delete marker replication is implemented for V2
configuration specified in AWS spec (though AWS
allows it only in the V1 configuration).

This PR also brings in a MinIO only extension of
replicating permanent deletes, i.e. deletes specifying
version id are replicated to target cluster.
 2020-11-10 15:24:14 -08:00
Minio Trusted 9f4ad873bc Update yaml files to latest version RELEASE.2020-11-10T21-02-24Z 2020-11-10 21:18:35 +00:00
Steven Reitsma 4683a623dc
fix: negative STS IAM token TTL value (#10866) 2020-11-10 12:24:01 -08:00
Klaus Post 06899210a7
Reduce health check output (#10859) 
This will make the health check clients 'silent'.
Use `IsNetworkOrHostDown` determine if network is ok so it mimics the functionality in the actual client.
 2020-11-10 09:28:23 -08:00
Harshavardhana cbdab62c1e
fix: heal user/metadata right away upon server startup (#10863) 
this is needed such that we make sure to heal the
users, policies and bucket metadata right away as
we do listing based on list cache which only lists
'3' sufficiently good drives, to avoid possibly
losing access to these users upon upgrade make
sure to heal them.
 2020-11-10 09:02:06 -08:00
Harshavardhana 8df6112204
fix: avoid divide by zero error single node distributed setup (#10862) 2020-11-09 20:40:39 -08:00
Anis Elleuch 8e8ddf7233
doc: Add definition of 1KB and 1MB in prometheus (#10857) 2020-11-09 10:05:01 -08:00
Klaus Post 311ab43d4c
Upgrade msgp to official release (#10858) 
Code identical, but official release.
 2020-11-09 07:52:24 -08:00
Harshavardhana 97692bc772
re-route requests if IAM is not initialized (#10850) 2020-11-07 21:03:06 -08:00
Minio Trusted 21016265e5 Update yaml files to latest version RELEASE.2020-11-06T23-17-07Z 2020-11-06 23:34:19 +00:00
Steven Reitsma 54120107ce
fix: infinite loop in cleanupStaleUploads of encrypted MPUs (#10845) 
fixes #10588
 2020-11-06 11:53:42 -08:00
Klaus Post 9bf5990ea9
metadata: Invalidate cache if unreadable and not updating (#10844) 
If a scanning server shuts down unexpectedly we may have "successful" caches that are incomplete on a set.

In this case mark the cache with an error so it will no longer be handed out.
 2020-11-06 08:54:09 -08:00
Steven Reitsma 74f7cf24ae
fix: s3 gateway SSE pagination (#10840) 
Fixes #10838
 2020-11-05 15:04:03 -08:00
Harshavardhana fb28aa847b
fix: add missing deleted key element in multiObjectDelete (#10839) 
fixes #10832
 2020-11-05 12:47:46 -08:00
Klaus Post 0724205f35
metacache: Add option for life extension (#10837) 
Add `MINIO_API_EXTEND_LIST_CACHE_LIFE` that will extend 
the life of generated caches for a while.

This changes caches to remain valid until no updates have been 
received for the specified time plus a fixed margin.

This also changes the caches from being invalidated when the *first* 
set finishes until the *last* set has finished plus the specified time 
has passed.
 2020-11-05 11:49:56 -08:00
Harshavardhana b72cac4cf3
fix: dangling objects on actual namespace (#10822) 2020-11-05 11:48:55 -08:00
Klaus Post 47d715f642
Update msgp to main branch (#10835) 
The main branch has error unwrapping to clean up logs on canceled contexts, etc.

Asked for a new release: https://github.com/tinylib/msgp/issues/282
 2020-11-05 08:13:03 -08:00
Klaus Post bd77f29fc4
Don't replace caches that are receiving updates (#10834) 
Keep caches while they are receiving updates.
Move update code to separate function.
 2020-11-05 07:34:08 -08:00
Klaus Post d1e1205036
metacache: Always close the s2 writer (#10836) 
The s2 writer could be leaked if there was an error.

Make sure it is always closed.
 2020-11-05 07:30:14 -08:00
Harshavardhana 71753e21e0
add missing TTL for STS credentials on etcd (#10828) 2020-11-04 13:06:05 -08:00
Harshavardhana fde3299bf3
re-use optimized readdir for isDirEmpty() (#10829) 
reduces effective memory usage by an order
of magnitude, also increases performance for
small objects
 2020-11-04 13:05:21 -08:00
Harshavardhana 1a1f00fa15
fix: use internode data for DisksInfo, VolsInfo in message pack (#10821) 
Similar to #10775 for fewer memory allocations, since we use
getOnlineDisks() extensively for listing we should optimize it
further.

Additionally, remove all unused walkers from the storage layer
 2020-11-04 10:10:54 -08:00
Bill Thorp 4a1efabda4
Context based AccessKey passing (#10615) 
A new field called AccessKey is added to the ReqInfo struct and populated.
Because ReqInfo is added to the context, this allows the AccessKey to be
accessed from 3rd-party code, such as a custom ObjectLayer.

Co-authored-by: Harshavardhana <harsha@minio.io>
Co-authored-by: Kaloyan Raev <kaloyan@storj.io>
 2020-11-04 09:13:34 -08:00