Commit Graph

182 Commits

Author SHA1 Message Date
Harshavardhana
442c50ff00
remove delimiter if not set by client, also fetchOwner is optional (#17366) 2023-06-06 21:31:47 -07:00
Poorna
90e2cc3d4c
Add audit logging of site replication multipart proxying (#17122) 2023-05-03 11:19:45 -07:00
Harshavardhana
a5835cecbf
fix: regression in counting total requests (#17024) 2023-04-12 14:37:19 -07:00
Shubhendu
d3f70ea340
Enable audit log for global handlers (#16964)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-04-06 21:03:39 -07:00
Anis Eleuch
d90d0c8931
Use one http response recorder per external http call (#16938) 2023-03-31 09:37:29 -07:00
Harshavardhana
8d6558b236
fix: convert '\' to '/' on windows (#16852) 2023-03-20 00:35:25 -07:00
Harshavardhana
5c98223c89
add correct HostId instead of deploymentId for error responses (#16686) 2023-02-22 15:41:09 +05:30
Harshavardhana
54b561898f
fix: anonymize the x-amz-id-2 value from hostname (#16478) 2023-01-25 10:25:36 -08:00
Harshavardhana
65c104a589
add x-amz-id-2 to indicate the node that received the request (#16474) 2023-01-25 09:14:10 -08:00
Anis Elleuch
441babdc41
Rename peer S3 prefix to avoid collision in the future (#16473) 2023-01-25 06:46:30 -08:00
Anis Elleuch
932d2c3c62
Add X-Amz-Request-Id to internode calls (#16146) 2022-12-06 09:27:26 -08:00
Anis Elleuch
1f1dcdce65
move HTTP recorder to an internal library (#16128) 2022-11-28 10:20:27 -08:00
Poorna
e32b948a49
fix: parsing multipart uploadID under site replicated setup (#16048)
continue the fix from #16034
2022-11-10 16:17:45 -08:00
Poorna
ce8456a1a9
proxy multipart to peers via multipart uploadID (#15926) 2022-10-25 10:52:29 -07:00
Javier Adriel
3307aa1260
Implement KMS handlers (#15737) 2022-10-04 10:05:09 -07:00
Harshavardhana
b579163802
limit number of buckets to 500k (#15668)
500k is a reasonable limit for any single MinIO
cluster deployment, in future we may increase this
value.

However for now we are going to keep this limit.
2022-09-09 03:06:34 -07:00
Harshavardhana
228c6686f8
allow non-standards fallback for all http.TimeFormats (#15662)
fixes #15645
2022-09-07 07:24:54 -07:00
Harshavardhana
2070c215a2
handle missing funcNames for handlers (#15188)
also use designated names for internal
calls

- storageREST calls are storageR
- lockREST calls are lockR
- peerREST calls are just peer

Named in this fashion to facilitate wildcard matches
by having prefixes of the same name.

Additionally, also enable funcNames for generic handlers
that return errors, currently we disable '<unknown>'
2022-06-28 05:04:10 -07:00
Harshavardhana
8082d1fed6
add bucket level S3 received/sent bytes (#15084)
adds bucket level metrics for bytes received and sent bytes on all S3 API calls.
2022-06-14 15:14:24 -07:00
Harshavardhana
7413045f0e
fix: add missing minio_s3_requests_total (#15070)
PR #15052 caused a regression, add the missing metrics back.

Bonus:

- internode information should be only for distributed setups 
- update the dashboard to include 4xx and 5xx error panels.
2022-06-11 00:50:31 -07:00
Anis Elleuch
5fb420c703
prometheus: Add S3 4xx and 5xx S3 monitoring (#15052)
Currently minio_s3_requests_errors_total covers 4xx and 
5xx S3 responses which can be confusing when s3 applications 
sent a lot of HEAD requests with obvious 404 responses or 
when the replication is enabled.

Add 
- minio_s3_requests_4xx_errors_total
- minio_s3_requests_5xx_errors_total

to help users monitor 4xx and 5xx HTTP status codes separately.
2022-06-08 11:22:34 -07:00
Harshavardhana
2719f1efaa
fix: reject invalid r.Host headers (#14846)
r.Host headers can come in unparsed that may contain
invalid hostnames, reject such requests as invalid.

This is a continuation fix from #14844
2022-05-02 04:42:41 -07:00
Harshavardhana
39ac62a1a1
fix: panic in browser redirect handler for unexpected r.Host (#14844)
```
panic: "GET /": invalid hostname
goroutine 148 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/minio/minio/cmd.setCriticalErrorHandler.func1.1()
	github.com/minio/minio/cmd/generic-handlers.go:469 +0x8e
panic({0x2201f00, 0xc001f1ddd0})
	runtime/panic.go:1038 +0x215
github.com/minio/pkg/net.URL.String({{0x25aa417, 0x5}, {0x0, 0x0}, 0x0, {0xc000174380, 0xd7}, {0x0, 0x0}, {0x0, ...}, ...})
	github.com/minio/pkg@v1.1.23/net/url.go:97 +0xfe
github.com/minio/minio/cmd.setBrowserRedirectHandler.func1({0x49af080, 0xc0003c20e0}, 0xc00002ea00)
	github.com/minio/minio/cmd/generic-handlers.go:136 +0x118
net/http.HandlerFunc.ServeHTTP(0xc00002ea00, {0x49af080, 0xc0003c20e0}, 0xa)
	net/http/server.go:2047 +0x2f
github.com/minio/minio/cmd.setAuthHandler.func1({0x49af080, 0xc0003c20e0}, 0xc00002ea00)
	github.com/minio/minio/cmd/auth-handler.go:525 +0x3d8
net/http.HandlerFunc.ServeHTTP(0xc00002e900, {0x49af080, 0xc0003c20e0}, 0xc001f33701)
	net/http/server.go:2047 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0025d0780, {0x49af080, 0xc0003c20e0}, 0xc00002e800)
	github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
github.com/rs/cors.(*Cors).Handler.func1({0x49af080, 0xc0003c20e0}, 0xc00002e800)
	github.com/rs/cors@v1.7.0/cors.go:219 +0x1bd
net/http.HandlerFunc.ServeHTTP(0x0, {0x49af080, 0xc0003c20e0}, 0xc00068d9f8)
	net/http/server.go:2047 +0x2f
github.com/minio/minio/cmd.setCriticalErrorHandler.func1({0x49af080, 0xc0003c20e0}, 0x4a5cd3)
	github.com/minio/minio/cmd/generic-handlers.go:476 +0x83
net/http.HandlerFunc.ServeHTTP(0x72, {0x49af080, 0xc0003c20e0}, 0x0)
	net/http/server.go:2047 +0x2f
github.com/minio/minio/internal/http.(*Server).Start.func1({0x49af080, 0xc0003c20e0}, 0x10000c001f1dda0)
	github.com/minio/minio/internal/http/server.go:105 +0x1b6
net/http.HandlerFunc.ServeHTTP(0x0, {0x49af080, 0xc0003c20e0}, 0x46982e)
	net/http/server.go:2047 +0x2f
net/http.serverHandler.ServeHTTP({0xc003dc1950}, {0x49af080, 0xc0003c20e0}, 0xc00002e800)
	net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc000514d20, {0x49cfc38, 0xc0010c0e70})
	net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
	net/http/server.go:3034 +0x4e8
```
2022-05-01 13:45:45 -07:00
Anis Elleuch
71bab74148
Fix adding bucket forwarder handler in server mode (#14288)
MinIO configuration is loaded after the initialization of the server
handlers, which will miss the initialization of the bucket forwarder
handler.

Though the federation is deprecated, let's fix this for the time being.
2022-02-10 08:49:36 -08:00
Klaus Post
d9800c8135
fix: make sure to log panic in handlers (#13611) 2021-11-08 09:28:13 -08:00
Harshavardhana
6d53e3c2d7
reduce number of middleware handlers (#13546)
- combine similar looking functionalities into single
  handlers, and remove unnecessary proxying of the
  requests at handler layer.

- remove bucket forwarding handler as part of default setup
  add it only if bucket federation is enabled.

Improvements observed for 1kiB object reads.
```
-------------------
Operation: GET
Operations: 4538555 -> 4595804
* Average: +1.26% (+0.2 MiB/s) throughput, +1.26% (+190.2) obj/s
* Fastest: +4.67% (+0.7 MiB/s) throughput, +4.67% (+739.8) obj/s
* 50% Median: +1.15% (+0.2 MiB/s) throughput, +1.15% (+173.9) obj/s
```
2021-11-01 08:04:03 -07:00
Harshavardhana
db84bb9bd3
avoid atomics for self contained reader/writers (#13531)
read/writers are not concurrent in handlers
and self contained - no need to use atomics on
them.

avoids unnecessary contentions where it's not
required.
2021-10-28 17:03:00 -07:00
Harshavardhana
a2cd3c9a1d
use ParseForm() to allow query param lookups once (#12900)
```
cpu: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
BenchmarkURLQueryForm
BenchmarkURLQueryForm-4         247099363                4.809 ns/op           0 B/op          0 allocs/op
BenchmarkURLQuery
BenchmarkURLQuery-4              2517624               462.1 ns/op           432 B/op          4 allocs/op
PASS
ok      github.com/minio/minio/cmd      3.848s
```
2021-08-07 22:43:01 -07:00
Harshavardhana
397637a042 remove x-frame-options HTTP header
fixes #12817
2021-07-28 08:57:22 -07:00
sakkiii
69e0faa278
add more XSS HTTP headers (#12256) 2021-07-19 16:05:02 -07:00
Harshavardhana
9dae5a7c85
fix: only redirect for GET/HEAD requests (#12702)
fixes #12701
2021-07-13 11:25:08 -07:00
Harshavardhana
da0fd5f056
fix: allow customizing console redirection (#12665)
MinIO might be running inside proxies, and
console while being on another port might not be
reachable on a specific port behind such proxies.

For such scenarios customize the redirect URL
such that console can be redirected to correct
proxy endpoint instead.

fixes #12661
2021-07-09 14:27:09 -07:00
Harshavardhana
c99d399d09
fix: handle redirects for specific resources (#12629) 2021-07-07 12:04:16 -07:00
Harshavardhana
8f1fe3b761
fix: --console-address when specified endpoints missing (#12534)
Additionally upgrade console dependency for reading
environment variables properly.
2021-06-20 23:04:47 -07:00
Harshavardhana
cdeccb5510
feat: Deprecate embedded browser and import console (#12460)
This feature also changes the default port where
the browser is running, now the port has moved
to 9001 and it can be configured with

```
--console-address ":9001"
```
2021-06-17 20:27:04 -07:00
Harshavardhana
1f262daf6f
rename all remaining packages to internal/ (#12418)
This is to ensure that there are no projects
that try to import `minio/minio/pkg` into
their own repo. Any such common packages should
go to `https://github.com/minio/pkg`
2021-06-01 14:59:40 -07:00
Harshavardhana
bf87c4b1e4
fix: no need to proxy if IAM not initialized (#12416)
IAM not initialized doesn't mean we can't still
read the content from the disk, we should just
allow the request to go-through if object layer
is initialized.
2021-06-01 12:23:13 -07:00
Harshavardhana
f7a87b30bf Revert "deprecate embedded browser (#12163)"
This reverts commit 736d8cbac4.

Bring contrib files for older contributions
2021-04-30 08:50:39 -07:00
Harshavardhana
77f9c71133 Revert "redirect to console project for browser (#12172)"
This reverts commit 301669cf7b.

fixes #12179
2021-04-28 12:22:15 -07:00
Harshavardhana
301669cf7b
redirect to console project for browser (#12172) 2021-04-27 16:39:41 -07:00
Harshavardhana
736d8cbac4
deprecate embedded browser (#12163)
https://github.com/minio/console takes over the functionality for the
future object browser development

Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-27 10:52:12 -07:00
Harshavardhana
069432566f update license change for MinIO
Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-23 11:58:53 -07:00
Ritesh H Shukla
3ddd8b04d1
fix: handle unsupported APIs more granularly (#11674) 2021-03-30 23:19:36 -07:00
Harshavardhana
b1bb3f7016
[feat]: implement GetBucketPolicyStatus API (#11673)
additionally also add more APIs in notImplemented
list, adjust routing rules appropriately
2021-03-01 23:10:33 -08:00
Krishna Srinivas
876b79b8d8
read-health check endpoint returns success if cluster can serve read requests (#11310) 2021-02-09 01:00:44 -08:00
Harshavardhana
6a95f412c9
avoid double CORS headers in federation (#11334)
CORS proxying adds double headers one
by the receiving server, one by proxied
server. Remove them before proxying
when 'Origin' header is found.
2021-01-23 18:27:23 -08:00
Harshavardhana
1b453728a3
initialize forwarder after init() to avoid crashes (#11330)
DNSCache dialer is a global value initialized in
init(), whereas `go` keeps `var =` before `init()`
, also we don't need to keep proxy routers as
global entities - register the forwarder as
necessary to avoid crashes.
2021-01-22 15:37:41 -08:00
Ritesh H Shukla
b4add82bb6
Updated Prometheus metrics (#11141)
* Add metrics for nodes online and offline
* Add cluster capacity metrics
* Introduce v2 metrics
2021-01-18 20:35:38 -08:00
Anis Elleuch
cb7fc99368
handlers: Avoid initializing a struct in each handler call (#11217) 2021-01-04 09:54:22 -08:00
Anis Elleuch
dfd99b6d8f
handlers: Little bit more optimizations (#11211) 2021-01-04 00:01:06 -08:00