Commit Graph

181 Commits

Author SHA1 Message Date
Poorna 90e2cc3d4c
Add audit logging of site replication multipart proxying (#17122) 2023-05-03 11:19:45 -07:00
Harshavardhana a5835cecbf
fix: regression in counting total requests (#17024) 2023-04-12 14:37:19 -07:00
Shubhendu d3f70ea340
Enable audit log for global handlers (#16964)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-04-06 21:03:39 -07:00
Anis Eleuch d90d0c8931
Use one http response recorder per external http call (#16938) 2023-03-31 09:37:29 -07:00
Harshavardhana 8d6558b236
fix: convert '\' to '/' on windows (#16852) 2023-03-20 00:35:25 -07:00
Harshavardhana 5c98223c89
add correct HostId instead of deploymentId for error responses (#16686) 2023-02-22 15:41:09 +05:30
Harshavardhana 54b561898f
fix: anonymize the x-amz-id-2 value from hostname (#16478) 2023-01-25 10:25:36 -08:00
Harshavardhana 65c104a589
add x-amz-id-2 to indicate the node that received the request (#16474) 2023-01-25 09:14:10 -08:00
Anis Elleuch 441babdc41
Rename peer S3 prefix to avoid collision in the future (#16473) 2023-01-25 06:46:30 -08:00
Anis Elleuch 932d2c3c62
Add X-Amz-Request-Id to internode calls (#16146) 2022-12-06 09:27:26 -08:00
Anis Elleuch 1f1dcdce65
move HTTP recorder to an internal library (#16128) 2022-11-28 10:20:27 -08:00
Poorna e32b948a49
fix: parsing multipart uploadID under site replicated setup (#16048)
continue the fix from #16034
2022-11-10 16:17:45 -08:00
Poorna ce8456a1a9
proxy multipart to peers via multipart uploadID (#15926) 2022-10-25 10:52:29 -07:00
Javier Adriel 3307aa1260
Implement KMS handlers (#15737) 2022-10-04 10:05:09 -07:00
Harshavardhana b579163802
limit number of buckets to 500k (#15668)
500k is a reasonable limit for any single MinIO
cluster deployment, in future we may increase this
value.

However for now we are going to keep this limit.
2022-09-09 03:06:34 -07:00
Harshavardhana 228c6686f8
allow non-standards fallback for all http.TimeFormats (#15662)
fixes #15645
2022-09-07 07:24:54 -07:00
Harshavardhana 2070c215a2
handle missing funcNames for handlers (#15188)
also use designated names for internal
calls

- storageREST calls are storageR
- lockREST calls are lockR
- peerREST calls are just peer

Named in this fashion to facilitate wildcard matches
by having prefixes of the same name.

Additionally, also enable funcNames for generic handlers
that return errors, currently we disable '<unknown>'
2022-06-28 05:04:10 -07:00
Harshavardhana 8082d1fed6
add bucket level S3 received/sent bytes (#15084)
adds bucket level metrics for bytes received and sent bytes on all S3 API calls.
2022-06-14 15:14:24 -07:00
Harshavardhana 7413045f0e
fix: add missing minio_s3_requests_total (#15070)
PR #15052 caused a regression, add the missing metrics back.

Bonus:

- internode information should be only for distributed setups 
- update the dashboard to include 4xx and 5xx error panels.
2022-06-11 00:50:31 -07:00
Anis Elleuch 5fb420c703
prometheus: Add S3 4xx and 5xx S3 monitoring (#15052)
Currently minio_s3_requests_errors_total covers 4xx and 
5xx S3 responses which can be confusing when s3 applications 
sent a lot of HEAD requests with obvious 404 responses or 
when the replication is enabled.

Add 
- minio_s3_requests_4xx_errors_total
- minio_s3_requests_5xx_errors_total

to help users monitor 4xx and 5xx HTTP status codes separately.
2022-06-08 11:22:34 -07:00
Harshavardhana 2719f1efaa
fix: reject invalid r.Host headers (#14846)
r.Host headers can come in unparsed that may contain
invalid hostnames, reject such requests as invalid.

This is a continuation fix from #14844
2022-05-02 04:42:41 -07:00
Harshavardhana 39ac62a1a1
fix: panic in browser redirect handler for unexpected r.Host (#14844)
```
panic: "GET /": invalid hostname
goroutine 148 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/minio/minio/cmd.setCriticalErrorHandler.func1.1()
	github.com/minio/minio/cmd/generic-handlers.go:469 +0x8e
panic({0x2201f00, 0xc001f1ddd0})
	runtime/panic.go:1038 +0x215
github.com/minio/pkg/net.URL.String({{0x25aa417, 0x5}, {0x0, 0x0}, 0x0, {0xc000174380, 0xd7}, {0x0, 0x0}, {0x0, ...}, ...})
	github.com/minio/pkg@v1.1.23/net/url.go:97 +0xfe
github.com/minio/minio/cmd.setBrowserRedirectHandler.func1({0x49af080, 0xc0003c20e0}, 0xc00002ea00)
	github.com/minio/minio/cmd/generic-handlers.go:136 +0x118
net/http.HandlerFunc.ServeHTTP(0xc00002ea00, {0x49af080, 0xc0003c20e0}, 0xa)
	net/http/server.go:2047 +0x2f
github.com/minio/minio/cmd.setAuthHandler.func1({0x49af080, 0xc0003c20e0}, 0xc00002ea00)
	github.com/minio/minio/cmd/auth-handler.go:525 +0x3d8
net/http.HandlerFunc.ServeHTTP(0xc00002e900, {0x49af080, 0xc0003c20e0}, 0xc001f33701)
	net/http/server.go:2047 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0025d0780, {0x49af080, 0xc0003c20e0}, 0xc00002e800)
	github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
github.com/rs/cors.(*Cors).Handler.func1({0x49af080, 0xc0003c20e0}, 0xc00002e800)
	github.com/rs/cors@v1.7.0/cors.go:219 +0x1bd
net/http.HandlerFunc.ServeHTTP(0x0, {0x49af080, 0xc0003c20e0}, 0xc00068d9f8)
	net/http/server.go:2047 +0x2f
github.com/minio/minio/cmd.setCriticalErrorHandler.func1({0x49af080, 0xc0003c20e0}, 0x4a5cd3)
	github.com/minio/minio/cmd/generic-handlers.go:476 +0x83
net/http.HandlerFunc.ServeHTTP(0x72, {0x49af080, 0xc0003c20e0}, 0x0)
	net/http/server.go:2047 +0x2f
github.com/minio/minio/internal/http.(*Server).Start.func1({0x49af080, 0xc0003c20e0}, 0x10000c001f1dda0)
	github.com/minio/minio/internal/http/server.go:105 +0x1b6
net/http.HandlerFunc.ServeHTTP(0x0, {0x49af080, 0xc0003c20e0}, 0x46982e)
	net/http/server.go:2047 +0x2f
net/http.serverHandler.ServeHTTP({0xc003dc1950}, {0x49af080, 0xc0003c20e0}, 0xc00002e800)
	net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc000514d20, {0x49cfc38, 0xc0010c0e70})
	net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
	net/http/server.go:3034 +0x4e8
```
2022-05-01 13:45:45 -07:00
Anis Elleuch 71bab74148
Fix adding bucket forwarder handler in server mode (#14288)
MinIO configuration is loaded after the initialization of the server
handlers, which will miss the initialization of the bucket forwarder
handler.

Though the federation is deprecated, let's fix this for the time being.
2022-02-10 08:49:36 -08:00
Klaus Post d9800c8135
fix: make sure to log panic in handlers (#13611) 2021-11-08 09:28:13 -08:00
Harshavardhana 6d53e3c2d7
reduce number of middleware handlers (#13546)
- combine similar looking functionalities into single
  handlers, and remove unnecessary proxying of the
  requests at handler layer.

- remove bucket forwarding handler as part of default setup
  add it only if bucket federation is enabled.

Improvements observed for 1kiB object reads.
```
-------------------
Operation: GET
Operations: 4538555 -> 4595804
* Average: +1.26% (+0.2 MiB/s) throughput, +1.26% (+190.2) obj/s
* Fastest: +4.67% (+0.7 MiB/s) throughput, +4.67% (+739.8) obj/s
* 50% Median: +1.15% (+0.2 MiB/s) throughput, +1.15% (+173.9) obj/s
```
2021-11-01 08:04:03 -07:00
Harshavardhana db84bb9bd3
avoid atomics for self contained reader/writers (#13531)
read/writers are not concurrent in handlers
and self contained - no need to use atomics on
them.

avoids unnecessary contentions where it's not
required.
2021-10-28 17:03:00 -07:00
Harshavardhana a2cd3c9a1d
use ParseForm() to allow query param lookups once (#12900)
```
cpu: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
BenchmarkURLQueryForm
BenchmarkURLQueryForm-4         247099363                4.809 ns/op           0 B/op          0 allocs/op
BenchmarkURLQuery
BenchmarkURLQuery-4              2517624               462.1 ns/op           432 B/op          4 allocs/op
PASS
ok      github.com/minio/minio/cmd      3.848s
```
2021-08-07 22:43:01 -07:00
Harshavardhana 397637a042 remove x-frame-options HTTP header
fixes #12817
2021-07-28 08:57:22 -07:00
sakkiii 69e0faa278
add more XSS HTTP headers (#12256) 2021-07-19 16:05:02 -07:00
Harshavardhana 9dae5a7c85
fix: only redirect for GET/HEAD requests (#12702)
fixes #12701
2021-07-13 11:25:08 -07:00
Harshavardhana da0fd5f056
fix: allow customizing console redirection (#12665)
MinIO might be running inside proxies, and
console while being on another port might not be
reachable on a specific port behind such proxies.

For such scenarios customize the redirect URL
such that console can be redirected to correct
proxy endpoint instead.

fixes #12661
2021-07-09 14:27:09 -07:00
Harshavardhana c99d399d09
fix: handle redirects for specific resources (#12629) 2021-07-07 12:04:16 -07:00
Harshavardhana 8f1fe3b761
fix: --console-address when specified endpoints missing (#12534)
Additionally upgrade console dependency for reading
environment variables properly.
2021-06-20 23:04:47 -07:00
Harshavardhana cdeccb5510
feat: Deprecate embedded browser and import console (#12460)
This feature also changes the default port where
the browser is running, now the port has moved
to 9001 and it can be configured with

```
--console-address ":9001"
```
2021-06-17 20:27:04 -07:00
Harshavardhana 1f262daf6f
rename all remaining packages to internal/ (#12418)
This is to ensure that there are no projects
that try to import `minio/minio/pkg` into
their own repo. Any such common packages should
go to `https://github.com/minio/pkg`
2021-06-01 14:59:40 -07:00
Harshavardhana bf87c4b1e4
fix: no need to proxy if IAM not initialized (#12416)
IAM not initialized doesn't mean we can't still
read the content from the disk, we should just
allow the request to go-through if object layer
is initialized.
2021-06-01 12:23:13 -07:00
Harshavardhana f7a87b30bf Revert "deprecate embedded browser (#12163)"
This reverts commit 736d8cbac4.

Bring contrib files for older contributions
2021-04-30 08:50:39 -07:00
Harshavardhana 77f9c71133 Revert "redirect to console project for browser (#12172)"
This reverts commit 301669cf7b.

fixes #12179
2021-04-28 12:22:15 -07:00
Harshavardhana 301669cf7b
redirect to console project for browser (#12172) 2021-04-27 16:39:41 -07:00
Harshavardhana 736d8cbac4
deprecate embedded browser (#12163)
https://github.com/minio/console takes over the functionality for the
future object browser development

Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-27 10:52:12 -07:00
Harshavardhana 069432566f update license change for MinIO
Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-23 11:58:53 -07:00
Ritesh H Shukla 3ddd8b04d1
fix: handle unsupported APIs more granularly (#11674) 2021-03-30 23:19:36 -07:00
Harshavardhana b1bb3f7016
[feat]: implement GetBucketPolicyStatus API (#11673)
additionally also add more APIs in notImplemented
list, adjust routing rules appropriately
2021-03-01 23:10:33 -08:00
Krishna Srinivas 876b79b8d8
read-health check endpoint returns success if cluster can serve read requests (#11310) 2021-02-09 01:00:44 -08:00
Harshavardhana 6a95f412c9
avoid double CORS headers in federation (#11334)
CORS proxying adds double headers one
by the receiving server, one by proxied
server. Remove them before proxying
when 'Origin' header is found.
2021-01-23 18:27:23 -08:00
Harshavardhana 1b453728a3
initialize forwarder after init() to avoid crashes (#11330)
DNSCache dialer is a global value initialized in
init(), whereas `go` keeps `var =` before `init()`
, also we don't need to keep proxy routers as
global entities - register the forwarder as
necessary to avoid crashes.
2021-01-22 15:37:41 -08:00
Ritesh H Shukla b4add82bb6
Updated Prometheus metrics (#11141)
* Add metrics for nodes online and offline
* Add cluster capacity metrics
* Introduce v2 metrics
2021-01-18 20:35:38 -08:00
Anis Elleuch cb7fc99368
handlers: Avoid initializing a struct in each handler call (#11217) 2021-01-04 09:54:22 -08:00
Anis Elleuch dfd99b6d8f
handlers: Little bit more optimizations (#11211) 2021-01-04 00:01:06 -08:00
Harshavardhana 5c451d1690
update x/net/http2 to address few bugs (#11144)
additionally also configure http2 healthcheck
values to quickly detect unstable connections
and let them timeout.

also use single transport for proxying requests
2020-12-21 21:42:38 -08:00