MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 07:05:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
36fc2f98ed
minio/cmd/api-router.go

384 lines
18 KiB
Go
Raw Normal View History

routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
/*
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
* MinIO Cloud Storage, (C) 2016-2020 MinIO, Inc.
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
server: Move all the top level files into cmd folder. (#2490) This change brings a change which was done for the 'mc' package to allow for clean repo and have a cleaner github drop in experience.
2016-08-18 19:23:42 -04:00
package cmd
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
Add disk based edge caching support. (#5182) This PR adds disk based edge caching support for minio server. Cache settings can be configured in config.json to take list of disk drives, cache expiry in days and file patterns to exclude from cache or via environment variables MINIO_CACHE_DRIVES, MINIO_CACHE_EXCLUDE and MINIO_CACHE_EXPIRY Design assumes that Atime support is enabled and the list of cache drives is fixed. - Objects are cached on both GET and PUT/POST operations. - Expiry is used as hint to evict older entries from cache, or if 80% of cache capacity is filled. - When object storage backend is down, GET, LIST and HEAD operations fetch object seamlessly from cache. Current Limitations - Bucket policies are not cached, so anonymous operations are not supported in offline mode. - Objects are distributed using deterministic hashing among list of cache drives specified.If one or more drives go offline, or cache drive configuration is altered - performance could degrade to linear lookup. Fixes #4026
2018-03-28 17:14:06 -04:00
import (
skip subdomain from bucket DNS which start with `minio.domain` (#10390) extend host matcher to reject the host match
2020-09-09 12:57:37 -04:00
"net"
Add disk based edge caching support. (#5182) This PR adds disk based edge caching support for minio server. Cache settings can be configured in config.json to take list of disk drives, cache expiry in days and file patterns to exclude from cache or via environment variables MINIO_CACHE_DRIVES, MINIO_CACHE_EXCLUDE and MINIO_CACHE_EXPIRY Design assumes that Atime support is enabled and the list of cache drives is fixed. - Objects are cached on both GET and PUT/POST operations. - Expiry is used as hint to evict older entries from cache, or if 80% of cache capacity is filled. - When object storage backend is down, GET, LIST and HEAD operations fetch object seamlessly from cache. Current Limitations - Bucket policies are not cached, so anonymous operations are not supported in offline mode. - Objects are distributed using deterministic hashing among list of cache drives specified.If one or more drives go offline, or cache drive configuration is altered - performance could degrade to linear lookup. Fixes #4026
2018-03-28 17:14:06 -04:00
"net/http"
use package name correctly (#5827)
2018-04-21 22:23:54 -04:00
"github.com/gorilla/mux"
Ensure that we use constants everywhere (#7845) This allows for canonicalization of the strings throughout our code and provides a common space for all these constants to reside. This list is rather non-exhaustive but captures all the headers used in AWS S3 API operations
2019-07-03 01:34:32 -04:00
xhttp "github.com/minio/minio/cmd/http"
fix: cors handling after gorilla mux update (#9980) fixes #9979
2020-07-06 23:55:19 -04:00
"github.com/minio/minio/pkg/wildcard"
"github.com/rs/cors"
Add disk based edge caching support. (#5182) This PR adds disk based edge caching support for minio server. Cache settings can be configured in config.json to take list of disk drives, cache expiry in days and file patterns to exclude from cache or via environment variables MINIO_CACHE_DRIVES, MINIO_CACHE_EXCLUDE and MINIO_CACHE_EXPIRY Design assumes that Atime support is enabled and the list of cache drives is fixed. - Objects are cached on both GET and PUT/POST operations. - Expiry is used as hint to evict older entries from cache, or if 80% of cache capacity is filled. - When object storage backend is down, GET, LIST and HEAD operations fetch object seamlessly from cache. Current Limitations - Bucket policies are not cached, so anonymous operations are not supported in offline mode. - Objects are distributed using deterministic hashing among list of cache drives specified.If one or more drives go offline, or cache drive configuration is altered - performance could degrade to linear lookup. Fixes #4026
2018-03-28 17:14:06 -04:00
)
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
Enable build verification with race (#8583)
2019-12-02 18:54:26 -05:00
func newHTTPServerFn() *xhttp.Server {
globalObjLayerMutex.Lock()
defer globalObjLayerMutex.Unlock()
return globalHTTPServer
}
Bring in safe mode support (#8478) This PR refactors object layer handling such that upon failure in sub-system initialization server reaches a stage of safe-mode operation wherein only certain API operations are enabled and available. This allows for fixing many scenarios such as - incorrect configuration in vault, etcd, notification targets - missing files, incomplete config migrations unable to read encrypted content etc - any other issues related to notification, policies, lifecycle etc
2019-11-09 12:27:23 -05:00
func newObjectLayerFn() ObjectLayer {
globalObjLayerMutex.Lock()
defer globalObjLayerMutex.Unlock()
return globalObjectAPI
}
func newCachedObjectLayerFn() CacheObjectLayer {
globalObjLayerMutex.Lock()
defer globalObjLayerMutex.Unlock()
return globalCacheObjectAPI
}
ListObjects Metadata Caching (#10648) Design: https://gist.github.com/klauspost/025c09b48ed4a1293c917cecfabdf21c Gist of improvements: * Cross-server caching and listing will use the same data across servers and requests. * Lists can be arbitrarily resumed at a constant speed. * Metadata for all files scanned is stored for streaming retrieval. * The existing bloom filters controlled by the crawler is used for validating caches. * Concurrent requests for the same data (or parts of it) will not spawn additional walkers. * Listing a subdirectory of an existing recursive cache will use the cache. * All listing operations are fully streamable so the number of objects in a bucket no longer dictates the amount of memory. * Listings can be handled by any server within the cluster. * Caches are cleaned up when out of date or superseded by a more recent one.
2020-10-28 12:18:35 -04:00
func setObjectLayer(o ObjectLayer) {
globalObjLayerMutex.Lock()
globalObjectAPI = o
globalObjLayerMutex.Unlock()
}
storage/server/client: Enable storage server, enable client storage.
2016-04-12 15:45:15 -04:00
// objectAPIHandler implements and provides http handlers for S3 API.
type objectAPIHandlers struct {
distribute: Make server work with multiple remote disks This change initializes rpc servers associated with disks that are local. It makes object layer initialization on demand, namely on the first request to the object layer. Also adds lock RPC service vendorized minio/dsync
2016-07-31 17:11:14 -04:00
ObjectAPI func() ObjectLayer
Add disk based edge caching support. (#5182) This PR adds disk based edge caching support for minio server. Cache settings can be configured in config.json to take list of disk drives, cache expiry in days and file patterns to exclude from cache or via environment variables MINIO_CACHE_DRIVES, MINIO_CACHE_EXCLUDE and MINIO_CACHE_EXPIRY Design assumes that Atime support is enabled and the list of cache drives is fixed. - Objects are cached on both GET and PUT/POST operations. - Expiry is used as hint to evict older entries from cache, or if 80% of cache capacity is filled. - When object storage backend is down, GET, LIST and HEAD operations fetch object seamlessly from cache. Current Limitations - Bucket policies are not cached, so anonymous operations are not supported in offline mode. - Objects are distributed using deterministic hashing among list of cache drives specified.If one or more drives go offline, or cache drive configuration is altered - performance could degrade to linear lookup. Fixes #4026
2018-03-28 17:14:06 -04:00
CacheAPI func() CacheObjectLayer
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
}
skip subdomain from bucket DNS which start with `minio.domain` (#10390) extend host matcher to reject the host match
2020-09-09 12:57:37 -04:00
// getHost tries its best to return the request host.
// According to section 14.23 of RFC 2616 the Host header
// can include the port number if the default value of 80 is not used.
func getHost(r *http.Request) string {
if r.URL.IsAbs() {
return r.URL.Host
}
return r.Host
}
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
// registerAPIRouter - registers S3 compatible APIs.
fix: S3 gateway doesn't support full passthrough for encryption (#10484) The entire encryption layer is dependent on the fact that KMS should be configured for S3 encryption to work properly and we only support passing the headers as is to the backend for encryption only if KMS is configured. Make sure that this predictability is maintained, currently the code was allowing encryption to go through and fail at later to indicate that KMS was not configured. We should simply reply "NotImplemented" if KMS is not configured, this allows clients to simply proceed with their tests.
2020-09-15 16:57:15 -04:00
func registerAPIRouter(router *mux.Router) {
server: Add more elaborate startup messages. (#2731) These messages based on our prep stage during XL and prints more informative message regarding drive information. This change also does a much needed refactoring.
2016-10-05 15:48:07 -04:00
// Initialize API.
api := objectAPIHandlers{
Bring in safe mode support (#8478) This PR refactors object layer handling such that upon failure in sub-system initialization server reaches a stage of safe-mode operation wherein only certain API operations are enabled and available. This allows for fixing many scenarios such as - incorrect configuration in vault, etcd, notification targets - missing files, incomplete config migrations unable to read encrypted content etc - any other issues related to notification, policies, lifecycle etc
2019-11-09 12:27:23 -05:00
ObjectAPI: newObjectLayerFn,
CacheAPI: newCachedObjectLayerFn,
server: Add more elaborate startup messages. (#2731) These messages based on our prep stage during XL and prints more informative message regarding drive information. This change also does a much needed refactoring.
2016-10-05 15:48:07 -04:00
}
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
// API Router
Use const slashSeparator instead of "/" everywhere (#8028)
2019-08-06 15:08:58 -04:00
apiRouter := router.PathPrefix(SlashSeparator).Subrouter()
skip subdomain from bucket DNS which start with `minio.domain` (#10390) extend host matcher to reject the host match
2020-09-09 12:57:37 -04:00
use package name correctly (#5827)
2018-04-21 22:23:54 -04:00
var routers []*mux.Router
Support multiple-domains in MINIO_DOMAIN (#7274) Fixes #7173
2019-02-22 22:18:01 -05:00
for _, domainName := range globalDomainNames {
skip subdomain from bucket DNS which start with `minio.domain` (#10390) extend host matcher to reject the host match
2020-09-09 12:57:37 -04:00
if IsKubernetes() {
routers = append(routers, apiRouter.MatcherFunc(func(r *http.Request, match *mux.RouteMatch) bool {
fix: host extraction without port
2020-09-23 15:10:09 -04:00
host, _, err := net.SplitHostPort(getHost(r))
if err != nil {
host = r.Host
}
skip subdomain from bucket DNS which start with `minio.domain` (#10390) extend host matcher to reject the host match
2020-09-09 12:57:37 -04:00
// Make sure to skip matching minio.<domain>` this is
// specifically meant for operator/k8s deployment
// The reason we need to skip this is for a special
// usecase where we need to make sure that
// minio.<namespace>.svc.<cluster_domain> is ignored
// by the bucketDNS style to ensure that path style
// is available and honored at this domain.
//
// All other `<bucket>.<namespace>.svc.<cluster_domain>`
// makes sure that buckets are routed through this matcher
// to match for `<bucket>`
return host != minioReservedBucket+"."+domainName
}).Host("{bucket:.+}."+domainName).Subrouter())
} else {
routers = append(routers, apiRouter.Host("{bucket:.+}."+domainName).Subrouter())
}
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
}
routers = append(routers, apiRouter.PathPrefix("/{bucket}").Subrouter())
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
for _, bucket := range routers {
// Object operations
// HeadObject
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodHead).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("headobject", httpTraceAll(maxClients(api.HeadObjectHandler))))
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// CopyObjectPart
skip subdomain from bucket DNS which start with `minio.domain` (#10390) extend host matcher to reject the host match
2020-09-09 12:57:37 -04:00
bucket.Methods(http.MethodPut).Path("/{object:.+}").
HeadersRegexp(xhttp.AmzCopySource, ".*?(\\/|%2F).*?").
HandlerFunc(maxClients(collectAPIStats("copyobjectpart", httpTraceAll(api.CopyObjectPartHandler)))).
Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// PutObjectPart
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putobjectpart", httpTraceHdrs(maxClients(api.PutObjectPartHandler)))).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
Add listBucketObjectsVersions implementation (#8093) This API implementation simply behaves like listObjects() but returns back single version for each object, this implementation should be considered dummy it is only meant for some applications which rely on this.
2019-08-19 17:02:54 -04:00
// ListObjectParts
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("listobjectparts", httpTraceAll(maxClients(api.ListObjectPartsHandler)))).Queries("uploadId", "{uploadId:.*}")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// CompleteMultipartUpload
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("completemutipartupload", httpTraceAll(maxClients(api.CompleteMultipartUploadHandler)))).Queries("uploadId", "{uploadId:.*}")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// NewMultipartUpload
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("newmultipartupload", httpTraceAll(maxClients(api.NewMultipartUploadHandler)))).Queries("uploads", "")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// AbortMultipartUpload
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("abortmultipartupload", httpTraceAll(maxClients(api.AbortMultipartUploadHandler)))).Queries("uploadId", "{uploadId:.*}")
Fix s3cmd issue with ACL handling (#5999) With the implementation of dummy GET ACL handlers, tools like s3cmd perform few operations which causes the ACL call to be invoked. Make sure that in our router configuration GET?acl comes before actual GET call to facilitate this dummy call.
2018-05-31 22:43:50 -04:00
// GetObjectACL - this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getobjectacl", httpTraceHdrs(maxClients(api.GetObjectACLHandler)))).Queries("acl", "")
acl: Support PUT calls with success for 'private' ACL's (#9000) Add dummy calls which respond success when ACL's are set to be private and fails, if user tries to change them from their default 'private' Some applications such as nuxeo may have an unnecessary requirement for this operation, we support this anyways such that don't have to fully implement the functionality just that we can respond with success for default ACLs
2020-02-16 01:07:52 -05:00
// PutObjectACL - this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putobjectacl", httpTraceHdrs(maxClients(api.PutObjectACLHandler)))).Queries("acl", "")
Add ObjectTagging Support (#8754) This PR adds support for AWS S3 ObjectTagging API as explained here https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
2020-01-20 11:45:59 -05:00
// GetObjectTagging
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getobjecttagging", httpTraceHdrs(maxClients(api.GetObjectTaggingHandler)))).Queries("tagging", "")
Add ObjectTagging Support (#8754) This PR adds support for AWS S3 ObjectTagging API as explained here https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
2020-01-20 11:45:59 -05:00
// PutObjectTagging
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putobjecttagging", httpTraceHdrs(maxClients(api.PutObjectTaggingHandler)))).Queries("tagging", "")
Add ObjectTagging Support (#8754) This PR adds support for AWS S3 ObjectTagging API as explained here https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
2020-01-20 11:45:59 -05:00
// DeleteObjectTagging
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deleteobjecttagging", httpTraceHdrs(maxClients(api.DeleteObjectTaggingHandler)))).Queries("tagging", "")
S3 Select API Support for CSV (#6127) Add support for trivial where clause cases
2018-08-15 06:30:19 -04:00
// SelectObjectContent
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("selectobjectcontent", httpTraceHdrs(maxClients(api.SelectObjectContentHandler)))).Queries("select", "").Queries("select-type", "2")
Add object retention at the per object (#8528) level - this PR builds on #8120 which added PutBucketObjectLockConfiguration and GetBucketObjectLockConfiguration APIS This PR implements PutObjectRetention, GetObjectRetention API and enhances PUT and GET API operations to display governance metadata if permissions allow.
2019-11-20 16:18:09 -05:00
// GetObjectRetention
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getobjectretention", httpTraceAll(maxClients(api.GetObjectRetentionHandler)))).Queries("retention", "")
Add support for object locking with legal hold. (#8634)
2020-01-16 18:41:56 -05:00
// GetObjectLegalHold
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getobjectlegalhold", httpTraceAll(maxClients(api.GetObjectLegalHoldHandler)))).Queries("legal-hold", "")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// GetObject
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getobject", httpTraceHdrs(maxClients(api.GetObjectHandler))))
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// CopyObject
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
bucket.Methods(http.MethodPut).Path("/{object:.+}").HeadersRegexp(xhttp.AmzCopySource, ".*?(\\/|%2F).*?").HandlerFunc(
collectAPIStats("copyobject", httpTraceAll(maxClients(api.CopyObjectHandler))))
Add object retention at the per object (#8528) level - this PR builds on #8120 which added PutBucketObjectLockConfiguration and GetBucketObjectLockConfiguration APIS This PR implements PutObjectRetention, GetObjectRetention API and enhances PUT and GET API operations to display governance metadata if permissions allow.
2019-11-20 16:18:09 -05:00
// PutObjectRetention
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putobjectretention", httpTraceAll(maxClients(api.PutObjectRetentionHandler)))).Queries("retention", "")
Add support for object locking with legal hold. (#8634)
2020-01-16 18:41:56 -05:00
// PutObjectLegalHold
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putobjectlegalhold", httpTraceAll(maxClients(api.PutObjectLegalHoldHandler)))).Queries("legal-hold", "")
Add support for object locking with legal hold. (#8634)
2020-01-16 18:41:56 -05:00
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// PutObject
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putobject", httpTraceHdrs(maxClients(api.PutObjectHandler))))
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// DeleteObject
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deleteobject", httpTraceAll(maxClients(api.DeleteObjectHandler))))
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
/// Bucket operations
// GetBucketLocation
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketlocation", httpTraceAll(maxClients(api.GetBucketLocationHandler)))).Queries("location", "")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// GetBucketPolicy
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketpolicy", httpTraceAll(maxClients(api.GetBucketPolicyHandler)))).Queries("policy", "")
Add initial support for bucket lifecycle (#7563) This PR is based off @sinhaashish's PR for object lifecycle management, which includes support only for, - Expiration of object - Filter using object prefix (_not_ object tags) N B the code for actual expiration of objects will be included in a subsequent PR.
2019-07-19 16:20:33 -04:00
// GetBucketLifecycle
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketlifecycle", httpTraceAll(maxClients(api.GetBucketLifecycleHandler)))).Queries("lifecycle", "")
Add support for bucket encryption feature (#8890) - pkg/bucket/encryption provides support for handling bucket encryption configuration - changes under cmd/ provide support for AES256 algorithm only Co-Authored-By: Poorna <poornas@users.noreply.github.com> Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-05 04:42:34 -05:00
// GetBucketEncryption
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketencryption", httpTraceAll(maxClients(api.GetBucketEncryptionHandler)))).Queries("encryption", "")
Add support for server side bucket replication (#9882)
2020-07-21 20:49:56 -04:00
// GetBucketObjectLockConfig
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketobjectlockconfiguration", httpTraceAll(maxClients(api.GetBucketObjectLockConfigHandler)))).Queries("object-lock", "")
Add support for server side bucket replication (#9882)
2020-07-21 20:49:56 -04:00
// GetBucketReplicationConfig
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketreplicationconfiguration", httpTraceAll(maxClients(api.GetBucketReplicationConfigHandler)))).Queries("replication", "")
Add support for server side bucket replication (#9882)
2020-07-21 20:49:56 -04:00
// GetBucketVersioning
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketversioning", httpTraceAll(maxClients(api.GetBucketVersioningHandler)))).Queries("versioning", "")
Add support for server side bucket replication (#9882)
2020-07-21 20:49:56 -04:00
// GetBucketNotification
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketnotification", httpTraceAll(maxClients(api.GetBucketNotificationHandler)))).Queries("notification", "")
Add support for server side bucket replication (#9882)
2020-07-21 20:49:56 -04:00
// ListenNotification
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
bucket.Methods(http.MethodGet).HandlerFunc(
collectAPIStats("listennotification", httpTraceAll(maxClients(api.ListenNotificationHandler)))).Queries("events", "{events:.*}")
Implement a dummy GetBucketACL for legacy reasons (#5906) GetBucketACL call returns empty for all GET in ACL requests, the primary purpose of this PR is to provide legacy API support for legacy applications. Fixes #5706
2018-05-10 00:02:26 -04:00
Implements dummy tagging handlers for Terraform (#7040)
2019-02-08 19:18:13 -05:00
// Dummy Bucket Calls
Implement a dummy GetBucketACL for legacy reasons (#5906) GetBucketACL call returns empty for all GET in ACL requests, the primary purpose of this PR is to provide legacy API support for legacy applications. Fixes #5706
2018-05-10 00:02:26 -04:00
// GetBucketACL -- this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketacl", httpTraceAll(maxClients(api.GetBucketACLHandler)))).Queries("acl", "")
acl: Support PUT calls with success for 'private' ACL's (#9000) Add dummy calls which respond success when ACL's are set to be private and fails, if user tries to change them from their default 'private' Some applications such as nuxeo may have an unnecessary requirement for this operation, we support this anyways such that don't have to fully implement the functionality just that we can respond with success for default ACLs
2020-02-16 01:07:52 -05:00
// PutBucketACL -- this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbucketacl", httpTraceAll(maxClients(api.PutBucketACLHandler)))).Queries("acl", "")
Implements dummy tagging handlers for Terraform (#7040)
2019-02-08 19:18:13 -05:00
// GetBucketCors - this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketcors", httpTraceAll(maxClients(api.GetBucketCorsHandler)))).Queries("cors", "")
Implements dummy tagging handlers for Terraform (#7040)
2019-02-08 19:18:13 -05:00
// GetBucketWebsiteHandler - this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketwebsite", httpTraceAll(maxClients(api.GetBucketWebsiteHandler)))).Queries("website", "")
Implements dummy tagging handlers for Terraform (#7040)
2019-02-08 19:18:13 -05:00
// GetBucketAccelerateHandler - this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketaccelerate", httpTraceAll(maxClients(api.GetBucketAccelerateHandler)))).Queries("accelerate", "")
Implements dummy tagging handlers for Terraform (#7040)
2019-02-08 19:18:13 -05:00
// GetBucketRequestPaymentHandler - this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketrequestpayment", httpTraceAll(maxClients(api.GetBucketRequestPaymentHandler)))).Queries("requestPayment", "")
Implements dummy tagging handlers for Terraform (#7040)
2019-02-08 19:18:13 -05:00
// GetBucketLoggingHandler - this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketlogging", httpTraceAll(maxClients(api.GetBucketLoggingHandler)))).Queries("logging", "")
Implements dummy tagging handlers for Terraform (#7040)
2019-02-08 19:18:13 -05:00
// GetBucketLifecycleHandler - this is a dummy call.
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbucketlifecycle", httpTraceAll(maxClients(api.GetBucketLifecycleHandler)))).Queries("lifecycle", "")
add bucket tagging support (#9389) This patch also simplifies object tagging support
2020-05-05 17:18:13 -04:00
// GetBucketTaggingHandler
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("getbuckettagging", httpTraceAll(maxClients(api.GetBucketTaggingHandler)))).Queries("tagging", "")
Implements dummy tagging handlers for Terraform (#7040)
2019-02-08 19:18:13 -05:00
//DeleteBucketWebsiteHandler
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodDelete).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deletebucketwebsite", httpTraceAll(maxClients(api.DeleteBucketWebsiteHandler)))).Queries("website", "")
Implements dummy tagging handlers for Terraform (#7040)
2019-02-08 19:18:13 -05:00
// DeleteBucketTaggingHandler
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodDelete).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deletebuckettagging", httpTraceAll(maxClients(api.DeleteBucketTaggingHandler)))).Queries("tagging", "")
Implement a dummy GetBucketACL for legacy reasons (#5906) GetBucketACL call returns empty for all GET in ACL requests, the primary purpose of this PR is to provide legacy API support for legacy applications. Fixes #5706
2018-05-10 00:02:26 -04:00
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// ListMultipartUploads
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("listmultipartuploads", httpTraceAll(maxClients(api.ListMultipartUploadsHandler)))).Queries("uploads", "")
Enhance ListObjectsV2 API to return UserDefined metadata (#8539)
2019-11-21 04:54:49 -05:00
// ListObjectsV2M
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("listobjectsv2M", httpTraceAll(maxClients(api.ListObjectsV2MHandler)))).Queries("list-type", "2", "metadata", "true")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// ListObjectsV2
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("listobjectsv2", httpTraceAll(maxClients(api.ListObjectsV2Handler)))).Queries("list-type", "2")
Support bucket versioning (#9377) - Implement a new xl.json 2.0.0 format to support, this moves the entire marshaling logic to POSIX layer, top layer always consumes a common FileInfo construct which simplifies the metadata reads. - Implement list object versions - Migrate to siphash from crchash for new deployments for object placements. Fixes #2111
2020-06-12 23:04:01 -04:00
// ListObjectVersions
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("listobjectversions", httpTraceAll(maxClients(api.ListObjectVersionsHandler)))).Queries("versions", "")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// ListObjectsV1 (Legacy)
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodGet).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("listobjectsv1", httpTraceAll(maxClients(api.ListObjectsV1Handler))))
Add initial support for bucket lifecycle (#7563) This PR is based off @sinhaashish's PR for object lifecycle management, which includes support only for, - Expiration of object - Filter using object prefix (_not_ object tags) N B the code for actual expiration of objects will be included in a subsequent PR.
2019-07-19 16:20:33 -04:00
// PutBucketLifecycle
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbucketlifecycle", httpTraceAll(maxClients(api.PutBucketLifecycleHandler)))).Queries("lifecycle", "")
Add support for server side bucket replication (#9882)
2020-07-21 20:49:56 -04:00
// PutBucketReplicationConfig
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbucketreplicationconfiguration", httpTraceAll(maxClients(api.PutBucketReplicationConfigHandler)))).Queries("replication", "")
Add support for server side bucket replication (#9882)
2020-07-21 20:49:56 -04:00
// GetObjectRetention
Add support for bucket encryption feature (#8890) - pkg/bucket/encryption provides support for handling bucket encryption configuration - changes under cmd/ provide support for AES256 algorithm only Co-Authored-By: Poorna <poornas@users.noreply.github.com> Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-05 04:42:34 -05:00
// PutBucketEncryption
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbucketencryption", httpTraceAll(maxClients(api.PutBucketEncryptionHandler)))).Queries("encryption", "")
Add support for bucket encryption feature (#8890) - pkg/bucket/encryption provides support for handling bucket encryption configuration - changes under cmd/ provide support for AES256 algorithm only Co-Authored-By: Poorna <poornas@users.noreply.github.com> Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-05 04:42:34 -05:00
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// PutBucketPolicy
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbucketpolicy", httpTraceAll(maxClients(api.PutBucketPolicyHandler)))).Queries("policy", "")
Add initial support for bucket lifecycle (#7563) This PR is based off @sinhaashish's PR for object lifecycle management, which includes support only for, - Expiration of object - Filter using object prefix (_not_ object tags) N B the code for actual expiration of objects will be included in a subsequent PR.
2019-07-19 16:20:33 -04:00
Add Get/Put Bucket Lock Configuration API support (#8120) This feature implements [PUT Bucket object lock configuration][1] and [GET Bucket object lock configuration][2]. After object lock configuration is set, existing and new objects are set to WORM for specified duration. Currently Governance mode works exactly like Compliance mode. Fixes #8101 [1] https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTObjectLockConfiguration.html [2] https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETObjectLockConfiguration.html
2019-11-12 17:50:18 -05:00
// PutBucketObjectLockConfig
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbucketobjectlockconfig", httpTraceAll(maxClients(api.PutBucketObjectLockConfigHandler)))).Queries("object-lock", "")
add bucket tagging support (#9389) This patch also simplifies object tagging support
2020-05-05 17:18:13 -04:00
// PutBucketTaggingHandler
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbuckettagging", httpTraceAll(maxClients(api.PutBucketTaggingHandler)))).Queries("tagging", "")
Add Get/Put Bucket Lock Configuration API support (#8120) This feature implements [PUT Bucket object lock configuration][1] and [GET Bucket object lock configuration][2]. After object lock configuration is set, existing and new objects are set to WORM for specified duration. Currently Governance mode works exactly like Compliance mode. Fixes #8101 [1] https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTObjectLockConfiguration.html [2] https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETObjectLockConfiguration.html
2019-11-12 17:50:18 -05:00
// PutBucketVersioning
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbucketversioning", httpTraceAll(maxClients(api.PutBucketVersioningHandler)))).Queries("versioning", "")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// PutBucketNotification
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbucketnotification", httpTraceAll(maxClients(api.PutBucketNotificationHandler)))).Queries("notification", "")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// PutBucket
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPut).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("putbucket", httpTraceAll(maxClients(api.PutBucketHandler))))
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// HeadBucket
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodHead).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("headbucket", httpTraceAll(maxClients(api.HeadBucketHandler))))
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// PostPolicy
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPost).HeadersRegexp(xhttp.ContentType, "multipart/form-data*").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("postpolicybucket", httpTraceHdrs(maxClients(api.PostPolicyBucketHandler))))
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// DeleteMultipleObjects
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodPost).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deletemultipleobjects", httpTraceAll(maxClients(api.DeleteMultipleObjectsHandler)))).Queries("delete", "")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// DeleteBucketPolicy
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodDelete).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deletebucketpolicy", httpTraceAll(maxClients(api.DeleteBucketPolicyHandler)))).Queries("policy", "")
Add support for server side bucket replication (#9882)
2020-07-21 20:49:56 -04:00
// DeleteBucketReplication
bucket.Methods(http.MethodDelete).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deletebucketreplicationconfiguration", httpTraceAll(maxClients(api.DeleteBucketReplicationConfigHandler)))).Queries("replication", "")
Add initial support for bucket lifecycle (#7563) This PR is based off @sinhaashish's PR for object lifecycle management, which includes support only for, - Expiration of object - Filter using object prefix (_not_ object tags) N B the code for actual expiration of objects will be included in a subsequent PR.
2019-07-19 16:20:33 -04:00
// DeleteBucketLifecycle
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodDelete).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deletebucketlifecycle", httpTraceAll(maxClients(api.DeleteBucketLifecycleHandler)))).Queries("lifecycle", "")
Add support for bucket encryption feature (#8890) - pkg/bucket/encryption provides support for handling bucket encryption configuration - changes under cmd/ provide support for AES256 algorithm only Co-Authored-By: Poorna <poornas@users.noreply.github.com> Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-05 04:42:34 -05:00
// DeleteBucketEncryption
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodDelete).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deletebucketencryption", httpTraceAll(maxClients(api.DeleteBucketEncryptionHandler)))).Queries("encryption", "")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
// DeleteBucket
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
bucket.Methods(http.MethodDelete).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("deletebucket", httpTraceAll(maxClients(api.DeleteBucketHandler))))
Add support for ILM transition (#10565) This PR adds transition support for ILM to transition data to another MinIO target represented by a storage class ARN. Subsequent GET or HEAD for that object will be streamed from the transition tier. If PostRestoreObject API is invoked, the transitioned object can be restored for duration specified to the source cluster.
2020-11-12 15:12:09 -05:00
// PostRestoreObject
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("restoreobject", httpTraceAll(maxClients(api.PostRestoreObjectHandler)))).Queries("restore", "")
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
}
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
/// Root operation
feat: Add notification support for bucketCreates and removal (#10075)
2020-07-20 15:52:49 -04:00
// ListenNotification
apiRouter.Methods(http.MethodGet).Path(SlashSeparator).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("listennotification", httpTraceAll(maxClients(api.ListenNotificationHandler)))).Queries("events", "{events:.*}")
feat: Add notification support for bucketCreates and removal (#10075)
2020-07-20 15:52:49 -04:00
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
// ListBuckets
Add rate limiter for S3 API layer (#9196) - total number of S3 API calls per server - maximum wait duration for any S3 API call This implementation is primarily meant for situations where HDDs are not capable enough to handle the incoming workload and there is no way to throttle the client. This feature allows MinIO server to throttle itself such that we do not overwhelm the HDDs.
2020-03-24 15:43:40 -04:00
apiRouter.Methods(http.MethodGet).Path(SlashSeparator).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("listbuckets", httpTraceAll(maxClients(api.ListBucketsHandler))))
Virtual host style S3 requests (#5095)
2017-11-14 19:56:24 -05:00
add missing signature v2 query params (#9670)
2020-05-21 21:51:23 -04:00
// S3 browser with signature v4 adds '//' for ListBuckets request, so rather
// than failing with UnknownAPIRequest we simply handle it for now.
apiRouter.Methods(http.MethodGet).Path(SlashSeparator + SlashSeparator).HandlerFunc(
fix: admin trace throttled requests (#11192)
2020-12-31 00:04:55 -05:00
collectAPIStats("listbuckets", httpTraceAll(maxClients(api.ListBucketsHandler))))
add missing signature v2 query params (#9670)
2020-05-21 21:51:23 -04:00
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-04 12:30:59 -05:00
// If none of the routes match add default error handler routes
Remove redundant http.HandlerFunc type conversion. (#10576)
2020-09-28 16:33:49 -04:00
apiRouter.NotFoundHandler = collectAPIStats("notfound", httpTraceAll(errorResponseHandler))
ListObjects Metadata Caching (#10648) Design: https://gist.github.com/klauspost/025c09b48ed4a1293c917cecfabdf21c Gist of improvements: * Cross-server caching and listing will use the same data across servers and requests. * Lists can be arbitrarily resumed at a constant speed. * Metadata for all files scanned is stored for streaming retrieval. * The existing bloom filters controlled by the crawler is used for validating caches. * Concurrent requests for the same data (or parts of it) will not spawn additional walkers. * Listing a subdirectory of an existing recursive cache will use the cache. * All listing operations are fully streamable so the number of objects in a bucket no longer dictates the amount of memory. * Listings can be handled by any server within the cluster. * Caches are cleaned up when out of date or superseded by a more recent one.
2020-10-28 12:18:35 -04:00
apiRouter.MethodNotAllowedHandler = collectAPIStats("methodnotallowed", httpTraceAll(methodNotAllowedHandler("S3")))
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-04 12:30:59 -05:00
routers: Move API and Web routers into their own files. This is done to ensure we have a clean way to add new routers such as - diskRouter - configRouter - lockRouter
2016-03-27 15:37:21 -04:00
}
fix: cors handling after gorilla mux update (#9980) fixes #9979
2020-07-06 23:55:19 -04:00
fix: cors handling again for not just OPTIONS request (#10025) CORS is notorious requires specific headers to be handled appropriately in request and response, using cors package as part of handlerFunc() for options method lacks the necessary control this package needs to add headers.
2020-07-12 13:56:57 -04:00
// corsHandler handler for CORS (Cross Origin Resource Sharing)
func corsHandler(handler http.Handler) http.Handler {
fix: cors handling after gorilla mux update (#9980) fixes #9979
2020-07-06 23:55:19 -04:00
commonS3Headers := []string{
xhttp.Date,
xhttp.ETag,
xhttp.ServerInfo,
xhttp.Connection,
xhttp.AcceptRanges,
xhttp.ContentRange,
xhttp.ContentEncoding,
xhttp.ContentLength,
xhttp.ContentType,
fix: add missing content-disposition from CORS handler (#10137)
2020-07-27 12:03:38 -04:00
xhttp.ContentDisposition,
fix: add missing response headers to CORS handler (#10124)
2020-07-24 03:46:51 -04:00
xhttp.LastModified,
xhttp.ContentLanguage,
xhttp.CacheControl,
xhttp.RetryAfter,
xhttp.AmzBucketRegion,
xhttp.Expires,
fix: cors handling after gorilla mux update (#9980) fixes #9979
2020-07-06 23:55:19 -04:00
"X-Amz*",
"x-amz*",
"*",
}
fix: cors handling again for not just OPTIONS request (#10025) CORS is notorious requires specific headers to be handled appropriately in request and response, using cors package as part of handlerFunc() for options method lacks the necessary control this package needs to add headers.
2020-07-12 13:56:57 -04:00
return cors.New(cors.Options{
fix: cors handling after gorilla mux update (#9980) fixes #9979
2020-07-06 23:55:19 -04:00
AllowOriginFunc: func(origin string) bool {
for _, allowedOrigin := range globalAPIConfig.getCorsAllowOrigins() {
if wildcard.MatchSimple(allowedOrigin, origin) {
return true
}
}
return false
},
AllowedMethods: []string{
http.MethodGet,
http.MethodPut,
http.MethodHead,
http.MethodPost,
http.MethodDelete,
http.MethodOptions,
http.MethodPatch,
},
AllowedHeaders: commonS3Headers,
ExposedHeaders: commonS3Headers,
AllowCredentials: true,
fix: cors handling again for not just OPTIONS request (#10025) CORS is notorious requires specific headers to be handled appropriately in request and response, using cors package as part of handlerFunc() for options method lacks the necessary control this package needs to add headers.
2020-07-12 13:56:57 -04:00
}).Handler(handler)
fix: cors handling after gorilla mux update (#9980) fixes #9979
2020-07-06 23:55:19 -04:00
}
Reference in New Issue Copy Permalink