MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
minio/cmd/storage-rest-common.go

87 lines
2.9 KiB
Go
Raw Normal View History

remove overzealous check during HEAD() (#19940) due to a historic bug in CopyObject() where an inlined object loses its metadata, the check causes an incorrect fallback verifying data-dir. CopyObject() bug was fixed in ffa91f97942 however the occurrence of this problem is historic, so the aforementioned check is stretching too much. Bonus: simplify fileInfoRaw() to read xl.json as well, also recreate buckets properly.
2024-06-17 10:29:18 -04:00
// Copyright (c) 2015-2024 MinIO, Inc.
update license change for MinIO Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-18 15:41:13 -04:00
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
package cmd
perf: websocket grid connectivity for all internode communication (#18461) This PR adds a WebSocket grid feature that allows servers to communicate via a single two-way connection. There are two request types: * Single requests, which are `[]byte => ([]byte, error)`. This is for efficient small roundtrips with small payloads. * Streaming requests which are `[]byte, chan []byte => chan []byte (and error)`, which allows for different combinations of full two-way streams with an initial payload. Only a single stream is created between two machines - and there is, as such, no server/client relation since both sides can initiate and handle requests. Which server initiates the request is decided deterministically on the server names. Requests are made through a mux client and server, which handles message passing, congestion, cancelation, timeouts, etc. If a connection is lost, all requests are canceled, and the calling server will try to reconnect. Registered handlers can operate directly on byte slices or use a higher-level generics abstraction. There is no versioning of handlers/clients, and incompatible changes should be handled by adding new handlers. The request path can be changed to a new one for any protocol changes. First, all servers create a "Manager." The manager must know its address as well as all remote addresses. This will manage all connections. To get a connection to any remote, ask the manager to provide it given the remote address using. ``` func (m *Manager) Connection(host string) *Connection ``` All serverside handlers must also be registered on the manager. This will make sure that all incoming requests are served. The number of in-flight requests and responses must also be given for streaming requests. The "Connection" returned manages the mux-clients. Requests issued to the connection will be sent to the remote. * `func (c *Connection) Request(ctx context.Context, h HandlerID, req []byte) ([]byte, error)` performs a single request and returns the result. Any deadline provided on the request is forwarded to the server, and canceling the context will make the function return at once. * `func (c *Connection) NewStream(ctx context.Context, h HandlerID, payload []byte) (st *Stream, err error)` will initiate a remote call and send the initial payload. ```Go // A Stream is a two-way stream. // All responses *must* be read by the caller. // If the call is canceled through the context, //The appropriate error will be returned. type Stream struct { // Responses from the remote server. // Channel will be closed after an error or when the remote closes. // All responses *must* be read by the caller until either an error is returned or the channel is closed. // Canceling the context will cause the context cancellation error to be returned. Responses <-chan Response // Requests sent to the server. // If the handler is defined with 0 incoming capacity this will be nil. // Channel *must* be closed to signal the end of the stream. // If the request context is canceled, the stream will no longer process requests. Requests chan<- []byte } type Response struct { Msg []byte Err error } ``` There are generic versions of the server/client handlers that allow the use of type safe implementations for data types that support msgpack marshal/unmarshal.
2023-11-20 20:09:35 -05:00
//go:generate msgp -file $GOFILE -unexported
admin/heal: Fix deep healing to heal objects under more conditions (#8321) - Heal if the part.1 is truncated from its original size - Heal if the part.1 fails while being verified in between - Heal if the part.1 fails while being at a certain offset Other cleanups include make sure to flush the HTTP responses properly from storage-rest-server, avoid using 'defer' to improve call latency. 'defer' incurs latency avoid them in our hot-paths such as storage-rest handlers. Fixes #8319
2019-10-01 16:12:15 -04:00
const (
implement a safer completeMultipart implementation (#20227) - optimize writing part.N.meta by writing both part.N and its meta in sequence without network component. - remove part.N.meta, part.N which were partially success ful, in quorum loss situations during renamePart() - allow for strict read quorum check arbitrated via ETag for the given part number, this makes it double safer upon final commit. - return an appropriate error when read quorum is missing, instead of returning InvalidPart{}, which is non-retryable error. This kind of situation can happen when many nodes are going offline in rotation, an example of such a restart() behavior is statefulset updates in k8s. fixes #20091
2024-08-12 04:38:15 -04:00
storageRESTVersion = "v63" // Introduce RenamePart and ReadParts API
Add data usage collect with its new admin API (#8553) Admin data usage info API returns the following (Only FS & XL, for now) - Number of buckets - Number of objects - The total size of objects - Objects histogram - Bucket sizes
2019-12-12 09:02:37 -05:00
storageRESTVersionPrefix = SlashSeparator + storageRESTVersion
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-04 12:30:59 -05:00
storageRESTPrefix = minioReservedBucketPath + "/storage"
admin/heal: Fix deep healing to heal objects under more conditions (#8321) - Heal if the part.1 is truncated from its original size - Heal if the part.1 fails while being verified in between - Heal if the part.1 fails while being at a certain offset Other cleanups include make sure to flush the HTTP responses properly from storage-rest-server, avoid using 'defer' to improve call latency. 'defer' incurs latency avoid them in our hot-paths such as storage-rest handlers. Fixes #8319
2019-10-01 16:12:15 -04:00
)
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
const (
remove all older unused APIs (#18769)
2024-01-17 23:41:23 -05:00
storageRESTMethodHealth = "/health"
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
change Read* calls over net/http to move to http.MethodGet (#20173) - ReadVersion - ReadFile - ReadXL Further changes include to - Compact internode resource RPC paths - Compact internode query params To optimize on parsing by gorilla/mux as the length of this string increases latency in gorilla/mux - reduce to a meaningful string.
2024-07-29 04:00:12 -04:00
storageRESTMethodAppendFile = "/afile"
storageRESTMethodCreateFile = "/cfile"
storageRESTMethodWriteAll = "/wall"
storageRESTMethodReadVersion = "/rver"
storageRESTMethodReadXL = "/rxl"
storageRESTMethodReadAll = "/rall"
storageRESTMethodReadFile = "/rfile"
storageRESTMethodReadFileStream = "/rfilest"
storageRESTMethodListDir = "/ls"
storageRESTMethodDeleteVersions = "/dvers"
storageRESTMethodRenameFile = "/rfile"
storageRESTMethodVerifyFile = "/vfile"
storageRESTMethodStatInfoFile = "/sfile"
storageRESTMethodReadMultiple = "/rmpl"
storageRESTMethodCleanAbandoned = "/cln"
add DeleteBulk support, add sufficient deadlines per rename() (#20185) deadlines per moveToTrash() allows for a more granular timeout approach for syscalls, instead of an aggregate timeout. This PR also enhances multipart state cleanup to be optimal by removing 100's of multipart network rename() calls into single network call.
2024-07-29 21:56:40 -04:00
storageRESTMethodDeleteBulk = "/dblk"
implement a safer completeMultipart implementation (#20227) - optimize writing part.N.meta by writing both part.N and its meta in sequence without network component. - remove part.N.meta, part.N which were partially success ful, in quorum loss situations during renamePart() - allow for strict read quorum check arbitrated via ETag for the given part number, this makes it double safer upon final commit. - return an appropriate error when read quorum is missing, instead of returning InvalidPart{}, which is non-retryable error. This kind of situation can happen when many nodes are going offline in rotation, an example of such a restart() behavior is statefulset updates in k8s. fixes #20091
2024-08-12 04:38:15 -04:00
storageRESTMethodReadParts = "/rps"
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
)
const (
change Read* calls over net/http to move to http.MethodGet (#20173) - ReadVersion - ReadFile - ReadXL Further changes include to - Compact internode resource RPC paths - Compact internode query params To optimize on parsing by gorilla/mux as the length of this string increases latency in gorilla/mux - reduce to a meaningful string.
2024-07-29 04:00:12 -04:00
storageRESTVolume = "vol"
storageRESTVolumes = "vols"
storageRESTDirPath = "dpath"
storageRESTFilePath = "fp"
storageRESTVersionID = "vid"
storageRESTHealing = "heal"
storageRESTTotalVersions = "tvers"
storageRESTSrcVolume = "svol"
storageRESTSrcPath = "spath"
storageRESTDstVolume = "dvol"
storageRESTDstPath = "dpath"
remove overzealous check during HEAD() (#19940) due to a historic bug in CopyObject() where an inlined object loses its metadata, the check causes an incorrect fallback verifying data-dir. CopyObject() bug was fixed in ffa91f97942 however the occurrence of this problem is historic, so the aforementioned check is stretching too much. Bonus: simplify fileInfoRaw() to read xl.json as well, also recreate buckets properly.
2024-06-17 10:29:18 -04:00
storageRESTOffset = "offset"
storageRESTLength = "length"
storageRESTCount = "count"
change Read* calls over net/http to move to http.MethodGet (#20173) - ReadVersion - ReadFile - ReadXL Further changes include to - Compact internode resource RPC paths - Compact internode query params To optimize on parsing by gorilla/mux as the length of this string increases latency in gorilla/mux - reduce to a meaningful string.
2024-07-29 04:00:12 -04:00
storageRESTBitrotAlgo = "balg"
storageRESTBitrotHash = "bhash"
storageRESTDiskID = "did"
storageRESTForceDelete = "fdel"
remove overzealous check during HEAD() (#19940) due to a historic bug in CopyObject() where an inlined object loses its metadata, the check causes an incorrect fallback verifying data-dir. CopyObject() bug was fixed in ffa91f97942 however the occurrence of this problem is historic, so the aforementioned check is stretching too much. Bonus: simplify fileInfoRaw() to read xl.json as well, also recreate buckets properly.
2024-06-17 10:29:18 -04:00
storageRESTGlob = "glob"
storageRESTMetrics = "metrics"
change Read* calls over net/http to move to http.MethodGet (#20173) - ReadVersion - ReadFile - ReadXL Further changes include to - Compact internode resource RPC paths - Compact internode query params To optimize on parsing by gorilla/mux as the length of this string increases latency in gorilla/mux - reduce to a meaningful string.
2024-07-29 04:00:12 -04:00
storageRESTDriveQuorum = "dquorum"
storageRESTOrigVolume = "ovol"
storageRESTInclFreeVersions = "incl-fv"
storageRESTRange = "rng"
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
)
perf: websocket grid connectivity for all internode communication (#18461) This PR adds a WebSocket grid feature that allows servers to communicate via a single two-way connection. There are two request types: * Single requests, which are `[]byte => ([]byte, error)`. This is for efficient small roundtrips with small payloads. * Streaming requests which are `[]byte, chan []byte => chan []byte (and error)`, which allows for different combinations of full two-way streams with an initial payload. Only a single stream is created between two machines - and there is, as such, no server/client relation since both sides can initiate and handle requests. Which server initiates the request is decided deterministically on the server names. Requests are made through a mux client and server, which handles message passing, congestion, cancelation, timeouts, etc. If a connection is lost, all requests are canceled, and the calling server will try to reconnect. Registered handlers can operate directly on byte slices or use a higher-level generics abstraction. There is no versioning of handlers/clients, and incompatible changes should be handled by adding new handlers. The request path can be changed to a new one for any protocol changes. First, all servers create a "Manager." The manager must know its address as well as all remote addresses. This will manage all connections. To get a connection to any remote, ask the manager to provide it given the remote address using. ``` func (m *Manager) Connection(host string) *Connection ``` All serverside handlers must also be registered on the manager. This will make sure that all incoming requests are served. The number of in-flight requests and responses must also be given for streaming requests. The "Connection" returned manages the mux-clients. Requests issued to the connection will be sent to the remote. * `func (c *Connection) Request(ctx context.Context, h HandlerID, req []byte) ([]byte, error)` performs a single request and returns the result. Any deadline provided on the request is forwarded to the server, and canceling the context will make the function return at once. * `func (c *Connection) NewStream(ctx context.Context, h HandlerID, payload []byte) (st *Stream, err error)` will initiate a remote call and send the initial payload. ```Go // A Stream is a two-way stream. // All responses *must* be read by the caller. // If the call is canceled through the context, //The appropriate error will be returned. type Stream struct { // Responses from the remote server. // Channel will be closed after an error or when the remote closes. // All responses *must* be read by the caller until either an error is returned or the channel is closed. // Canceling the context will cause the context cancellation error to be returned. Responses <-chan Response // Requests sent to the server. // If the handler is defined with 0 incoming capacity this will be nil. // Channel *must* be closed to signal the end of the stream. // If the request context is canceled, the stream will no longer process requests. Requests chan<- []byte } type Response struct { Msg []byte Err error } ``` There are generic versions of the server/client handlers that allow the use of type safe implementations for data types that support msgpack marshal/unmarshal.
2023-11-20 20:09:35 -05:00
type nsScannerOptions struct {
DiskID string `msg:"id"`
ScanMode int `msg:"m"`
Cache *dataUsageCache `msg:"c"`
}
type nsScannerResp struct {
Update *dataUsageEntry `msg:"u"`
Final *dataUsageCache `msg:"f"`
}