MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
minio/cmd/storage-rest-common.go

82 lines
3.2 KiB
Go
Raw Normal View History

update license change for MinIO Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-18 15:41:13 -04:00
// Copyright (c) 2015-2021 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
package cmd
admin/heal: Fix deep healing to heal objects under more conditions (#8321) - Heal if the part.1 is truncated from its original size - Heal if the part.1 fails while being verified in between - Heal if the part.1 fails while being at a certain offset Other cleanups include make sure to flush the HTTP responses properly from storage-rest-server, avoid using 'defer' to improve call latency. 'defer' incurs latency avoid them in our hot-paths such as storage-rest handlers. Fixes #8319
2019-10-01 16:12:15 -04:00
const (
fix: remove parentIsObject() check (#12851) we will allow situations such as ``` a/b/1.txt a/b ``` and ``` a/b a/b/1.txt ``` we are going to document that this usecase is not supported and we will never support it, if any application does this users have to delete the top level parent to make sure namespace is accessible at lower level. rest of the situations where the prefixes get created across sets are supported as is.
2021-08-03 16:26:57 -04:00
storageRESTVersion = "v38" // Remove CheckFile API
Add data usage collect with its new admin API (#8553) Admin data usage info API returns the following (Only FS & XL, for now) - Number of buckets - Number of objects - The total size of objects - Objects histogram - Bucket sizes
2019-12-12 09:02:37 -05:00
storageRESTVersionPrefix = SlashSeparator + storageRESTVersion
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-04 12:30:59 -05:00
storageRESTPrefix = minioReservedBucketPath + "/storage"
admin/heal: Fix deep healing to heal objects under more conditions (#8321) - Heal if the part.1 is truncated from its original size - Heal if the part.1 fails while being verified in between - Heal if the part.1 fails while being at a certain offset Other cleanups include make sure to flush the HTTP responses properly from storage-rest-server, avoid using 'defer' to improve call latency. 'defer' incurs latency avoid them in our hot-paths such as storage-rest handlers. Fixes #8319
2019-10-01 16:12:15 -04:00
)
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
const (
rename all references from crawl -> scanner (#11621)
2021-02-26 18:11:42 -05:00
storageRESTMethodHealth = "/health"
storageRESTMethodDiskInfo = "/diskinfo"
storageRESTMethodNSScanner = "/nsscanner"
storageRESTMethodMakeVol = "/makevol"
storageRESTMethodMakeVolBulk = "/makevolbulk"
storageRESTMethodStatVol = "/statvol"
storageRESTMethodDeleteVol = "/deletevol"
storageRESTMethodListVols = "/listvols"
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-04 12:30:59 -05:00
storageRESTMethodAppendFile = "/appendfile"
storageRESTMethodCreateFile = "/createfile"
storageRESTMethodWriteAll = "/writeall"
Support bucket versioning (#9377) - Implement a new xl.json 2.0.0 format to support, this moves the entire marshaling logic to POSIX layer, top layer always consumes a common FileInfo construct which simplifies the metadata reads. - Implement list object versions - Migrate to siphash from crchash for new deployments for object placements. Fixes #2111
2020-06-12 23:04:01 -04:00
storageRESTMethodWriteMetadata = "/writemetadata"
api: Introduce metadata update APIs to update only metadata (#11962) Current implementation heavily relies on readAllFileInfo but with the advent of xl.meta inlined with data, we cannot easily avoid reading data when we are only interested is updating metadata, this leads to invariably write amplification during metadata updates, repeatedly reading data when we are only interested in updating metadata. This PR ensures that we implement a metadata only update API at storage layer, that handles updates to metadata alone for any given version - given the version is valid and present. This helps reduce the chattiness for following calls.. - PutObjectTags - DeleteObjectTags - PutObjectLegalHold - PutObjectRetention - ReplicateObject (updates metadata on replication status)
2021-04-04 16:32:31 -04:00
storageRESTMethodUpdateMetadata = "/updatemetadata"
Support bucket versioning (#9377) - Implement a new xl.json 2.0.0 format to support, this moves the entire marshaling logic to POSIX layer, top layer always consumes a common FileInfo construct which simplifies the metadata reads. - Implement list object versions - Migrate to siphash from crchash for new deployments for object placements. Fixes #2111
2020-06-12 23:04:01 -04:00
storageRESTMethodDeleteVersion = "/deleteversion"
storageRESTMethodReadVersion = "/readversion"
storageRESTMethodRenameData = "/renamedata"
storageRESTMethodCheckParts = "/checkparts"
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-04 12:30:59 -05:00
storageRESTMethodReadAll = "/readall"
storageRESTMethodReadFile = "/readfile"
storageRESTMethodReadFileStream = "/readfilestream"
storageRESTMethodListDir = "/listdir"
storageRESTMethodDeleteFile = "/deletefile"
Support bucket versioning (#9377) - Implement a new xl.json 2.0.0 format to support, this moves the entire marshaling logic to POSIX layer, top layer always consumes a common FileInfo construct which simplifies the metadata reads. - Implement list object versions - Migrate to siphash from crchash for new deployments for object placements. Fixes #2111
2020-06-12 23:04:01 -04:00
storageRESTMethodDeleteVersions = "/deleteverions"
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-04 12:30:59 -05:00
storageRESTMethodRenameFile = "/renamefile"
storageRESTMethodVerifyFile = "/verifyfile"
ListObjects Metadata Caching (#10648) Design: https://gist.github.com/klauspost/025c09b48ed4a1293c917cecfabdf21c Gist of improvements: * Cross-server caching and listing will use the same data across servers and requests. * Lists can be arbitrarily resumed at a constant speed. * Metadata for all files scanned is stored for streaming retrieval. * The existing bloom filters controlled by the crawler is used for validating caches. * Concurrent requests for the same data (or parts of it) will not spawn additional walkers. * Listing a subdirectory of an existing recursive cache will use the cache. * All listing operations are fully streamable so the number of objects in a bucket no longer dictates the amount of memory. * Listings can be handled by any server within the cluster. * Caches are cleaned up when out of date or superseded by a more recent one.
2020-10-28 12:18:35 -04:00
storageRESTMethodWalkDir = "/walkdir"
Add admin file inspector (#12635) Download files from *any* bucket/path as an encrypted zip file. The key is included in the response but can be separated so zip and the key doesn't have to be sent on the same channel. Requires https://github.com/minio/pkg/pull/6
2021-07-09 14:29:16 -04:00
storageRESTMethodStatInfoFile = "/statfile"
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
)
const (
fix: heal optimizations in crawler to avoid multiple healing attempts (#11173) Fixes two problems - Double healing when bitrot is enabled, instead heal attempt once in applyActions() before lifecycle is applied. - If applyActions() is successful and getSize() returns proper value, then object is accounted for and should be removed from the oldCache namespace map to avoid double heal attempts.
2020-12-28 13:31:00 -05:00
storageRESTVolume = "volume"
storageRESTVolumes = "volumes"
storageRESTDirPath = "dir-path"
storageRESTFilePath = "file-path"
xl: Make it clear when to create delete marker for a non existant object (#11423)
2021-02-03 13:33:43 -05:00
storageRESTForceDelMarker = "force-delete-marker"
fix: heal optimizations in crawler to avoid multiple healing attempts (#11173) Fixes two problems - Double healing when bitrot is enabled, instead heal attempt once in applyActions() before lifecycle is applied. - If applyActions() is successful and getSize() returns proper value, then object is accounted for and should be removed from the oldCache namespace map to avoid double heal attempts.
2020-12-28 13:31:00 -05:00
storageRESTVersionID = "version-id"
fix: readData in bulk call using messagepack byte wrappers (#11228) This PR refactors the way we use buffers for O_DIRECT and to re-use those buffers for messagepack reader writer. After some extensive benchmarking found that not all objects have this benefit, and only objects smaller than 64KiB see this benefit overall. Benefits are seen from almost all objects from 1KiB - 32KiB Beyond this no objects see benefit with bulk call approach as the latency of bytes sent over the wire v/s streaming content directly from disk negate each other with no remarkable benefits. All other optimizations include reuse of msgp.Reader, msgp.Writer using sync.Pool's for all internode calls.
2021-01-07 22:27:31 -05:00
storageRESTReadData = "read-data"
fix: heal optimizations in crawler to avoid multiple healing attempts (#11173) Fixes two problems - Double healing when bitrot is enabled, instead heal attempt once in applyActions() before lifecycle is applied. - If applyActions() is successful and getSize() returns proper value, then object is accounted for and should be removed from the oldCache namespace map to avoid double heal attempts.
2020-12-28 13:31:00 -05:00
storageRESTTotalVersions = "total-versions"
storageRESTSrcVolume = "source-volume"
storageRESTSrcPath = "source-path"
storageRESTDstVolume = "destination-volume"
storageRESTDstPath = "destination-path"
storageRESTOffset = "offset"
storageRESTLength = "length"
storageRESTCount = "count"
storageRESTPrefixFilter = "prefix"
fix: faster healing when disk is replaced. (#11520)
2021-02-18 14:06:54 -05:00
storageRESTForwardFilter = "forward"
fix: heal optimizations in crawler to avoid multiple healing attempts (#11173) Fixes two problems - Double healing when bitrot is enabled, instead heal attempt once in applyActions() before lifecycle is applied. - If applyActions() is successful and getSize() returns proper value, then object is accounted for and should be removed from the oldCache namespace map to avoid double heal attempts.
2020-12-28 13:31:00 -05:00
storageRESTRecursive = "recursive"
storageRESTReportNotFound = "report-notfound"
storageRESTBitrotAlgo = "bitrot-algo"
storageRESTBitrotHash = "bitrot-hash"
storageRESTDiskID = "disk-id"
storageRESTForceDelete = "force-delete"
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-04 20:44:06 -04:00
)