MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2025-11-10 22:10:12 -05:00
Code Issues Packages Projects Releases Wiki Activity

Move admin APIs to new path and add redesigned heal APIs (#5351)

Browse Source 
- Changes related to moving admin APIs
   - admin APIs now have an endpoint under /minio/admin
   - admin APIs are now versioned - a new API to server the version is
     added at "GET /minio/admin/version" and all API operations have the
     path prefix /minio/admin/v1/<operation>
   - new service stop API added
   - credentials change API is moved to /minio/admin/v1/config/credential
   - credentials change API and configuration get/set API now require TLS
     so that credentials are protected
   - all API requests now receive JSON
   - heal APIs are disabled as they will be changed substantially

- Heal API changes
   Heal API is now provided at a single endpoint with the ability for a
   client to start a heal sequence on all the data in the server, a
   single bucket, or under a prefix within a bucket.

   When a heal sequence is started, the server returns a unique token
   that needs to be used for subsequent 'status' requests to fetch heal
   results.

   On each status request from the client, the server returns heal result
   records that it has accumulated since the previous status request. The
   server accumulates upto 1000 records and pauses healing further
   objects until the client requests for status. If the client does not
   request any further records for a long time, the server aborts the
   heal sequence automatically.

   A heal result record is returned for each entity healed on the server,
   such as system metadata, object metadata, buckets and objects, and has
   information about the before and after states on each disk.

   A client may request to force restart a heal sequence - this causes
   the running heal sequence to be aborted at the next safe spot and
   starts a new heal sequence.
This commit is contained in:
Aditya Manthramurthy
2018-01-22 14:54:55 -08:00
committed by Harshavardhana
parent f3f09ed14e
commit a337ea4d11
43 changed files with 2414 additions and 2319 deletions
Download Patch File Download Diff File Expand all files Collapse all files

226
pkg/madmin/API.md
View File

@@ -36,13 +36,11 @@ func main() {
```
| Service operations|LockInfo operations|Healing operations|Config operations| Misc |
|:---|:---|:---|:---|:---|
|[`ServiceStatus`](#ServiceStatus)| [`ListLocks`](#ListLocks)| [`ListObjectsHeal`](#ListObjectsHeal)|[`GetConfig`](#GetConfig)| [`SetCredentials`](#SetCredentials)|
|[`ServiceRestart`](#ServiceRestart)| [`ClearLocks`](#ClearLocks)| [`ListBucketsHeal`](#ListBucketsHeal)|[`SetConfig`](#SetConfig)||
| | |[`HealBucket`](#HealBucket) |||
| | |[`HealObject`](#HealObject)|||
| | |[`HealFormat`](#HealFormat)|||
| Service operations | LockInfo operations | Healing operations | Config operations | Misc |
|:------------------------------------|:----------------------------|:--------------------------------------|:--------------------------|:------------------------------------|
| [`ServiceStatus`](#ServiceStatus) | [`ListLocks`](#ListLocks) | [`Heal`](#Heal) | [`GetConfig`](#GetConfig) | [`SetCredentials`](#SetCredentials) |
| [`ServiceSendAction`](#ServiceSendAction) | [`ClearLocks`](#ClearLocks) | | [`SetConfig`](#SetConfig) | |
## 1. Constructor
<a name="Minio"></a>
@@ -60,8 +58,25 @@ __Parameters__
|`secretAccessKey` | _string_ |Secret key for the object storage endpoint. |
|`ssl` | _bool_ | Set this value to 'true' to enable secure (HTTPS) access. |
## 2. Admin API Version
## 2. Service operations
<a name="VersionInfo"></a>
### VersionInfo() (AdminAPIVersionInfo, error)
Fetch server's supported Administrative API version.
__Example__
``` go
info, err := madmClnt.VersionInfo()
if err != nil {
log.Fatalln(err)
}
log.Printf("%s\n", info.Version)
```
## 3. Service operations
<a name="ServiceStatus"></a>
### ServiceStatus() (ServiceStatusMetadata, error)
@@ -102,17 +117,19 @@ Fetch service status, replies disk space used, backend type and total disks offl
```
<a name="ServiceRestart"></a>
### ServiceRestart() (error)
If successful restarts the running minio service, for distributed setup restarts all remote minio servers.
<a name="ServiceSendAction"></a>
### ServiceSendAction(act ServiceActionValue) (error)
Sends a service action command to service - possible actions are restarting and stopping the server.
__Example__
```go
st, err := madmClnt.ServiceRestart()
// to restart
st, err := madmClnt.ServiceSendAction(ServiceActionValueRestart)
// or to stop
// st, err := madmClnt.ServiceSendAction(ServiceActionValueStop)
if err != nil {
log.Fatalln(err)
}
@@ -120,7 +137,7 @@ If successful restarts the running minio service, for distributed setup restarts
```
## 3. Info operations
## 4. Info operations
<a name="ServerInfo"></a>
### ServerInfo() ([]ServerInfo, error)
@@ -143,7 +160,7 @@ Fetch all information for all cluster nodes, such as uptime, region, network sta
```
## 4. Lock operations
## 5. Lock operations
<a name="ListLocks"></a>
### ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error)
@@ -175,146 +192,95 @@ __Example__
```
## 5. Heal operations
## 6. Heal operations
<a name="ListObjectsHeal"></a>
### ListObjectsHeal(bucket, prefix string, recursive bool, doneCh <-chan struct{}) (<-chan ObjectInfo, error)
If successful returns information on the list of objects that need healing in ``bucket`` matching ``prefix``.
<a name="Heal"></a>
### Heal(bucket, prefix string, healOpts HealOpts, clientToken string, forceStart bool) (start HealStartSuccess, status HealTaskStatus, err error)
Start a heal sequence that scans data under given (possible empty)
`bucket` and `prefix`. The `recursive` bool turns on recursive
traversal under the given path. `dryRun` does not mutate on-disk data,
but performs data validation. `incomplete` enables healing of
multipart uploads that are in progress. `removeBadFiles` removes
unrecoverable files. `statisticsOnly` turns off detailed
heal-operations reporting in the status call.
Two heal sequences on overlapping paths may not be initiated.
The progress of a heal should be followed using the `HealStatus`
API. The server accumulates results of the heal traversal and waits
for the client to receive and acknowledge them using the status
API. When the statistics-only option is set, the server only maintains
aggregates statistics - in this case, no acknowledgement of results is
required.
__Example__
``` go
// Create a done channel to control 'ListObjectsHeal' go routine.
doneCh := make(chan struct{})
// Indicate to our routine to exit cleanly upon return.
defer close(doneCh)
// Set true if recursive listing is needed.
isRecursive := true
// List objects that need healing for a given bucket and
// prefix.
healObjectCh, err := madmClnt.ListObjectsHeal("mybucket", "myprefix", isRecursive, doneCh)
if err != nil {
fmt.Println(err)
return
}
for object := range healObjectsCh {
if object.Err != nil {
log.Fatalln(err)
return
}
if object.HealObjectInfo != nil {
switch healInfo := *object.HealObjectInfo; healInfo.Status {
case madmin.CanHeal:
fmt.Println(object.Key, " can be healed.")
case madmin.QuorumUnavailable:
fmt.Println(object.Key, " can't be healed until quorum is available.")
case madmin.Corrupted:
fmt.Println(object.Key, " can't be healed, not enough information.")
}
}
fmt.Println("object: ", object)
}
```
<a name="ListBucketsHeal"></a>
### ListBucketsHeal() error
If successful returns information on the list of buckets that need healing.
__Example__
``` go
// List buckets that need healing
healBucketsList, err := madmClnt.ListBucketsHeal()
if err != nil {
fmt.Println(err)
return
}
for bucket := range healBucketsList {
if bucket.HealBucketInfo != nil {
switch healInfo := *object.HealBucketInfo; healInfo.Status {
case madmin.CanHeal:
fmt.Println(bucket.Key, " can be healed.")
case madmin.QuorumUnavailable:
fmt.Println(bucket.Key, " can't be healed until quorum is available.")
case madmin.Corrupted:
fmt.Println(bucket.Key, " can't be healed, not enough information.")
}
}
fmt.Println("bucket: ", bucket)
}
```
<a name="HealBucket"></a>
### HealBucket(bucket string, isDryRun bool) error
If bucket is successfully healed returns nil, otherwise returns error indicating the reason for failure. If isDryRun is true, then the bucket is not healed, but heal bucket request is validated by the server. e.g, if the bucket exists, if bucket name is valid etc.
__Example__
``` go
isDryRun := false
err := madmClnt.HealBucket("mybucket", isDryRun)
healPath, err := madmClnt.HealStart("", "", true, false, true, false, false)
if err != nil {
log.Fatalln(err)
}
log.Println("successfully healed mybucket")
log.Printf("Heal sequence started at %s", healPath)
```
<a name="HealObject"></a>
### HealObject(bucket, object string, isDryRun bool) (HealResult, error)
If object is successfully healed returns nil, otherwise returns error indicating the reason for failure. If isDryRun is true, then the object is not healed, but heal object request is validated by the server. e.g, if the object exists, if object name is valid etc.
#### HealTaskStatus structure
| Param | Type | Description |
|---|---|---|
|`h.State` | _HealState_ | Represents the result of heal operation. It could be one of `HealNone`, `HealPartial` or `HealOK`. |
| Param | Type | Description |
|----|--------|--------|
| s.Summary | _string_ | Short status of heal sequence |
| s.FailureDetail | _string_ | Error message in case of heal sequence failure |
| s.HealSettings | _HealOpts_ | Contains the booleans set in the `HealStart` call |
| s.Items | _[]HealResultItem_ | Heal records for actions performed by server |
| s.Statistics | _HealStatistics_ | Aggregate of heal records from beginning |
#### HealResultItem structure
| Value | Description |
|---|---|
|`HealNone` | Object wasn't healed on any of the disks |
|`HealPartial` | Object was healed on some of the disks needing heal |
| `HealOK` | Object was healed on all the disks needing heal |
| Param | Type | Description |
|------|-------|---------|
| ResultIndex | _int64_ | Index of the heal-result record |
| Type | _HealItemType_ | Represents kind of heal operation in the heal record |
| Bucket | _string_ | Bucket name |
| Object | _string_ | Object name |
| Detail | _string_ | Details about heal operation |
| DiskInfo.AvailableOn | _[]int_ | List of disks on which the healed entity is present and healthy |
| DiskInfo.HealedOn | _[]int_ | List of disks on which the healed entity was restored |
#### HealStatistics structure
Most parameters represent the aggregation of heal operations since the
start of the heal sequence.
| Param | Type | Description |
|-------|-----|----------|
| NumDisks | _int_ | Number of disks configured in the backend |
| NumBucketsScanned | _int64_ | Number of buckets scanned |
| BucketsMissingByDisk | _map[int]int64_ | Map of disk to number of buckets missing |
| BucketsAvailableByDisk | _map[int]int64_ | Map of disk to number of buckets available |
| BucketsHealedByDisk | _map[int]int64_ | Map of disk to number of buckets healed on |
| NumObjectsScanned | _int64_ | Number of objects scanned |
| NumUploadsScanned | _int64_ | Number of uploads scanned |
| ObjectsByAvailablePC | _map[int64]_ | Map of available part counts (after heal) to number of objects |
| ObjectsByHealedPC | _map[int64]_ | Map of healed part counts to number of objects |
| ObjectsMissingByDisk | _map[int64]_ | Map of disk number to number of objects with parts missing on that disk |
| ObjectsAvailableByDisk | _map[int64]_ | Map of disk number to number of objects available on that disk |
| ObjectsHealedByDisk | _map[int64]_ | Map of disk number to number of objects healed on that disk |
__Example__
``` go
isDryRun = false
healResult, err := madmClnt.HealObject("mybucket", "myobject", isDryRun)
res, err := madmClnt.HealStatus("", "")
if err != nil {
log.Fatalln(err)
}
log.Println("Heal-object result: ", healResult)
log.Printf("Heal sequence status data %#v", res)
```
<a name="HealFormat"></a>
### HealFormat(isDryRun bool) error
Heal storage format on available disks. This is used when disks were replaced or were found with missing format. This is supported only for erasure-coded backend.
__Example__
``` go
isDryRun := true
err := madmClnt.HealFormat(isDryRun)
if err != nil {
log.Fatalln(err)
}
isDryRun = false
err = madmClnt.HealFormat(isDryRun)
if err != nil {
log.Fatalln(err)
}
log.Println("successfully healed storage format on available disks.")
```
## 6. Config operations
## 7. Config operations
<a name="GetConfig"></a>
### GetConfig() ([]byte, error)
@@ -373,7 +339,7 @@ __Example__
log.Println("SetConfig: ", string(buf.Bytes()))
```
## 7. Misc operations
## 8. Misc operations
<a name="SetCredentials"></a>