MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2025-11-09 05:34:56 -05:00
Code Issues Packages Projects Releases Wiki Activity

Move admin APIs to new path and add redesigned heal APIs (#5351)

Browse Source 
- Changes related to moving admin APIs
   - admin APIs now have an endpoint under /minio/admin
   - admin APIs are now versioned - a new API to server the version is
     added at "GET /minio/admin/version" and all API operations have the
     path prefix /minio/admin/v1/<operation>
   - new service stop API added
   - credentials change API is moved to /minio/admin/v1/config/credential
   - credentials change API and configuration get/set API now require TLS
     so that credentials are protected
   - all API requests now receive JSON
   - heal APIs are disabled as they will be changed substantially

- Heal API changes
   Heal API is now provided at a single endpoint with the ability for a
   client to start a heal sequence on all the data in the server, a
   single bucket, or under a prefix within a bucket.

   When a heal sequence is started, the server returns a unique token
   that needs to be used for subsequent 'status' requests to fetch heal
   results.

   On each status request from the client, the server returns heal result
   records that it has accumulated since the previous status request. The
   server accumulates upto 1000 records and pauses healing further
   objects until the client requests for status. If the client does not
   request any further records for a long time, the server aborts the
   heal sequence automatically.

   A heal result record is returned for each entity healed on the server,
   such as system metadata, object metadata, buckets and objects, and has
   information about the before and after states on each disk.

   A client may request to force restart a heal sequence - this causes
   the running heal sequence to be aborted at the next safe spot and
   starts a new heal sequence.
This commit is contained in:
Aditya Manthramurthy
2018-01-22 14:54:55 -08:00
committed by Harshavardhana
parent f3f09ed14e
commit a337ea4d11
43 changed files with 2414 additions and 2319 deletions
Download Patch File Download Diff File Expand all files Collapse all files

226
pkg/madmin/API.md
View File

@@ -36,13 +36,11 @@ func main() {
```
| Service operations|LockInfo operations|Healing operations|Config operations| Misc |
|:---|:---|:---|:---|:---|
|[`ServiceStatus`](#ServiceStatus)| [`ListLocks`](#ListLocks)| [`ListObjectsHeal`](#ListObjectsHeal)|[`GetConfig`](#GetConfig)| [`SetCredentials`](#SetCredentials)|
|[`ServiceRestart`](#ServiceRestart)| [`ClearLocks`](#ClearLocks)| [`ListBucketsHeal`](#ListBucketsHeal)|[`SetConfig`](#SetConfig)||
| | |[`HealBucket`](#HealBucket) |||
| | |[`HealObject`](#HealObject)|||
| | |[`HealFormat`](#HealFormat)|||
| Service operations | LockInfo operations | Healing operations | Config operations | Misc |
|:------------------------------------|:----------------------------|:--------------------------------------|:--------------------------|:------------------------------------|
| [`ServiceStatus`](#ServiceStatus) | [`ListLocks`](#ListLocks) | [`Heal`](#Heal) | [`GetConfig`](#GetConfig) | [`SetCredentials`](#SetCredentials) |
| [`ServiceSendAction`](#ServiceSendAction) | [`ClearLocks`](#ClearLocks) | | [`SetConfig`](#SetConfig) | |
## 1. Constructor
<a name="Minio"></a>
@@ -60,8 +58,25 @@ __Parameters__
|`secretAccessKey` | _string_ |Secret key for the object storage endpoint. |
|`ssl` | _bool_ | Set this value to 'true' to enable secure (HTTPS) access. |
## 2. Admin API Version
## 2. Service operations
<a name="VersionInfo"></a>
### VersionInfo() (AdminAPIVersionInfo, error)
Fetch server's supported Administrative API version.
__Example__
``` go
info, err := madmClnt.VersionInfo()
if err != nil {
log.Fatalln(err)
}
log.Printf("%s\n", info.Version)
```
## 3. Service operations
<a name="ServiceStatus"></a>
### ServiceStatus() (ServiceStatusMetadata, error)
@@ -102,17 +117,19 @@ Fetch service status, replies disk space used, backend type and total disks offl
```
<a name="ServiceRestart"></a>
### ServiceRestart() (error)
If successful restarts the running minio service, for distributed setup restarts all remote minio servers.
<a name="ServiceSendAction"></a>
### ServiceSendAction(act ServiceActionValue) (error)
Sends a service action command to service - possible actions are restarting and stopping the server.
__Example__
```go
st, err := madmClnt.ServiceRestart()
// to restart
st, err := madmClnt.ServiceSendAction(ServiceActionValueRestart)
// or to stop
// st, err := madmClnt.ServiceSendAction(ServiceActionValueStop)
if err != nil {
log.Fatalln(err)
}
@@ -120,7 +137,7 @@ If successful restarts the running minio service, for distributed setup restarts
```
## 3. Info operations
## 4. Info operations
<a name="ServerInfo"></a>
### ServerInfo() ([]ServerInfo, error)
@@ -143,7 +160,7 @@ Fetch all information for all cluster nodes, such as uptime, region, network sta
```
## 4. Lock operations
## 5. Lock operations
<a name="ListLocks"></a>
### ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error)
@@ -175,146 +192,95 @@ __Example__
```
## 5. Heal operations
## 6. Heal operations
<a name="ListObjectsHeal"></a>
### ListObjectsHeal(bucket, prefix string, recursive bool, doneCh <-chan struct{}) (<-chan ObjectInfo, error)
If successful returns information on the list of objects that need healing in ``bucket`` matching ``prefix``.
<a name="Heal"></a>
### Heal(bucket, prefix string, healOpts HealOpts, clientToken string, forceStart bool) (start HealStartSuccess, status HealTaskStatus, err error)
Start a heal sequence that scans data under given (possible empty)
`bucket` and `prefix`. The `recursive` bool turns on recursive
traversal under the given path. `dryRun` does not mutate on-disk data,
but performs data validation. `incomplete` enables healing of
multipart uploads that are in progress. `removeBadFiles` removes
unrecoverable files. `statisticsOnly` turns off detailed
heal-operations reporting in the status call.
Two heal sequences on overlapping paths may not be initiated.
The progress of a heal should be followed using the `HealStatus`
API. The server accumulates results of the heal traversal and waits
for the client to receive and acknowledge them using the status
API. When the statistics-only option is set, the server only maintains
aggregates statistics - in this case, no acknowledgement of results is
required.
__Example__
``` go
// Create a done channel to control 'ListObjectsHeal' go routine.
doneCh := make(chan struct{})
// Indicate to our routine to exit cleanly upon return.
defer close(doneCh)
// Set true if recursive listing is needed.
isRecursive := true
// List objects that need healing for a given bucket and
// prefix.
healObjectCh, err := madmClnt.ListObjectsHeal("mybucket", "myprefix", isRecursive, doneCh)
if err != nil {
fmt.Println(err)
return
}
for object := range healObjectsCh {
if object.Err != nil {
log.Fatalln(err)
return
}
if object.HealObjectInfo != nil {
switch healInfo := *object.HealObjectInfo; healInfo.Status {
case madmin.CanHeal:
fmt.Println(object.Key, " can be healed.")
case madmin.QuorumUnavailable:
fmt.Println(object.Key, " can't be healed until quorum is available.")
case madmin.Corrupted:
fmt.Println(object.Key, " can't be healed, not enough information.")
}
}
fmt.Println("object: ", object)
}
```
<a name="ListBucketsHeal"></a>
### ListBucketsHeal() error
If successful returns information on the list of buckets that need healing.
__Example__
``` go
// List buckets that need healing
healBucketsList, err := madmClnt.ListBucketsHeal()
if err != nil {
fmt.Println(err)
return
}
for bucket := range healBucketsList {
if bucket.HealBucketInfo != nil {
switch healInfo := *object.HealBucketInfo; healInfo.Status {
case madmin.CanHeal:
fmt.Println(bucket.Key, " can be healed.")
case madmin.QuorumUnavailable:
fmt.Println(bucket.Key, " can't be healed until quorum is available.")
case madmin.Corrupted:
fmt.Println(bucket.Key, " can't be healed, not enough information.")
}
}
fmt.Println("bucket: ", bucket)
}
```
<a name="HealBucket"></a>
### HealBucket(bucket string, isDryRun bool) error
If bucket is successfully healed returns nil, otherwise returns error indicating the reason for failure. If isDryRun is true, then the bucket is not healed, but heal bucket request is validated by the server. e.g, if the bucket exists, if bucket name is valid etc.
__Example__
``` go
isDryRun := false
err := madmClnt.HealBucket("mybucket", isDryRun)
healPath, err := madmClnt.HealStart("", "", true, false, true, false, false)
if err != nil {
log.Fatalln(err)
}
log.Println("successfully healed mybucket")
log.Printf("Heal sequence started at %s", healPath)
```
<a name="HealObject"></a>
### HealObject(bucket, object string, isDryRun bool) (HealResult, error)
If object is successfully healed returns nil, otherwise returns error indicating the reason for failure. If isDryRun is true, then the object is not healed, but heal object request is validated by the server. e.g, if the object exists, if object name is valid etc.
#### HealTaskStatus structure
| Param | Type | Description |
|---|---|---|
|`h.State` | _HealState_ | Represents the result of heal operation. It could be one of `HealNone`, `HealPartial` or `HealOK`. |
| Param | Type | Description |
|----|--------|--------|
| s.Summary | _string_ | Short status of heal sequence |
| s.FailureDetail | _string_ | Error message in case of heal sequence failure |
| s.HealSettings | _HealOpts_ | Contains the booleans set in the `HealStart` call |
| s.Items | _[]HealResultItem_ | Heal records for actions performed by server |
| s.Statistics | _HealStatistics_ | Aggregate of heal records from beginning |
#### HealResultItem structure
| Value | Description |
|---|---|
|`HealNone` | Object wasn't healed on any of the disks |
|`HealPartial` | Object was healed on some of the disks needing heal |
| `HealOK` | Object was healed on all the disks needing heal |
| Param | Type | Description |
|------|-------|---------|
| ResultIndex | _int64_ | Index of the heal-result record |
| Type | _HealItemType_ | Represents kind of heal operation in the heal record |
| Bucket | _string_ | Bucket name |
| Object | _string_ | Object name |
| Detail | _string_ | Details about heal operation |
| DiskInfo.AvailableOn | _[]int_ | List of disks on which the healed entity is present and healthy |
| DiskInfo.HealedOn | _[]int_ | List of disks on which the healed entity was restored |
#### HealStatistics structure
Most parameters represent the aggregation of heal operations since the
start of the heal sequence.
| Param | Type | Description |
|-------|-----|----------|
| NumDisks | _int_ | Number of disks configured in the backend |
| NumBucketsScanned | _int64_ | Number of buckets scanned |
| BucketsMissingByDisk | _map[int]int64_ | Map of disk to number of buckets missing |
| BucketsAvailableByDisk | _map[int]int64_ | Map of disk to number of buckets available |
| BucketsHealedByDisk | _map[int]int64_ | Map of disk to number of buckets healed on |
| NumObjectsScanned | _int64_ | Number of objects scanned |
| NumUploadsScanned | _int64_ | Number of uploads scanned |
| ObjectsByAvailablePC | _map[int64]_ | Map of available part counts (after heal) to number of objects |
| ObjectsByHealedPC | _map[int64]_ | Map of healed part counts to number of objects |
| ObjectsMissingByDisk | _map[int64]_ | Map of disk number to number of objects with parts missing on that disk |
| ObjectsAvailableByDisk | _map[int64]_ | Map of disk number to number of objects available on that disk |
| ObjectsHealedByDisk | _map[int64]_ | Map of disk number to number of objects healed on that disk |
__Example__
``` go
isDryRun = false
healResult, err := madmClnt.HealObject("mybucket", "myobject", isDryRun)
res, err := madmClnt.HealStatus("", "")
if err != nil {
log.Fatalln(err)
}
log.Println("Heal-object result: ", healResult)
log.Printf("Heal sequence status data %#v", res)
```
<a name="HealFormat"></a>
### HealFormat(isDryRun bool) error
Heal storage format on available disks. This is used when disks were replaced or were found with missing format. This is supported only for erasure-coded backend.
__Example__
``` go
isDryRun := true
err := madmClnt.HealFormat(isDryRun)
if err != nil {
log.Fatalln(err)
}
isDryRun = false
err = madmClnt.HealFormat(isDryRun)
if err != nil {
log.Fatalln(err)
}
log.Println("successfully healed storage format on available disks.")
```
## 6. Config operations
## 7. Config operations
<a name="GetConfig"></a>
### GetConfig() ([]byte, error)
@@ -373,7 +339,7 @@ __Example__
log.Println("SetConfig: ", string(buf.Bytes()))
```
## 7. Misc operations
## 8. Misc operations
<a name="SetCredentials"></a>

6
pkg/madmin/api-error-response.go
View File

@@ -54,7 +54,7 @@ func (e ErrorResponse) Error() string {
}
const (
reportIssue = "Please report this issue at https://github.com/minio/minio-go/issues."
reportIssue = "Please report this issue at https://github.com/minio/minio/issues."
)
// httpRespToErrorResponse returns a new encoded ErrorResponse
@@ -65,8 +65,8 @@ func httpRespToErrorResponse(resp *http.Response) error {
return ErrInvalidArgument(msg)
}
var errResp ErrorResponse
// Decode the xml error
err := xmlDecoder(resp.Body, &errResp)
// Decode the json error
err := jsonDecoder(resp.Body, &errResp)
if err != nil {
return ErrorResponse{
Code: resp.Status,

15
pkg/madmin/api.go
View File

@@ -71,6 +71,8 @@ type AdminClient struct {
const (
libraryName = "madmin-go"
libraryVersion = "0.0.1"
libraryAdminURLPrefix = "/minio/admin"
)
// User Agent should always following the below style.
@@ -176,6 +178,9 @@ type requestData struct {
customHeaders http.Header
queryValues url.Values
// Url path relative to admin API base endpoint
relPath string
contentBody io.Reader
contentLength int64
contentSHA256Bytes []byte
@@ -388,7 +393,7 @@ func (c AdminClient) newRequest(method string, reqData requestData) (req *http.R
location := "us-east-1"
// Construct a new target URL.
targetURL, err := c.makeTargetURL(reqData.queryValues)
targetURL, err := c.makeTargetURL(reqData)
if err != nil {
return nil, err
}
@@ -440,16 +445,16 @@ func (c AdminClient) newRequest(method string, reqData requestData) (req *http.R
}
// makeTargetURL make a new target url.
func (c AdminClient) makeTargetURL(queryValues url.Values) (*url.URL, error) {
func (c AdminClient) makeTargetURL(r requestData) (*url.URL, error) {
host := c.endpointURL.Host
scheme := c.endpointURL.Scheme
urlStr := scheme + "://" + host + "/"
urlStr := scheme + "://" + host + libraryAdminURLPrefix + r.relPath
// If there are any query values, add them to the end.
if len(queryValues) > 0 {
urlStr = urlStr + "?" + s3utils.QueryEncode(queryValues)
if len(r.queryValues) > 0 {
urlStr = urlStr + "?" + s3utils.QueryEncode(r.queryValues)
}
u, err := url.Parse(urlStr)
if err != nil {

58
pkg/madmin/config-commands.go
View File

@@ -20,14 +20,10 @@ package madmin
import (
"bytes"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
)
const (
configQueryParam = "config"
)
// NodeSummary - represents the result of an operation part of
@@ -47,20 +43,14 @@ type SetConfigResult struct {
// GetConfig - returns the config.json of a minio setup.
func (adm *AdminClient) GetConfig() ([]byte, error) {
queryVal := make(url.Values)
queryVal.Set(configQueryParam, "")
hdrs := make(http.Header)
hdrs.Set(minioAdminOpHeader, "get")
reqData := requestData{
queryValues: queryVal,
customHeaders: hdrs,
// No TLS?
if !adm.secure {
return nil, fmt.Errorf("credentials/configuration cannot be retrieved over an insecure connection")
}
// Execute GET on /?config to get config of a setup.
resp, err := adm.executeMethod("GET", reqData)
// Execute GET on /minio/admin/v1/config to get config of a setup.
resp, err := adm.executeMethod("GET",
requestData{relPath: "/v1/config"})
defer closeResponse(resp)
if err != nil {
return nil, err
@@ -75,50 +65,42 @@ func (adm *AdminClient) GetConfig() ([]byte, error) {
}
// SetConfig - set config supplied as config.json for the setup.
func (adm *AdminClient) SetConfig(config io.Reader) (SetConfigResult, error) {
queryVal := url.Values{}
queryVal.Set(configQueryParam, "")
// Set x-minio-operation to set.
hdrs := make(http.Header)
hdrs.Set(minioAdminOpHeader, "set")
func (adm *AdminClient) SetConfig(config io.Reader) (r SetConfigResult, err error) {
// No TLS?
if !adm.secure {
return r, fmt.Errorf("credentials/configuration cannot be updated over an insecure connection")
}
// Read config bytes to calculate MD5, SHA256 and content length.
configBytes, err := ioutil.ReadAll(config)
if err != nil {
return SetConfigResult{}, err
return r, err
}
reqData := requestData{
queryValues: queryVal,
customHeaders: hdrs,
relPath: "/v1/config",
contentBody: bytes.NewReader(configBytes),
contentMD5Bytes: sumMD5(configBytes),
contentSHA256Bytes: sum256(configBytes),
}
// Execute PUT on /?config to set config.
// Execute PUT on /minio/admin/v1/config to set config.
resp, err := adm.executeMethod("PUT", reqData)
defer closeResponse(resp)
if err != nil {
return SetConfigResult{}, err
return r, err
}
if resp.StatusCode != http.StatusOK {
return SetConfigResult{}, httpRespToErrorResponse(resp)
return r, httpRespToErrorResponse(resp)
}
var result SetConfigResult
jsonBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return SetConfigResult{}, err
return r, err
}
err = json.Unmarshal(jsonBytes, &result)
if err != nil {
return SetConfigResult{}, err
}
return result, nil
err = json.Unmarshal(jsonBytes, &r)
return r, err
}

3
pkg/madmin/constants.go
View File

@@ -19,7 +19,4 @@ package madmin
const (
// Unsigned payload.
unsignedPayload = "UNSIGNED-PAYLOAD"
// Admin operation header.
minioAdminOpHeader = "X-Minio-Operation"
)

48
pkg/madmin/generic-commands.go
View File

@@ -19,46 +19,50 @@ package madmin
import (
"bytes"
"encoding/xml"
"encoding/json"
"fmt"
"net/http"
"net/url"
)
// setCredsReq - xml to send to the server to set new credentials
type setCredsReq struct {
Username string `xml:"username"`
Password string `xml:"password"`
// SetCredsReq - xml to send to the server to set new credentials
type SetCredsReq struct {
AccessKey string `json:"accessKey"`
SecretKey string `json:"secretKey"`
}
// SetCredentials - Call Set Credentials API to set new access and secret keys in the specified Minio server
// SetCredentials - Call Set Credentials API to set new access and
// secret keys in the specified Minio server
func (adm *AdminClient) SetCredentials(access, secret string) error {
// Setup new request
reqData := requestData{}
reqData.queryValues = make(url.Values)
reqData.queryValues.Set("service", "")
reqData.customHeaders = make(http.Header)
reqData.customHeaders.Set(minioAdminOpHeader, "set-credentials")
// Setup request's body
body, err := xml.Marshal(setCredsReq{Username: access, Password: secret})
body, err := json.Marshal(SetCredsReq{access, secret})
if err != nil {
return err
}
reqData.contentBody = bytes.NewReader(body)
reqData.contentLength = int64(len(body))
reqData.contentMD5Bytes = sumMD5(body)
reqData.contentSHA256Bytes = sum256(body)
// No TLS?
if !adm.secure {
return fmt.Errorf("credentials cannot be updated over an insecure connection")
}
// Setup new request
reqData := requestData{
relPath: "/v1/config/credential",
contentBody: bytes.NewReader(body),
contentLength: int64(len(body)),
contentMD5Bytes: sumMD5(body),
contentSHA256Bytes: sum256(body),
}
// Execute GET on bucket to list objects.
resp, err := adm.executeMethod("POST", reqData)
resp, err := adm.executeMethod("PUT", reqData)
defer closeResponse(resp)
if err != nil {
return err
}
// Return error to the caller if http response code is different from 200
// Return error to the caller if http response code is
// different from 200
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp)
}

509
pkg/madmin/heal-commands.go
View File

@@ -20,456 +20,157 @@
package madmin
import (
"bytes"
"encoding/json"
"encoding/xml"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"time"
)
// listBucketHealResult container for listObjects response.
type listBucketHealResult struct {
// A response can contain CommonPrefixes only if you have
// specified a delimiter.
CommonPrefixes []commonPrefix
// Metadata about each object returned.
Contents []ObjectInfo
Delimiter string
// Encoding type used to encode object keys in the response.
EncodingType string
// A flag that indicates whether or not ListObjects returned all of the results
// that satisfied the search criteria.
IsTruncated bool
Marker string
MaxKeys int64
Name string
// When response is truncated (the IsTruncated element value in
// the response is true), you can use the key name in this field
// as marker in the subsequent request to get next set of objects.
// Object storage lists objects in alphabetical order Note: This
// element is returned only if you have delimiter request
// parameter specified. If response does not include the NextMaker
// and it is truncated, you can use the value of the last Key in
// the response as the marker in the subsequent request to get the
// next set of object keys.
NextMarker string
Prefix string
// HealOpts - collection of options for a heal sequence
type HealOpts struct {
Recursive bool `json:"recursive"`
DryRun bool `json:"dryRun"`
}
// commonPrefix container for prefix response.
type commonPrefix struct {
Prefix string
// HealStartSuccess - holds information about a successfully started
// heal operation
type HealStartSuccess struct {
ClientToken string `json:"clientToken"`
ClientAddress string `json:"clientAddress"`
StartTime time.Time `json:"startTime"`
}
// Owner - bucket owner/principal
type Owner struct {
ID string
DisplayName string
// HealTaskStatus - status struct for a heal task
type HealTaskStatus struct {
Summary string `json:"summary"`
FailureDetail string `json:"detail"`
StartTime time.Time `json:"startTime"`
HealSettings HealOpts `json:"settings"`
NumDisks int `json:"numDisks"`
Items []HealResultItem `json:"items,omitempty"`
}
// Bucket container for bucket metadata
type Bucket struct {
Name string
CreationDate string // time string of format "2006-01-02T15:04:05.000Z"
HealBucketInfo *HealBucketInfo `xml:"HealBucketInfo,omitempty"`
}
// ListBucketsHealResponse - format for list buckets response
type ListBucketsHealResponse struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListAllMyBucketsResult" json:"-"`
Owner Owner
// Container for one or more buckets.
Buckets struct {
Buckets []Bucket `xml:"Bucket"`
} // Buckets are nested
}
// HealStatus - represents different states of healing an object could be in.
type HealStatus int
// HealItemType - specify the type of heal operation in a healing
// result
type HealItemType string
// HealItemType constants
const (
// Healthy - Object that is already healthy
Healthy HealStatus = iota
// CanHeal - Object can be healed
CanHeal
// Corrupted - Object can't be healed
Corrupted
// QuorumUnavailable - Object can't be healed until read
// quorum is available
QuorumUnavailable
// CanPartiallyHeal - Object can't be healed completely until
// disks with missing parts come online
CanPartiallyHeal
HealItemMetadata HealItemType = "metadata"
HealItemBucket = "bucket"
HealItemBucketMetadata = "bucket-metadata"
HealItemObject = "object"
)
// HealBucketInfo - represents healing related information of a bucket.
type HealBucketInfo struct {
Status HealStatus
}
// BucketInfo - represents bucket metadata.
type BucketInfo struct {
// Name of the bucket.
Name string
// Date and time when the bucket was created.
Created time.Time
// Healing information
HealBucketInfo *HealBucketInfo `xml:"HealBucketInfo,omitempty"`
}
// HealObjectInfo - represents healing related information of an object.
type HealObjectInfo struct {
Status HealStatus
MissingDataCount int
MissingParityCount int
}
// ObjectInfo container for object metadata.
type ObjectInfo struct {
// An ETag is optionally set to md5sum of an object. In case of multipart objects,
// ETag is of the form MD5SUM-N where MD5SUM is md5sum of all individual md5sums of
// each parts concatenated into one string.
ETag string `json:"etag"`
Key string `json:"name"` // Name of the object
LastModified time.Time `json:"lastModified"` // Date and time the object was last modified.
Size int64 `json:"size"` // Size in bytes of the object.
ContentType string `json:"contentType"` // A standard MIME type describing the format of the object data.
// Collection of additional metadata on the object.
// eg: x-amz-meta-*, content-encoding etc.
Metadata http.Header `json:"metadata"`
// Owner name.
Owner struct {
DisplayName string `json:"name"`
ID string `json:"id"`
} `json:"owner"`
// The class of storage used to store the object.
StorageClass string `json:"storageClass"`
// Error
Err error `json:"-"`
HealObjectInfo *HealObjectInfo `json:"healObjectInfo,omitempty"`
}
type healQueryKey string
// Drive state constants
const (
healBucket healQueryKey = "bucket"
healObject healQueryKey = "object"
healPrefix healQueryKey = "prefix"
healMarker healQueryKey = "marker"
healDelimiter healQueryKey = "delimiter"
healMaxKey healQueryKey = "max-key"
healDryRun healQueryKey = "dry-run"
DriveStateOk string = "ok"
DriveStateOffline = "offline"
DriveStateCorrupt = "corrupt"
DriveStateMissing = "missing"
)
// mkHealQueryVal - helper function to construct heal REST API query params.
func mkHealQueryVal(bucket, prefix, marker, delimiter, maxKeyStr string) url.Values {
queryVal := make(url.Values)
queryVal.Set("heal", "")
queryVal.Set(string(healBucket), bucket)
queryVal.Set(string(healPrefix), prefix)
queryVal.Set(string(healMarker), marker)
queryVal.Set(string(healDelimiter), delimiter)
queryVal.Set(string(healMaxKey), maxKeyStr)
return queryVal
// HealResultItem - struct for an individual heal result item
type HealResultItem struct {
ResultIndex int64 `json:"resultId"`
Type HealItemType `json:"type"`
Bucket string `json:"bucket"`
Object string `json:"object"`
Detail string `json:"detail"`
ParityBlocks int `json:"parityBlocks,omitempty"`
DataBlocks int `json:"dataBlocks,omitempty"`
DiskCount int `json:"diskCount"`
DriveInfo struct {
// below maps are from drive endpoint to drive state
Before map[string]string `json:"before"`
After map[string]string `json:"after"`
} `json:"drives"`
ObjectSize int64 `json:"objectSize"`
}
// listObjectsHeal - issues heal list API request for a batch of maxKeys objects to be healed.
func (adm *AdminClient) listObjectsHeal(bucket, prefix, marker, delimiter string, maxKeys int) (listBucketHealResult, error) {
// Construct query params.
maxKeyStr := fmt.Sprintf("%d", maxKeys)
queryVal := mkHealQueryVal(bucket, prefix, marker, delimiter, maxKeyStr)
hdrs := make(http.Header)
hdrs.Set(minioAdminOpHeader, "list-objects")
reqData := requestData{
queryValues: queryVal,
customHeaders: hdrs,
}
// Empty 'list' of objects to be healed.
toBeHealedObjects := listBucketHealResult{}
// Execute GET on /?heal to list objects needing heal.
resp, err := adm.executeMethod("GET", reqData)
defer closeResponse(resp)
if err != nil {
return listBucketHealResult{}, err
}
if resp.StatusCode != http.StatusOK {
return toBeHealedObjects, httpRespToErrorResponse(resp)
}
err = xml.NewDecoder(resp.Body).Decode(&toBeHealedObjects)
return toBeHealedObjects, err
// InitDrives - initialize maps used to represent drive info
func (hri *HealResultItem) InitDrives() {
hri.DriveInfo.Before = make(map[string]string)
hri.DriveInfo.After = make(map[string]string)
}
// ListObjectsHeal - Lists upto maxKeys objects that needing heal matching bucket, prefix, marker, delimiter.
func (adm *AdminClient) ListObjectsHeal(bucket, prefix string, recursive bool, doneCh <-chan struct{}) (<-chan ObjectInfo, error) {
// Allocate new list objects channel.
objectStatCh := make(chan ObjectInfo, 1)
// Default listing is delimited at "/"
delimiter := "/"
if recursive {
// If recursive we do not delimit.
delimiter = ""
// GetOnlineCounts - returns the number of online disks before and
// after heal
func (hri *HealResultItem) GetOnlineCounts() (b, a int) {
if hri == nil {
return
}
// Initiate list objects goroutine here.
go func(objectStatCh chan<- ObjectInfo) {
defer close(objectStatCh)
// Save marker for next request.
var marker string
for {
// Get list of objects a maximum of 1000 per request.
result, err := adm.listObjectsHeal(bucket, prefix, marker, delimiter, 1000)
if err != nil {
objectStatCh <- ObjectInfo{
Err: err,
}
return
}
// If contents are available loop through and send over channel.
for _, object := range result.Contents {
// Save the marker.
marker = object.Key
select {
// Send object content.
case objectStatCh <- object:
// If receives done from the caller, return here.
case <-doneCh:
return
}
}
// Send all common prefixes if any.
// NOTE: prefixes are only present if the request is delimited.
for _, obj := range result.CommonPrefixes {
object := ObjectInfo{}
object.Key = obj.Prefix
object.Size = 0
select {
// Send object prefixes.
case objectStatCh <- object:
// If receives done from the caller, return here.
case <-doneCh:
return
}
}
// If next marker present, save it for next request.
if result.NextMarker != "" {
marker = result.NextMarker
}
// Listing ends result is not truncated, return right here.
if !result.IsTruncated {
return
}
for _, v := range hri.DriveInfo.Before {
if v == DriveStateOk {
b++
}
}(objectStatCh)
return objectStatCh, nil
}
const timeFormatAMZLong = "2006-01-02T15:04:05.000Z" // Reply date format with nanosecond precision.
// ListBucketsHeal - issues heal bucket list API request
func (adm *AdminClient) ListBucketsHeal() ([]BucketInfo, error) {
queryVal := url.Values{}
queryVal.Set("heal", "")
hdrs := make(http.Header)
hdrs.Set(minioAdminOpHeader, "list-buckets")
reqData := requestData{
queryValues: queryVal,
customHeaders: hdrs,
}
// Execute GET on /?heal to list objects needing heal.
resp, err := adm.executeMethod("GET", reqData)
defer closeResponse(resp)
if err != nil {
return []BucketInfo{}, err
}
if resp.StatusCode != http.StatusOK {
return []BucketInfo{}, httpRespToErrorResponse(resp)
}
var listBucketsHealResult ListBucketsHealResponse
err = xml.NewDecoder(resp.Body).Decode(&listBucketsHealResult)
if err != nil {
return []BucketInfo{}, err
}
var bucketsToBeHealed []BucketInfo
for _, bucket := range listBucketsHealResult.Buckets.Buckets {
creationDate, err := time.Parse(timeFormatAMZLong, bucket.CreationDate)
if err != nil {
return []BucketInfo{}, err
for _, v := range hri.DriveInfo.After {
if v == DriveStateOk {
a++
}
bucketsToBeHealed = append(bucketsToBeHealed,
BucketInfo{
Name: bucket.Name,
Created: creationDate,
HealBucketInfo: bucket.HealBucketInfo,
})
}
return bucketsToBeHealed, nil
return
}
// HealBucket - Heal the given bucket
func (adm *AdminClient) HealBucket(bucket string, dryrun bool) error {
// Construct query params.
queryVal := url.Values{}
queryVal.Set("heal", "")
queryVal.Set(string(healBucket), bucket)
if dryrun {
queryVal.Set(string(healDryRun), "")
// Heal - API endpoint to start heal and to fetch status
func (adm *AdminClient) Heal(bucket, prefix string, healOpts HealOpts,
clientToken string, forceStart bool) (
healStart HealStartSuccess, healTaskStatus HealTaskStatus, err error) {
body, err := json.Marshal(healOpts)
if err != nil {
return healStart, healTaskStatus, err
}
hdrs := make(http.Header)
hdrs.Set(minioAdminOpHeader, "bucket")
reqData := requestData{
queryValues: queryVal,
customHeaders: hdrs,
path := fmt.Sprintf("/v1/heal/%s", bucket)
if bucket != "" && prefix != "" {
path += "/" + prefix
}
// Execute POST on /?heal&bucket=mybucket to heal a bucket.
resp, err := adm.executeMethod("POST", reqData)
// execute POST request to heal api
queryVals := make(url.Values)
var contentBody io.Reader
if clientToken != "" {
queryVals.Set("clientToken", clientToken)
} else {
// Set a body only if clientToken is not given
contentBody = bytes.NewReader(body)
}
if forceStart {
queryVals.Set("forceStart", "true")
}
resp, err := adm.executeMethod("POST", requestData{
relPath: path,
contentBody: contentBody,
contentSHA256Bytes: sum256(body),
queryValues: queryVals,
})
defer closeResponse(resp)
if err != nil {
return err
return healStart, healTaskStatus, err
}
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp)
return healStart, healTaskStatus, httpRespToErrorResponse(resp)
}
return nil
}
// HealResult - represents result of heal-object admin API.
type HealResult struct {
State HealState `json:"state"`
}
// HealState - different states of heal operation
type HealState int
const (
// HealNone - none of the disks healed
HealNone HealState = iota
// HealPartial - some disks were healed, others were offline
HealPartial
// HealOK - all disks were healed
HealOK
)
// HealObject - Heal the given object.
func (adm *AdminClient) HealObject(bucket, object string, dryrun bool) (HealResult, error) {
// Construct query params.
queryVal := url.Values{}
queryVal.Set("heal", "")
queryVal.Set(string(healBucket), bucket)
queryVal.Set(string(healObject), object)
if dryrun {
queryVal.Set(string(healDryRun), "")
}
hdrs := make(http.Header)
hdrs.Set(minioAdminOpHeader, "object")
reqData := requestData{
queryValues: queryVal,
customHeaders: hdrs,
}
// Execute POST on /?heal&bucket=mybucket&object=myobject to heal an object.
resp, err := adm.executeMethod("POST", reqData)
defer closeResponse(resp)
respBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return HealResult{}, err
return healStart, healTaskStatus, err
}
if resp.StatusCode != http.StatusOK {
return HealResult{}, httpRespToErrorResponse(resp)
// Was it a status request?
if clientToken == "" {
err = json.Unmarshal(respBytes, &healStart)
} else {
err = json.Unmarshal(respBytes, &healTaskStatus)
}
// Healing is not performed so heal object result is empty.
if dryrun {
return HealResult{}, nil
}
jsonBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return HealResult{}, err
}
healResult := HealResult{}
err = json.Unmarshal(jsonBytes, &healResult)
if err != nil {
return HealResult{}, err
}
return healResult, nil
}
// HealFormat - heal storage format on available disks.
func (adm *AdminClient) HealFormat(dryrun bool) error {
queryVal := url.Values{}
queryVal.Set("heal", "")
if dryrun {
queryVal.Set(string(healDryRun), "")
}
// Set x-minio-operation to format.
hdrs := make(http.Header)
hdrs.Set(minioAdminOpHeader, "format")
reqData := requestData{
queryValues: queryVal,
customHeaders: hdrs,
}
// Execute POST on /?heal to heal storage format.
resp, err := adm.executeMethod("POST", reqData)
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp)
}
return nil
return healStart, healTaskStatus, err
}

9
pkg/madmin/info-commands.go
View File

@@ -21,7 +21,6 @@ import (
"encoding/json"
"io/ioutil"
"net/http"
"net/url"
"time"
)
@@ -115,13 +114,7 @@ type ServerInfo struct {
// ServerInfo - Connect to a minio server and call Server Info Management API
// to fetch server's information represented by ServerInfo structure
func (adm *AdminClient) ServerInfo() ([]ServerInfo, error) {
// Prepare web service request
reqData := requestData{}
reqData.queryValues = make(url.Values)
reqData.queryValues.Set("info", "")
reqData.customHeaders = make(http.Header)
resp, err := adm.executeMethod("GET", reqData)
resp, err := adm.executeMethod("GET", requestData{relPath: "/v1/info"})
defer closeResponse(resp)
if err != nil {
return nil, err

42
pkg/madmin/lock-commands.go
View File

@@ -82,24 +82,19 @@ func getLockInfos(body io.Reader) ([]VolumeLockInfo, error) {
// ListLocks - Calls List Locks Management API to fetch locks matching
// bucket, prefix and held before the duration supplied.
func (adm *AdminClient) ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error) {
func (adm *AdminClient) ListLocks(bucket, prefix string,
duration time.Duration) ([]VolumeLockInfo, error) {
queryVal := make(url.Values)
queryVal.Set("lock", "")
queryVal.Set("bucket", bucket)
queryVal.Set("prefix", prefix)
queryVal.Set("duration", duration.String())
hdrs := make(http.Header)
hdrs.Set(minioAdminOpHeader, "list")
reqData := requestData{
queryValues: queryVal,
customHeaders: hdrs,
}
// Execute GET on /?lock to list locks.
resp, err := adm.executeMethod("GET", reqData)
queryVal.Set("older-than", duration.String())
// Execute GET on /minio/admin/v1/locks to list locks.
resp, err := adm.executeMethod("GET", requestData{
queryValues: queryVal,
relPath: "/v1/locks",
})
defer closeResponse(resp)
if err != nil {
return nil, err
@@ -114,24 +109,19 @@ func (adm *AdminClient) ListLocks(bucket, prefix string, duration time.Duration)
// ClearLocks - Calls Clear Locks Management API to clear locks held
// on bucket, matching prefix older than duration supplied.
func (adm *AdminClient) ClearLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error) {
func (adm *AdminClient) ClearLocks(bucket, prefix string,
duration time.Duration) ([]VolumeLockInfo, error) {
queryVal := make(url.Values)
queryVal.Set("lock", "")
queryVal.Set("bucket", bucket)
queryVal.Set("prefix", prefix)
queryVal.Set("duration", duration.String())
hdrs := make(http.Header)
hdrs.Set(minioAdminOpHeader, "clear")
reqData := requestData{
queryValues: queryVal,
customHeaders: hdrs,
}
// Execute POST on /?lock to clear locks.
resp, err := adm.executeMethod("POST", reqData)
resp, err := adm.executeMethod("DELETE", requestData{
queryValues: queryVal,
relPath: "/v1/locks",
})
defer closeResponse(resp)
if err != nil {
return nil, err

86
pkg/madmin/service-commands.go
View File

@@ -18,69 +18,79 @@
package madmin
import (
"bytes"
"encoding/json"
"io/ioutil"
"net/http"
"net/url"
"time"
)
// ServiceStatusMetadata - contains the response of service status API
type ServiceStatusMetadata struct {
Uptime time.Duration `json:"uptime"`
// ServerVersion - server version
type ServerVersion struct {
Version string `json:"version"`
CommitID string `json:"commitID"`
}
// ServiceStatus - Connect to a minio server and call Service Status Management API
// to fetch server's storage information represented by ServiceStatusMetadata structure
func (adm *AdminClient) ServiceStatus() (ServiceStatusMetadata, error) {
// ServiceStatus - contains the response of service status API
type ServiceStatus struct {
ServerVersion ServerVersion `json:"serverVersion"`
Uptime time.Duration `json:"uptime"`
}
// Prepare web service request
reqData := requestData{}
reqData.queryValues = make(url.Values)
reqData.queryValues.Set("service", "")
reqData.customHeaders = make(http.Header)
reqData.customHeaders.Set(minioAdminOpHeader, "status")
// Execute GET on bucket to list objects.
resp, err := adm.executeMethod("GET", reqData)
// ServiceStatus - Connect to a minio server and call Service Status
// Management API to fetch server's storage information represented by
// ServiceStatusMetadata structure
func (adm *AdminClient) ServiceStatus() (ss ServiceStatus, err error) {
// Request API to GET service status
resp, err := adm.executeMethod("GET", requestData{relPath: "/v1/service"})
defer closeResponse(resp)
if err != nil {
return ServiceStatusMetadata{}, err
return ss, err
}
// Check response http status code
if resp.StatusCode != http.StatusOK {
return ServiceStatusMetadata{}, httpRespToErrorResponse(resp)
return ss, httpRespToErrorResponse(resp)
}
// Unmarshal the server's json response
var serviceStatus ServiceStatusMetadata
respBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return ServiceStatusMetadata{}, err
return ss, err
}
err = json.Unmarshal(respBytes, &serviceStatus)
if err != nil {
return ServiceStatusMetadata{}, err
}
return serviceStatus, nil
err = json.Unmarshal(respBytes, &ss)
return ss, err
}
// ServiceRestart - Call Service Restart API to restart a specified Minio server
func (adm *AdminClient) ServiceRestart() error {
//
reqData := requestData{}
reqData.queryValues = make(url.Values)
reqData.queryValues.Set("service", "")
reqData.customHeaders = make(http.Header)
reqData.customHeaders.Set(minioAdminOpHeader, "restart")
// ServiceActionValue - type to restrict service-action values
type ServiceActionValue string
// Execute GET on bucket to list objects.
resp, err := adm.executeMethod("POST", reqData)
const (
// ServiceActionValueRestart represents restart action
ServiceActionValueRestart ServiceActionValue = "restart"
// ServiceActionValueStop represents stop action
ServiceActionValueStop = "stop"
)
// ServiceAction - represents POST body for service action APIs
type ServiceAction struct {
Action ServiceActionValue `json:"action"`
}
// ServiceSendAction - Call Service Restart/Stop API to restart/stop a
// Minio server
func (adm *AdminClient) ServiceSendAction(action ServiceActionValue) error {
body, err := json.Marshal(ServiceAction{action})
if err != nil {
return err
}
// Request API to Restart server
resp, err := adm.executeMethod("POST", requestData{
relPath: "/v1/service",
contentBody: bytes.NewReader(body),
contentSHA256Bytes: sum256(body),
})
defer closeResponse(resp)
if err != nil {
return err

8
pkg/madmin/utils.go
View File

@@ -18,7 +18,7 @@ package madmin
import (
"crypto/md5"
"encoding/xml"
"encoding/json"
"io"
"io/ioutil"
"net"
@@ -45,9 +45,9 @@ func sumMD5(data []byte) []byte {
return hash.Sum(nil)
}
// xmlDecoder provide decoded value in xml.
func xmlDecoder(body io.Reader, v interface{}) error {
d := xml.NewDecoder(body)
// jsonDecoder decode json to go type.
func jsonDecoder(body io.Reader, v interface{}) error {
d := json.NewDecoder(body)
return d.Decode(v)
}

54
pkg/madmin/version-commands.go Normal file
View File

@@ -0,0 +1,54 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package madmin
import (
"encoding/json"
"io/ioutil"
"net/http"
)
// AdminAPIVersionInfo - contains admin API version information
type AdminAPIVersionInfo struct {
Version string `json:"version"`
}
// VersionInfo - Connect to minio server and call the version API to
// retrieve the server API version
func (adm *AdminClient) VersionInfo() (verInfo AdminAPIVersionInfo, err error) {
var resp *http.Response
resp, err = adm.executeMethod("GET", requestData{relPath: "/version"})
defer closeResponse(resp)
if err != nil {
return verInfo, err
}
// Check response http status code
if resp.StatusCode != http.StatusOK {
return verInfo, httpRespToErrorResponse(resp)
}
respBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return verInfo, err
}
// Unmarshal the server's json response
err = json.Unmarshal(respBytes, &verInfo)
return verInfo, err
}