From 9dda1fd6241955f6268fa158a25e70a9c24591d2 Mon Sep 17 00:00:00 2001 From: Harshavardhana Date: Thu, 7 May 2020 19:00:30 -0700 Subject: [PATCH] Remove B2 gateway implementation (#9547) S3 is now natively supported by B2 cloud storage provider there is no reason to use specialized gateway for B2 anymore, our current S3 gateway with caching would work with B2. Resolves #8584 --- CREDITS | 252 +------ cmd/gateway/b2/gateway-b2.go | 1073 ----------------------------- cmd/gateway/b2/gateway-b2_test.go | 146 ---- cmd/gateway/gateway.go | 26 +- docs/gateway/README.md | 2 +- docs/gateway/b2.md | 60 -- go.mod | 1 - go.sum | 65 +- 8 files changed, 104 insertions(+), 1521 deletions(-) delete mode 100644 cmd/gateway/b2/gateway-b2.go delete mode 100644 cmd/gateway/b2/gateway-b2_test.go delete mode 100644 docs/gateway/b2.md diff --git a/CREDITS b/CREDITS index 076085b99..a41a33132 100644 --- a/CREDITS +++ b/CREDITS @@ -4487,214 +4487,6 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ================================================================ -github.com/google/gofuzz -https://github.com/google/gofuzz ----------------------------------------------------------------- - - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - -================================================================ - github.com/google/martian https://github.com/google/martian ---------------------------------------------------------------- @@ -10936,25 +10728,6 @@ THE SOFTWARE. ================================================================ -github.com/kurin/blazer -https://github.com/kurin/blazer ----------------------------------------------------------------- -Copyright 2016, the Blazer authors - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. - -================================================================ - github.com/lib/pq https://github.com/lib/pq ---------------------------------------------------------------- @@ -17942,6 +17715,31 @@ https://go.opencensus.io limitations under the License. ================================================================ +go.uber.org/atomic +https://go.uber.org/atomic +---------------------------------------------------------------- +Copyright (c) 2016 Uber Technologies, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +================================================================ + go.uber.org/multierr https://go.uber.org/multierr ---------------------------------------------------------------- diff --git a/cmd/gateway/b2/gateway-b2.go b/cmd/gateway/b2/gateway-b2.go deleted file mode 100644 index 09ea16b6a..000000000 --- a/cmd/gateway/b2/gateway-b2.go +++ /dev/null @@ -1,1073 +0,0 @@ -/* - * MinIO Cloud Storage, (C) 2017-2020 MinIO, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package b2 - -import ( - "context" - "crypto/sha1" - "fmt" - "hash" - "io" - "net/http" - - "strings" - "sync" - "time" - - b2 "github.com/kurin/blazer/base" - "github.com/minio/cli" - miniogopolicy "github.com/minio/minio-go/v6/pkg/policy" - "github.com/minio/minio/cmd/logger" - "github.com/minio/minio/pkg/auth" - "github.com/minio/minio/pkg/bucket/policy" - "github.com/minio/minio/pkg/bucket/policy/condition" - h2 "github.com/minio/minio/pkg/hash" - - minio "github.com/minio/minio/cmd" -) - -// Supported bucket types by B2 backend. -const ( - bucketTypePrivate = "allPrivate" - bucketTypeReadOnly = "allPublic" - b2Backend = "b2" -) - -func init() { - const b2GatewayTemplate = `NAME: - {{.HelpName}} - {{.Usage}} - -USAGE: - {{.HelpName}} {{if .VisibleFlags}}[FLAGS]{{end}} -{{if .VisibleFlags}} -FLAGS: - {{range .VisibleFlags}}{{.}} - {{end}}{{end}} - -EXAMPLES: - 1. Start minio gateway server for B2 backend - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accountID - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}applicationKey - {{.Prompt}} {{.HelpName}} - - 2. Start minio gateway server for B2 backend with edge caching enabled - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accountID - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}applicationKey - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_DRIVES{{.AssignmentOperator}}"/mnt/drive1,/mnt/drive2,/mnt/drive3,/mnt/drive4" - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXCLUDE{{.AssignmentOperator}}"bucket1/*,*.png" - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_QUOTA{{.AssignmentOperator}}90 - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_AFTER{{.AssignmentOperator}}3 - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_WATERMARK_LOW{{.AssignmentOperator}}75 - {{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_WATERMARK_HIGH{{.AssignmentOperator}}85 - - {{.Prompt}} {{.HelpName}} -` - minio.RegisterGatewayCommand(cli.Command{ - Name: b2Backend, - Usage: "Backblaze B2", - Action: b2GatewayMain, - CustomHelpTemplate: b2GatewayTemplate, - HideHelpCommand: true, - }) -} - -// Handler for 'minio gateway b2' command line. -func b2GatewayMain(ctx *cli.Context) { - strictS3Compat := true - if ctx.IsSet("no-compat") || ctx.GlobalIsSet("no-compat") { - strictS3Compat = false - } - - minio.StartGateway(ctx, &B2{ - strictS3Compat: strictS3Compat, - }) -} - -// B2 implements MinIO Gateway -type B2 struct { - strictS3Compat bool -} - -// Name implements Gateway interface. -func (g *B2) Name() string { - return b2Backend -} - -// NewGatewayLayer returns b2 gateway layer, implements ObjectLayer interface to -// talk to B2 remote backend. -func (g *B2) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, error) { - ctx := minio.GlobalContext - client, err := b2.AuthorizeAccount(ctx, creds.AccessKey, creds.SecretKey, b2.Transport(minio.NewGatewayHTTPTransport())) - if err != nil { - return nil, err - } - - return &b2Objects{ - creds: creds, - b2Client: client, - httpClient: &http.Client{ - Transport: minio.NewGatewayHTTPTransport(), - }, - ctx: ctx, - strictS3Compat: g.strictS3Compat, - }, nil -} - -// Production - Ready for production use. -func (g *B2) Production() bool { - return true -} - -// b2Object implements gateway for MinIO and BackBlaze B2 compatible object storage servers. -type b2Objects struct { - minio.GatewayUnsupported - mu sync.Mutex - creds auth.Credentials - b2Client *b2.B2 - httpClient *http.Client - ctx context.Context - buckets []*b2.Bucket - strictS3Compat bool -} - -// Convert B2 errors to minio object layer errors. -func b2ToObjectError(err error, params ...string) error { - if err == nil { - return nil - } - - code, msgCode, msg := b2.MsgCode(err) - if code == 0 { - // We don't interpret non B2 errors. B2 errors have statusCode - // to help us convert them to S3 object errors. - return err - } - - objErr := b2MsgCodeToObjectError(code, msgCode, msg, params...) - if objErr == nil { - return err - } - return objErr -} - -func b2MsgCodeToObjectError(code int, msgCode string, msg string, params ...string) error { - // Following code is a non-exhaustive check to convert - // B2 errors into S3 compatible errors. - // - // For a more complete information - https://www.backblaze.com/b2/docs/ - - var err error - - bucket := "" - object := "" - uploadID := "" - if len(params) >= 1 { - bucket = params[0] - } - if len(params) == 2 { - object = params[1] - } - if len(params) == 3 { - uploadID = params[2] - } - - switch msgCode { - case "duplicate_bucket_name": - err = minio.BucketAlreadyOwnedByYou{Bucket: bucket} - case "bad_request": - if object != "" { - err = minio.ObjectNameInvalid{ - Bucket: bucket, - Object: object, - } - } else if bucket != "" { - err = minio.BucketNotFound{Bucket: bucket} - } - case "bad_json": - if object != "" { - err = minio.ObjectNameInvalid{ - Bucket: bucket, - Object: object, - } - } else if bucket != "" { - err = minio.BucketNameInvalid{Bucket: bucket} - } - case "bad_bucket_id": - err = minio.BucketNotFound{Bucket: bucket} - case "file_not_present", "not_found": - err = minio.ObjectNotFound{ - Bucket: bucket, - Object: object, - } - case "cannot_delete_non_empty_bucket": - err = minio.BucketNotEmpty{Bucket: bucket} - } - - // Special interpretation like this is required for Multipart sessions. - if strings.Contains(msg, "No active upload for") && uploadID != "" { - err = minio.InvalidUploadID{UploadID: uploadID} - } - - return err -} - -// Shutdown saves any gateway metadata to disk -// if necessary and reload upon next restart. -func (l *b2Objects) Shutdown(ctx context.Context) error { - return nil -} - -// StorageInfo is not relevant to B2 backend. -func (l *b2Objects) StorageInfo(ctx context.Context, _ bool) (si minio.StorageInfo) { - si.Backend.Type = minio.BackendGateway - si.Backend.GatewayOnline = minio.IsBackendOnline(ctx, l.httpClient, "https://api.backblazeb2.com/b2api/v1") - return si -} - -// MakeBucket creates a new container on B2 backend. -func (l *b2Objects) MakeBucketWithLocation(ctx context.Context, bucket, location string) error { - // location is ignored for B2 backend. - - // All buckets are set to private by default. - _, err := l.b2Client.CreateBucket(l.ctx, bucket, bucketTypePrivate, nil, nil) - logger.LogIf(ctx, err) - return b2ToObjectError(err, bucket) -} - -func (l *b2Objects) reAuthorizeAccount(ctx context.Context) error { - client, err := b2.AuthorizeAccount(l.ctx, l.creds.AccessKey, l.creds.SecretKey, b2.Transport(minio.NewGatewayHTTPTransport())) - if err != nil { - return err - } - l.mu.Lock() - l.b2Client.Update(client) - l.mu.Unlock() - return nil -} - -// getETag returns an S3-compatible md5sum-ETag if requested and possible. -func (l *b2Objects) getETag(f *b2.File, fi *b2.FileInfo) string { - if l.strictS3Compat && fi.MD5 != "" { - return fi.MD5 - } - - return minio.ToS3ETag(f.ID) -} - -// listBuckets is a wrapper similar to ListBuckets, which re-authorizes -// the account and updates the B2 client safely. Once successfully -// authorized performs the call again and returns list of buckets. -// For any errors which are not actionable we return an error. -func (l *b2Objects) listBuckets(ctx context.Context, err error) ([]*b2.Bucket, error) { - if err != nil { - if b2.Action(err) != b2.ReAuthenticate { - return nil, err - } - if rerr := l.reAuthorizeAccount(ctx); rerr != nil { - return nil, rerr - } - } - if len(l.buckets) == 0 { - bktList, lerr := l.b2Client.ListBuckets(l.ctx) - if lerr != nil { - return l.listBuckets(ctx, lerr) - } - l.buckets = bktList - } - return l.buckets, nil -} - -// Bucket - is a helper which provides a *Bucket instance -// for performing an API operation. B2 API doesn't -// provide a direct way to access the bucket so we need -// to employ following technique. -func (l *b2Objects) Bucket(ctx context.Context, bucket string) (*b2.Bucket, error) { - bktList, err := l.listBuckets(ctx, nil) - if err != nil { - logger.LogIf(ctx, err) - return nil, b2ToObjectError(err, bucket) - } - for _, bkt := range bktList { - if bkt.Name == bucket { - return bkt, nil - } - } - return nil, minio.BucketNotFound{Bucket: bucket} -} - -// GetBucketInfo gets bucket metadata.. -func (l *b2Objects) GetBucketInfo(ctx context.Context, bucket string) (bi minio.BucketInfo, err error) { - if _, err = l.Bucket(ctx, bucket); err != nil { - return bi, err - } - return minio.BucketInfo{ - Name: bucket, - Created: time.Unix(0, 0), - }, nil -} - -// ListBuckets lists all B2 buckets -func (l *b2Objects) ListBuckets(ctx context.Context) ([]minio.BucketInfo, error) { - bktList, err := l.listBuckets(ctx, nil) - if err != nil { - return nil, err - } - var bktInfo []minio.BucketInfo - for _, bkt := range bktList { - bktInfo = append(bktInfo, minio.BucketInfo{ - Name: bkt.Name, - Created: time.Unix(0, 0), - }) - } - return bktInfo, nil -} - -// DeleteBucket deletes a bucket on B2 -func (l *b2Objects) DeleteBucket(ctx context.Context, bucket string, forceDelete bool) error { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - err = bkt.DeleteBucket(l.ctx) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - - err = bkt.DeleteBucket(l.ctx) - - logger.LogIf(ctx, err) - return b2ToObjectError(err, bucket) - } - - logger.LogIf(ctx, err) - return b2ToObjectError(err, bucket) -} - -// ListObjects lists all objects in B2 bucket filtered by prefix, returns upto at max 1000 entries at a time. -func (l *b2Objects) ListObjects(ctx context.Context, bucket string, prefix string, marker string, delimiter string, maxKeys int) (loi minio.ListObjectsInfo, err error) { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return loi, err - } - files, next, lerr := bkt.ListFileNames(l.ctx, maxKeys, marker, prefix, delimiter) - if lerr != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return loi, err - } - - files, next, lerr = bkt.ListFileNames(l.ctx, maxKeys, marker, prefix, delimiter) - if lerr != nil { - logger.LogIf(ctx, lerr) - return loi, b2ToObjectError(lerr, bucket) - } - } - loi.IsTruncated = next != "" - loi.NextMarker = next - for _, file := range files { - switch file.Status { - case "folder": - loi.Prefixes = append(loi.Prefixes, file.Name) - case "upload": - loi.Objects = append(loi.Objects, minio.ObjectInfo{ - Bucket: bucket, - Name: file.Name, - ModTime: file.Timestamp, - Size: file.Size, - ETag: l.getETag(file, file.Info), - ContentType: file.Info.ContentType, - UserDefined: file.Info.Info, - }) - } - } - return loi, nil -} - -// ListObjectsV2 lists all objects in B2 bucket filtered by prefix, returns upto max 1000 entries at a time. -func (l *b2Objects) ListObjectsV2(ctx context.Context, bucket, prefix, continuationToken, delimiter string, maxKeys int, - fetchOwner bool, startAfter string) (loi minio.ListObjectsV2Info, err error) { - // fetchOwner is not supported and unused. - marker := continuationToken - if marker == "" { - // B2's continuation token is an object name to "start at" rather than "start after" - // startAfter plus the lowest character B2 supports is used so that the startAfter - // object isn't included in the results - marker = startAfter + " " - } - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return loi, err - } - files, next, lerr := bkt.ListFileNames(l.ctx, maxKeys, marker, prefix, delimiter) - if lerr != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return loi, err - } - - files, next, lerr = bkt.ListFileNames(l.ctx, maxKeys, marker, prefix, delimiter) - if lerr != nil { - logger.LogIf(ctx, lerr) - return loi, b2ToObjectError(lerr, bucket) - } - } - loi.IsTruncated = next != "" - loi.ContinuationToken = continuationToken - loi.NextContinuationToken = next - for _, file := range files { - switch file.Status { - case "folder": - loi.Prefixes = append(loi.Prefixes, file.Name) - case "upload": - loi.Objects = append(loi.Objects, minio.ObjectInfo{ - Bucket: bucket, - Name: file.Name, - ModTime: file.Timestamp, - Size: file.Size, - ETag: l.getETag(file, file.Info), - ContentType: file.Info.ContentType, - UserDefined: file.Info.Info, - }) - } - } - return loi, nil -} - -// GetObjectNInfo - returns object info and locked object ReadCloser -func (l *b2Objects) GetObjectNInfo(ctx context.Context, bucket, object string, rs *minio.HTTPRangeSpec, h http.Header, lockType minio.LockType, opts minio.ObjectOptions) (gr *minio.GetObjectReader, err error) { - var objInfo minio.ObjectInfo - objInfo, err = l.GetObjectInfo(ctx, bucket, object, opts) - if err != nil { - return nil, err - } - - var startOffset, length int64 - startOffset, length, err = rs.GetOffsetLength(objInfo.Size) - if err != nil { - return nil, err - } - - pr, pw := io.Pipe() - go func() { - err := l.GetObject(ctx, bucket, object, startOffset, length, pw, "", opts) - pw.CloseWithError(err) - }() - // Setup cleanup function to cause the above go-routine to - // exit in case of partial read - pipeCloser := func() { pr.Close() } - return minio.NewGetObjectReaderFromReader(pr, objInfo, opts, pipeCloser) -} - -// GetObject reads an object from B2. Supports additional -// parameters like offset and length which are synonymous with -// HTTP Range requests. -// -// startOffset indicates the starting read location of the object. -// length indicates the total length of the object. -func (l *b2Objects) GetObject(ctx context.Context, bucket string, object string, startOffset int64, length int64, writer io.Writer, etag string, opts minio.ObjectOptions) error { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - reader, err := bkt.DownloadFileByName(l.ctx, object, startOffset, length, false) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - - reader, err = bkt.DownloadFileByName(l.ctx, object, startOffset, length, false) - if err != nil { - logger.LogIf(ctx, err) - return b2ToObjectError(err, bucket, object) - } - } - defer reader.Close() - _, err = io.Copy(writer, reader) - logger.LogIf(ctx, err) - return b2ToObjectError(err, bucket, object) -} - -// GetObjectInfo reads object info and replies back ObjectInfo -func (l *b2Objects) GetObjectInfo(ctx context.Context, bucket string, object string, opts minio.ObjectOptions) (objInfo minio.ObjectInfo, err error) { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return objInfo, err - } - - f, _, err := bkt.ListFileNames(l.ctx, 1, object, "", "") - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return objInfo, err - } - - f, _, err = bkt.ListFileNames(l.ctx, 1, object, "", "") - if err != nil { - logger.LogIf(ctx, err) - return objInfo, b2ToObjectError(err, bucket, object) - } - } - - // B2's list will return the next item in the bucket if the object doesn't - // exist so we need to perform a name check too - if len(f) != 1 || (len(f) == 1 && f[0].Name != object) { - return objInfo, minio.ObjectNotFound{ - Bucket: bucket, - Object: object, - } - } - - fi, err := bkt.File(f[0].ID, object).GetFileInfo(l.ctx) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return objInfo, err - } - - fi, err = bkt.File(f[0].ID, object).GetFileInfo(l.ctx) - if err != nil { - logger.LogIf(ctx, err) - return objInfo, b2ToObjectError(err, bucket, object) - } - } - return minio.ObjectInfo{ - Bucket: bucket, - Name: object, - ETag: l.getETag(f[0], fi), - Size: fi.Size, - ModTime: fi.Timestamp, - ContentType: fi.ContentType, - UserDefined: fi.Info, - }, nil -} - -// In B2 - You must always include the X-Bz-Content-Sha1 header with -// your upload request. The value you provide can be: -// (1) the 40-character hex checksum of the file, -// (2) the string hex_digits_at_end, or -// (3) the string do_not_verify. -// For more reference - https://www.backblaze.com/b2/docs/uploading.html -// -// In our case we are going to use (2) option -const sha1AtEOF = "hex_digits_at_end" - -// With the second option mentioned above, you append the 40-character hex sha1 -// to the end of the request body, immediately after the contents of the file -// being uploaded. Note that the content length is the size of the file plus 40 -// of the original size of the reader. -// -// newB2Reader implements a B2 compatible reader by wrapping the hash.Reader into -// a new io.Reader which will emit out the sha1 hex digits at io.EOF. -// It also means that your overall content size is now original size + 40 bytes. -// Additionally this reader also verifies Hash encapsulated inside hash.Reader -// at io.EOF if the verification failed we return an error and do not send -// the content to server. -func newB2Reader(r *h2.Reader, size int64) *Reader { - return &Reader{ - r: r, - size: size, - sha1Hash: sha1.New(), - } -} - -// Reader - is a Reader wraps the hash.Reader which will emit out the sha1 -// hex digits at io.EOF. It also means that your overall content size is -// now original size + 40 bytes. Additionally this reader also verifies -// Hash encapsulated inside hash.Reader at io.EOF if the verification -// failed we return an error and do not send the content to server. -type Reader struct { - r *h2.Reader - size int64 - sha1Hash hash.Hash - - isEOF bool - buf *strings.Reader -} - -// Size - Returns the total size of Reader. -func (nb *Reader) Size() int64 { return nb.size + 40 } -func (nb *Reader) Read(p []byte) (int, error) { - if nb.isEOF { - return nb.buf.Read(p) - } - // Read into hash to update the on going checksum. - n, err := io.TeeReader(nb.r, nb.sha1Hash).Read(p) - if err == io.EOF { - // Stream is not corrupted on this end - // now fill in the last 40 bytes of sha1 hex - // so that the server can verify the stream on - // their end. - err = nil - nb.isEOF = true - nb.buf = strings.NewReader(fmt.Sprintf("%x", nb.sha1Hash.Sum(nil))) - } - return n, err -} - -// PutObject uploads the single upload to B2 backend by using *b2_upload_file* API, uploads upto 5GiB. -func (l *b2Objects) PutObject(ctx context.Context, bucket string, object string, r *minio.PutObjReader, opts minio.ObjectOptions) (objInfo minio.ObjectInfo, err error) { - data := r.Reader - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return objInfo, err - } - contentType := opts.UserDefined["content-type"] - delete(opts.UserDefined, "content-type") - - var u *b2.URL - u, err = bkt.GetUploadURL(l.ctx) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return objInfo, err - } - - u, err = bkt.GetUploadURL(l.ctx) - if err != nil { - logger.LogIf(ctx, err) - return objInfo, b2ToObjectError(err, bucket, object) - } - } - - hr := newB2Reader(data, data.Size()) - var f *b2.File - f, err = u.UploadFile(l.ctx, hr, int(hr.Size()), object, contentType, sha1AtEOF, opts.UserDefined) - if err != nil { - logger.LogIf(ctx, err) - return objInfo, b2ToObjectError(err, bucket, object) - } - - var fi *b2.FileInfo - fi, err = f.GetFileInfo(l.ctx) - if err != nil { - logger.LogIf(ctx, err) - return objInfo, b2ToObjectError(err, bucket, object) - } - - return minio.ObjectInfo{ - Bucket: bucket, - Name: object, - ETag: l.getETag(f, fi), - Size: fi.Size, - ModTime: fi.Timestamp, - ContentType: fi.ContentType, - UserDefined: fi.Info, - }, nil -} - -// DeleteObject deletes a blob in bucket -func (l *b2Objects) DeleteObject(ctx context.Context, bucket string, object string) error { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - - // If we hide the file we'll conform to B2's versioning policy, it also - // saves an additional call to check if the file exists first - _, err = bkt.HideFile(l.ctx, object) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - - _, err = bkt.HideFile(l.ctx, object) - - logger.LogIf(ctx, err) - return b2ToObjectError(err, bucket, object) - } - logger.LogIf(ctx, err) - return b2ToObjectError(err, bucket, object) -} - -func (l *b2Objects) DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error) { - errs := make([]error, len(objects)) - for idx, object := range objects { - errs[idx] = l.DeleteObject(ctx, bucket, object) - } - return errs, nil -} - -// ListMultipartUploads lists all multipart uploads. -func (l *b2Objects) ListMultipartUploads(ctx context.Context, bucket string, prefix string, keyMarker string, uploadIDMarker string, - delimiter string, maxUploads int) (lmi minio.ListMultipartsInfo, err error) { - // keyMarker, prefix, delimiter are all ignored, Backblaze B2 doesn't support any - // of these parameters only equivalent parameter is uploadIDMarker. - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return lmi, err - } - // The maximum number of files to return from this call. - // The default value is 100, and the maximum allowed is 100. - if maxUploads > 100 { - maxUploads = 100 - } - largeFiles, nextMarker, err := bkt.ListUnfinishedLargeFiles(l.ctx, maxUploads, uploadIDMarker) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return lmi, err - } - - largeFiles, nextMarker, err = bkt.ListUnfinishedLargeFiles(l.ctx, maxUploads, uploadIDMarker) - if err != nil { - logger.LogIf(ctx, err) - return lmi, b2ToObjectError(err, bucket) - } - } - lmi = minio.ListMultipartsInfo{ - MaxUploads: maxUploads, - } - if nextMarker != "" { - lmi.IsTruncated = true - lmi.NextUploadIDMarker = nextMarker - } - for _, largeFile := range largeFiles { - lmi.Uploads = append(lmi.Uploads, minio.MultipartInfo{ - Object: largeFile.Name, - UploadID: largeFile.ID, - Initiated: largeFile.Timestamp, - }) - } - return lmi, nil -} - -// NewMultipartUpload upload object in multiple parts, uses B2's LargeFile upload API. -// Large files can range in size from 5MB to 10TB. -// Each large file must consist of at least 2 parts, and all of the parts except the -// last one must be at least 5MB in size. The last part must contain at least one byte. -// For more information - https://www.backblaze.com/b2/docs/large_files.html -func (l *b2Objects) NewMultipartUpload(ctx context.Context, bucket string, object string, opts minio.ObjectOptions) (string, error) { - var uploadID string - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return uploadID, err - } - - contentType := opts.UserDefined["content-type"] - delete(opts.UserDefined, "content-type") - lf, err := bkt.StartLargeFile(l.ctx, object, contentType, opts.UserDefined) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return uploadID, err - } - - lf, err = bkt.StartLargeFile(l.ctx, object, contentType, opts.UserDefined) - if err != nil { - logger.LogIf(ctx, err) - return uploadID, b2ToObjectError(err, bucket, object) - } - } - - return lf.ID, nil -} - -// PutObjectPart puts a part of object in bucket, uses B2's LargeFile upload API. -func (l *b2Objects) PutObjectPart(ctx context.Context, bucket string, object string, uploadID string, partID int, r *minio.PutObjReader, opts minio.ObjectOptions) (pi minio.PartInfo, err error) { - data := r.Reader - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return pi, err - } - - fc, err := bkt.File(uploadID, object).CompileParts(0, nil).GetUploadPartURL(l.ctx) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return pi, err - } - - fc, err = bkt.File(uploadID, object).CompileParts(0, nil).GetUploadPartURL(l.ctx) - if err != nil { - logger.LogIf(ctx, err) - return pi, b2ToObjectError(err, bucket, object, uploadID) - } - } - - hr := newB2Reader(data, data.Size()) - _, err = fc.UploadPart(l.ctx, hr, sha1AtEOF, int(hr.Size()), partID) - if err != nil { - logger.LogIf(ctx, err) - return pi, b2ToObjectError(err, bucket, object, uploadID) - } - - return minio.PartInfo{ - PartNumber: partID, - LastModified: minio.UTCNow(), - ETag: minio.ToS3ETag(fmt.Sprintf("%x", hr.sha1Hash.Sum(nil))), - Size: data.Size(), - }, nil -} - -// ListObjectParts returns all object parts for specified object in specified bucket, uses B2's LargeFile upload API. -func (l *b2Objects) ListObjectParts(ctx context.Context, bucket string, object string, uploadID string, partNumberMarker int, maxParts int, opts minio.ObjectOptions) (lpi minio.ListPartsInfo, err error) { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return lpi, err - } - lpi = minio.ListPartsInfo{ - Bucket: bucket, - Object: object, - UploadID: uploadID, - MaxParts: maxParts, - PartNumberMarker: partNumberMarker, - } - // startPartNumber must be in the range 1 - 10000 for B2. - partNumberMarker++ - partsList, next, err := bkt.File(uploadID, object).ListParts(l.ctx, partNumberMarker, maxParts) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return lpi, err - } - - partsList, next, err = bkt.File(uploadID, object).ListParts(l.ctx, partNumberMarker, maxParts) - if err != nil { - logger.LogIf(ctx, err) - return lpi, b2ToObjectError(err, bucket, object, uploadID) - } - } - if next != 0 { - lpi.IsTruncated = true - lpi.NextPartNumberMarker = next - } - for _, part := range partsList { - lpi.Parts = append(lpi.Parts, minio.PartInfo{ - PartNumber: part.Number, - ETag: minio.ToS3ETag(part.SHA1), - Size: part.Size, - }) - } - return lpi, nil -} - -// AbortMultipartUpload aborts a on going multipart upload, uses B2's LargeFile upload API. -func (l *b2Objects) AbortMultipartUpload(ctx context.Context, bucket string, object string, uploadID string) error { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - err = bkt.File(uploadID, object).CompileParts(0, nil).CancelLargeFile(l.ctx) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - - err = bkt.File(uploadID, object).CompileParts(0, nil).CancelLargeFile(l.ctx) - - logger.LogIf(ctx, err) - return b2ToObjectError(err, bucket, object, uploadID) - } - logger.LogIf(ctx, err) - return b2ToObjectError(err, bucket, object, uploadID) -} - -// CompleteMultipartUpload completes ongoing multipart upload and finalizes object, uses B2's LargeFile upload API. -func (l *b2Objects) CompleteMultipartUpload(ctx context.Context, bucket string, object string, uploadID string, uploadedParts []minio.CompletePart, opts minio.ObjectOptions) (oi minio.ObjectInfo, err error) { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return oi, err - } - hashes := make(map[int]string) - for i, uploadedPart := range uploadedParts { - // B2 requires contigous part numbers starting with 1, they do not support - // hand picking part numbers, we return an S3 compatible error instead. - if i+1 != uploadedPart.PartNumber { - logger.LogIf(ctx, minio.InvalidPart{}) - return oi, b2ToObjectError(minio.InvalidPart{}, bucket, object, uploadID) - } - - // Trim "-1" suffix in ETag as PutObjectPart() treats B2 returned SHA1 as ETag. - hashes[uploadedPart.PartNumber] = strings.TrimSuffix(uploadedPart.ETag, "-1") - } - - if _, err = bkt.File(uploadID, object).CompileParts(0, hashes).FinishLargeFile(l.ctx); err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return oi, err - } - - if _, err = bkt.File(uploadID, object).CompileParts(0, hashes).FinishLargeFile(l.ctx); err != nil { - logger.LogIf(ctx, err) - return oi, b2ToObjectError(err, bucket, object, uploadID) - } - } - - return l.GetObjectInfo(ctx, bucket, object, minio.ObjectOptions{}) -} - -// SetBucketPolicy - B2 supports 2 types of bucket policies: -// bucketType.AllPublic - bucketTypeReadOnly means that anybody can download the files is the bucket; -// bucketType.AllPrivate - bucketTypePrivate means that you need an authorization token to download them. -// Default is AllPrivate for all buckets. -func (l *b2Objects) SetBucketPolicy(ctx context.Context, bucket string, bucketPolicy *policy.Policy) error { - policyInfo, err := minio.PolicyToBucketAccessPolicy(bucketPolicy) - if err != nil { - // This should not happen. - return b2ToObjectError(err, bucket) - } - - var policies []minio.BucketAccessPolicy - for prefix, policy := range miniogopolicy.GetPolicies(policyInfo.Statements, bucket, "") { - policies = append(policies, minio.BucketAccessPolicy{ - Prefix: prefix, - Policy: policy, - }) - } - prefix := bucket + "/*" // For all objects inside the bucket. - if len(policies) != 1 { - logger.LogIf(ctx, minio.NotImplemented{}) - return minio.NotImplemented{} - } - if policies[0].Prefix != prefix { - logger.LogIf(ctx, minio.NotImplemented{}) - return minio.NotImplemented{} - } - if policies[0].Policy != miniogopolicy.BucketPolicyReadOnly { - logger.LogIf(ctx, minio.NotImplemented{}) - return minio.NotImplemented{} - } - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - bkt.Type = bucketTypeReadOnly - _, err = bkt.Update(l.ctx) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - bkt.Type = bucketTypeReadOnly - - _, err = bkt.Update(l.ctx) - - logger.LogIf(ctx, err) - return b2ToObjectError(err) - } - logger.LogIf(ctx, err) - return b2ToObjectError(err) -} - -// GetBucketPolicy, returns the current bucketType from B2 backend and convert -// it into S3 compatible bucket policy info. -func (l *b2Objects) GetBucketPolicy(ctx context.Context, bucket string) (*policy.Policy, error) { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return nil, err - } - - // bkt.Type can also be snapshot, but it is only allowed through B2 browser console, - // just return back as policy not found for all cases. - // CreateBucket always sets the value to allPrivate by default. - if bkt.Type != bucketTypeReadOnly { - return nil, minio.BucketPolicyNotFound{Bucket: bucket} - } - - return &policy.Policy{ - Version: policy.DefaultVersion, - Statements: []policy.Statement{ - policy.NewStatement( - policy.Allow, - policy.NewPrincipal("*"), - policy.NewActionSet( - policy.GetBucketLocationAction, - policy.ListBucketAction, - policy.GetObjectAction, - ), - policy.NewResourceSet( - policy.NewResource(bucket, ""), - policy.NewResource(bucket, "*"), - ), - condition.NewFunctions(), - ), - }, - }, nil -} - -// DeleteBucketPolicy - resets the bucketType of bucket on B2 to 'allPrivate'. -func (l *b2Objects) DeleteBucketPolicy(ctx context.Context, bucket string) error { - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - bkt.Type = bucketTypePrivate - _, err = bkt.Update(l.ctx) - if err != nil { - l.buckets = []*b2.Bucket{} - - bkt, err := l.Bucket(ctx, bucket) - if err != nil { - return err - } - bkt.Type = bucketTypePrivate - - _, err = bkt.Update(l.ctx) - - logger.LogIf(ctx, err) - return b2ToObjectError(err) - } - logger.LogIf(ctx, err) - return b2ToObjectError(err) -} - -// IsCompressionSupported returns whether compression is applicable for this layer. -func (l *b2Objects) IsCompressionSupported() bool { - return false -} - -// IsReady returns whether the layer is ready to take requests. -func (l *b2Objects) IsReady(ctx context.Context) bool { - return minio.IsBackendOnline(ctx, l.httpClient, "https://api.backblazeb2.com/b2api/v1") -} diff --git a/cmd/gateway/b2/gateway-b2_test.go b/cmd/gateway/b2/gateway-b2_test.go deleted file mode 100644 index d0c91b425..000000000 --- a/cmd/gateway/b2/gateway-b2_test.go +++ /dev/null @@ -1,146 +0,0 @@ -/* - * MinIO Cloud Storage, (C) 2017 MinIO, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package b2 - -import ( - "fmt" - "testing" - - minio "github.com/minio/minio/cmd" -) - -// Test b2 object error. -func TestB2ObjectError(t *testing.T) { - testCases := []struct { - params []string - b2Err error - expectedErr error - }{ - { - []string{}, nil, nil, - }, - { - []string{}, fmt.Errorf("Not *Error"), fmt.Errorf("Not *Error"), - }, - { - []string{}, fmt.Errorf("Non B2 Error"), fmt.Errorf("Non B2 Error"), - }, - } - - for i, testCase := range testCases { - actualErr := b2ToObjectError(testCase.b2Err, testCase.params...) - if actualErr != nil { - if actualErr.Error() != testCase.expectedErr.Error() { - t.Errorf("Test %d: Expected %s, got %s", i+1, testCase.expectedErr, actualErr) - } - } - } -} - -// Test b2 msg code to object error. -func TestB2MsgCodeToObjectError(t *testing.T) { - testCases := []struct { - params []string - code int - msgCode string - msg string - expectedErr error - }{ - { - []string{"bucket"}, - 1, - "duplicate_bucket_name", - "", - minio.BucketAlreadyOwnedByYou{ - Bucket: "bucket", - }, - }, - { - []string{"bucket"}, - 1, - "bad_request", - "", - minio.BucketNotFound{ - Bucket: "bucket", - }, - }, - { - []string{"bucket", "object"}, - 1, - "bad_request", - "", - minio.ObjectNameInvalid{ - Bucket: "bucket", - Object: "object", - }, - }, - { - []string{"bucket"}, - 1, - "bad_bucket_id", - "", - minio.BucketNotFound{Bucket: "bucket"}, - }, - { - []string{"bucket", "object"}, - 1, - "file_not_present", - "", - minio.ObjectNotFound{ - Bucket: "bucket", - Object: "object", - }, - }, - { - []string{"bucket", "object"}, - 1, - "not_found", - "", - minio.ObjectNotFound{ - Bucket: "bucket", - Object: "object", - }, - }, - { - []string{"bucket"}, - 1, - "cannot_delete_non_empty_bucket", - "", - minio.BucketNotEmpty{ - Bucket: "bucket", - }, - }, - { - []string{"bucket", "object", "uploadID"}, - 1, - "", - "No active upload for", - minio.InvalidUploadID{ - UploadID: "uploadID", - }, - }, - } - - for i, testCase := range testCases { - actualErr := b2MsgCodeToObjectError(testCase.code, testCase.msgCode, testCase.msg, testCase.params...) - if actualErr != nil { - if actualErr.Error() != testCase.expectedErr.Error() { - t.Errorf("Test %d: Expected %s, got %s", i+1, testCase.expectedErr, actualErr) - } - } - } -} diff --git a/cmd/gateway/gateway.go b/cmd/gateway/gateway.go index b88a7575c..627ae1146 100644 --- a/cmd/gateway/gateway.go +++ b/cmd/gateway/gateway.go @@ -1,5 +1,5 @@ /* - * MinIO Cloud Storage, (C) 2017 MinIO, Inc. + * MinIO Cloud Storage, (C) 2017-2020 MinIO, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -17,15 +17,23 @@ package gateway import ( - // Import all gateways. - _ "github.com/minio/minio/cmd/gateway/azure" - _ "github.com/minio/minio/cmd/gateway/gcs" - _ "github.com/minio/minio/cmd/gateway/hdfs" + // Import all gateways please keep the order + + // NAS _ "github.com/minio/minio/cmd/gateway/nas" + + // Azure + _ "github.com/minio/minio/cmd/gateway/azure" + + // S3 _ "github.com/minio/minio/cmd/gateway/s3" - // B2 is specifically kept here to avoid re-ordering by goimports, - // please ask on github.com/minio/minio/issues before changing this. - _ "github.com/minio/minio/cmd/gateway/b2" - // Add your gateway here. + // HDFS + _ "github.com/minio/minio/cmd/gateway/hdfs" + + // GCS (use only if you must, GCS already supports S3 API) + _ "github.com/minio/minio/cmd/gateway/gcs" + // gateway functionality is frozen, no new gateways are being implemented + // or considered for upstream inclusion at this point in time. if needed + // please keep a fork of the project. ) diff --git a/docs/gateway/README.md b/docs/gateway/README.md index 9f4190f55..4a3a4fb9c 100644 --- a/docs/gateway/README.md +++ b/docs/gateway/README.md @@ -2,7 +2,7 @@ MinIO Gateway adds Amazon S3 compatibility to third party cloud storage providers. - [NAS](https://github.com/minio/minio/blob/master/docs/gateway/nas.md) - [Microsoft Azure Blob Storage](https://github.com/minio/minio/blob/master/docs/gateway/azure.md) +- [HDFS](https://github.com/minio/minio/blob/master/docs/gateway/hdfs.md) - [S3](https://github.com/minio/minio/blob/master/docs/gateway/s3.md) - [Google Cloud Storage](https://github.com/minio/minio/blob/master/docs/gateway/gcs.md) -- [Backblaze B2](https://github.com/minio/minio/blob/master/docs/gateway/b2.md) diff --git a/docs/gateway/b2.md b/docs/gateway/b2.md deleted file mode 100644 index 891acae99..000000000 --- a/docs/gateway/b2.md +++ /dev/null @@ -1,60 +0,0 @@ -# MinIO B2 Gateway [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) -MinIO Gateway adds Amazon S3 compatibility to Backblaze B2 Cloud Storage. - -## Run MinIO Gateway for Backblaze B2 Cloud Storage -Please follow this [guide](https://www.backblaze.com/b2/docs/quick_account.html) to create an account on backblaze.com to obtain your access credentials for B2 Cloud storage. - -### Using Docker -``` -docker run -p 9000:9000 --name b2-s3 \ - -e "MINIO_ACCESS_KEY=b2_account_id" \ - -e "MINIO_SECRET_KEY=b2_application_key" \ - minio/minio gateway b2 -``` - -### Using Binary -``` -export MINIO_ACCESS_KEY=b2_account_id -export MINIO_SECRET_KEY=b2_application_key -minio gateway b2 -``` - -## Test using MinIO Browser -MinIO Gateway comes with an embedded web based object browser. Point your web browser to http://127.0.0.1:9000 to ensure that your server has started successfully. - -![Screenshot](https://raw.githubusercontent.com/minio/minio/master/docs/screenshots/minio-browser-gateway.png) - -## Test using MinIO Client `mc` -`mc` provides a modern alternative to UNIX commands such as ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services. - -### Configure `mc` -``` -mc config host add myb2 http://gateway-ip:9000 b2_account_id b2_application_key -``` - -### List buckets on Backblaze B2 -``` -mc ls myb2 -[2017-02-22 01:50:43 PST] 0B ferenginar/ -[2017-02-26 21:43:51 PST] 0B my-bucket/ -[2017-02-26 22:10:11 PST] 0B test-bucket1/ -``` - -### Known limitations -Gateway inherits the following B2 limitations: - -- No support for CopyObject S3 API (yet). -- No support for CopyObjectPart S3 API (yet). -- Only read-only bucket policy supported at bucket level, all other variations will return API Notimplemented error. -- DeleteObject() might not delete the object right away on Backblaze B2, so you might see the object immediately after a Delete request. -- Multipart uploads don't return an MD5-hash ETag, not even with the `--compat` flag. - -Other limitations: - -- Bucket notification APIs are not supported. -- The PutObject S3 API only returns the md5sum of an uploaded file as it's ETag response when minio is started in `--compat` mode. Don't use this unless you are stuck with a client library where this check cannot be turned off (e.g. boto). - -## Explore Further -- [`mc` command-line interface](https://docs.min.io/docs/minio-client-quickstart-guide) -- [`aws` command-line interface](https://docs.min.io/docs/aws-cli-with-minio) -- [`minio-go` Go SDK](https://docs.min.io/docs/golang-client-quickstart-guide) diff --git a/go.mod b/go.mod index 65cbd8986..f25e9d024 100644 --- a/go.mod +++ b/go.mod @@ -55,7 +55,6 @@ require ( github.com/klauspost/pgzip v1.2.1 github.com/klauspost/readahead v1.3.1 github.com/klauspost/reedsolomon v1.9.3 - github.com/kurin/blazer v0.5.4-0.20200327014341-8f90a40f8af7 github.com/lib/pq v1.1.1 github.com/mattn/go-colorable v0.1.1 github.com/mattn/go-ieproxy v0.0.0-20190805055040-f9202b1cfdeb // indirect diff --git a/go.sum b/go.sum index b1b223f46..938fcca82 100644 --- a/go.sum +++ b/go.sum @@ -2,6 +2,7 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMT cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= cloud.google.com/go v0.39.0 h1:UgQP9na6OTfp4dsAiz/eFpFA1C6tPdH5wiRdi19tuMw= cloud.google.com/go v0.39.0/go.mod h1:rVLT6fkc8chs9sfPtFc1SBH6em7n+ZoXaG+87tDISts= +contrib.go.opencensus.io/exporter/ocagent v0.5.0 h1:TKXjQSRS0/cCDrP7KvkgU6SmILtF/yV2TOs/02K/WZQ= contrib.go.opencensus.io/exporter/ocagent v0.5.0/go.mod h1:ImxhfLRpxoYiSq891pBrLVhN+qmP8BTVvdH2YLs7Gl0= git.apache.org/thrift.git v0.13.0 h1:/3bz5WZ+sqYArk7MBBBbDufMxKKOA56/6JO6psDpUDY= git.apache.org/thrift.git v0.13.0/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg= @@ -9,11 +10,14 @@ github.com/Azure/azure-pipeline-go v0.2.1 h1:OLBdZJ3yvOn2MezlWvbrBMTEUQC72zAftRZ github.com/Azure/azure-pipeline-go v0.2.1/go.mod h1:UGSo8XybXnIGZ3epmeBw7Jdz+HiUVpqIlpz/HKHylF4= github.com/Azure/azure-storage-blob-go v0.8.0 h1:53qhf0Oxa0nOjgbDeeYPUeyiNmafAFEY95rZLK0Tj6o= github.com/Azure/azure-storage-blob-go v0.8.0/go.mod h1:lPI3aLPpuLTeUwh1sViKXFxwl2B6teiRqI0deQUvsw0= +github.com/Azure/go-autorest v11.7.1+incompatible h1:M2YZIajBBVekV86x0rr1443Lc1F/Ylxb9w+5EtSyX3Q= github.com/Azure/go-autorest v11.7.1+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24= +github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/DataDog/datadog-go v2.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3tL4fMGNddJ+vMq1mwgfaqoQ= github.com/Shopify/sarama v1.24.1 h1:svn9vfN3R1Hz21WR2Gj0VW9ehaDGkiOS+VqlIcZOkMI= github.com/Shopify/sarama v1.24.1/go.mod h1:fGP8eQ6PugKEI0iUETYYtnP6d1pH/bdDMTel1X5ajsU= +github.com/Shopify/toxiproxy v2.1.4+incompatible h1:TKdv8HiTLgE5wdJuEML90aBgNWsokNbMijUGhmcoBJc= github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI= github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d h1:G0m3OIz70MZUWq3EgK3CesDbo8upS2Vm9/P3FtgI+Jk= github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg= @@ -22,6 +26,7 @@ github.com/alecthomas/participle v0.2.1/go.mod h1:SW6HZGeZgSIpcUWX3fXpfZhuaWHnmo github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY= +github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878 h1:EFSB7Zo9Eg91v7MJPVsifUysc/wPdN+NOnVe6bWbdBM= github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878/go.mod h1:3AMJUQhVx52RsWOnlkpikZr01T/yAVN2gn0861vByNg= github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8= github.com/aws/aws-sdk-go v1.20.21 h1:22vHWL9rur+SRTYPHAXlxJMFIA9OSYsYDIAHFDhQ7Z0= @@ -34,6 +39,7 @@ github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLM github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs= github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps= +github.com/census-instrumentation/opencensus-proto v0.2.0 h1:LzQXZOgg4CQfE6bFvXGM30YZL1WW/M337pXml+GrcZ4= github.com/census-instrumentation/opencensus-proto v0.2.0/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY= github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= @@ -44,11 +50,15 @@ github.com/circonus-labs/circonusllhist v0.1.3/go.mod h1:kMXHVDlOchFAehlya5ePtbp github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/coredns/coredns v1.4.0 h1:RubBkYmkByUqZWWkjRHvNLnUHgkRVqAWgSMmRFvpE1A= github.com/coredns/coredns v1.4.0/go.mod h1:zASH/MVDgR6XZTbxvOnsZfffS+31vg6Ackf/wo1+AM0= +github.com/coreos/bbolt v1.3.3 h1:n6AiVyVRKQFNb6mJlwESEvvLoDyiTzXX7ORAUlkeBdY= github.com/coreos/bbolt v1.3.3/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk= github.com/coreos/etcd v3.3.12+incompatible h1:pAWNwdf7QiT1zfaWyqCtNZQWCLByQyA3JrSQyuYAqnQ= github.com/coreos/etcd v3.3.12+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= +github.com/coreos/go-semver v0.3.0 h1:wkHLiw0WNATZnSG7epLsujiMCgPAc9xhjJ4tgnAxmfM= github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e h1:Wf6HqHfScWJN9/ZjdUKyjop4mf3Qdd+1TvvltAvM3m8= github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= +github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f h1:lBNOc5arjvs8E5mO2tbpBpLoyyu8B6e44T7hJy6potg= github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= @@ -75,10 +85,13 @@ github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5Kwzbycv github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo= github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M= github.com/fortytw2/leaktest v1.2.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g= +github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw= github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g= github.com/frankban/quicktest v1.4.1/go.mod h1:36zfPVQyHxymz4cH7wlDmVwDrJuljRB60qkgn7rorfQ= +github.com/frankban/quicktest v1.7.2 h1:2QxQoC1TS09S7fhCPsrvqYdvP1H5M1P1ih5ABm3BTYk= github.com/frankban/quicktest v1.7.2/go.mod h1:jaStnuzAqU1AJdCO0l53JDCJrVDKcS03DbaAcR7Ks/o= github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= +github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32 h1:Mn26/9ZMNWSw9C9ERFA1PUxfmGpolnw2v0bKOREu5ew= github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32/go.mod h1:GIjDIg/heH5DOkXY3YJ/wNhfHsQHoXGjl8G8amsYQ1I= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-ldap/ldap v3.0.2+incompatible/go.mod h1:qfd9rJvER9Q0/D/Sqn1DfHRoBp40uXYvFoEVrNEPqRc= @@ -92,7 +105,9 @@ github.com/go-test/deep v1.0.2-0.20181118220953-042da051cf31/go.mod h1:wGDj63lr6 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE= github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= @@ -105,11 +120,14 @@ github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4= github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/gomodule/redigo v2.0.0+incompatible h1:K/R+8tc58AaqLkqG2Ol3Qk+DR/TlNuhuh457pBFPtt0= github.com/gomodule/redigo v2.0.0+incompatible/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4= +github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c h1:964Od4U6p2jUkFxvCydnIczKteheJEzHRToSGK3Bnlw= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg= github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no= github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= @@ -118,6 +136,7 @@ github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+ github.com/googleapis/gax-go/v2 v2.0.4 h1:hU4mGcQI4DaAYW+IbTun+2qEZVFxK0ySjQLTbS0VQKc= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= +github.com/gopherjs/gopherjs v0.0.0-20190328170749-bb2674552d8f h1:4Gslotqbs16iAg+1KR/XdabIfq8TlAWHdwS5QJFksLc= github.com/gopherjs/gopherjs v0.0.0-20190328170749-bb2674552d8f/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gorilla/handlers v1.4.0 h1:XulKRWSQK5uChr4pEgSE4Tc/OcmnU9GJuSwdog/tZsA= github.com/gorilla/handlers v1.4.0/go.mod h1:Qkdc/uu4tH4g6mTK6auzZ766c4CA0Ng8+o/OAirnOIQ= @@ -125,10 +144,14 @@ github.com/gorilla/mux v1.7.0 h1:tOSd0UKHQd6urX6ApfOn4XdBMY6Sh1MfxV3kmaazO+U= github.com/gorilla/mux v1.7.0/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs= github.com/gorilla/rpc v1.2.0 h1:WvvdC2lNeT1SP32zrIce5l0ECBfbAlmrmSBsuc57wfk= github.com/gorilla/rpc v1.2.0/go.mod h1:V4h9r+4sF5HnzqbwIez0fKSpANP0zlYd3qR7p36jkTQ= +github.com/gorilla/websocket v1.4.1 h1:q7AeDBpnBk8AogcD4DSag/Ukw/KV+YhzLj2bP5HvKCM= github.com/gorilla/websocket v1.4.1/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4 h1:z53tR0945TRRQO/fLEVPI6SMv7ZflF0TEaTAoU7tOzg= github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= +github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 h1:Ovs26xHkKqVztRpIrF/92BcuyuQ/YW4NSIpoGtfXNho= github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk= github.com/grpc-ecosystem/grpc-gateway v1.8.5/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY= +github.com/grpc-ecosystem/grpc-gateway v1.9.0 h1:bM6ZAFZmc/wPFaRDi0d5L7hGEZEx/2u+Tmr2evNHDiI= github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY= github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA= github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= @@ -138,8 +161,11 @@ github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtng github.com/hashicorp/go-hclog v0.0.0-20180709165350-ff2cf002a8dd/go.mod h1:9bjs9uLqI8l75knNv3lV1kA55veR+WUPSiKIWcQHudI= github.com/hashicorp/go-hclog v0.8.0/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ= github.com/hashicorp/go-hclog v0.9.1/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ= +github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI= github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ= +github.com/hashicorp/go-immutable-radix v1.0.0 h1:AKDB1HM5PWEA7i4nhcpwOrO2byshxBjXVn/J/3+z5/0= github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= +github.com/hashicorp/go-msgpack v0.5.5 h1:i9R9JSrqIz0QVLz3sz+i3YJdT7TTSLcfLLzJi9aZTuI= github.com/hashicorp/go-msgpack v0.5.5/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM= github.com/hashicorp/go-multierror v1.0.0 h1:iVjPR7a6H0tWELX5NxNe7bYopibicUzc7uPribsnS6o= github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= @@ -160,6 +186,7 @@ github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+ github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= +github.com/hashicorp/raft v1.1.1-0.20190703171940-f639636d18e0 h1:msEDtkZC3STZq6Pthlju+jKruuNHXCZAWhghDK47HcM= github.com/hashicorp/raft v1.1.1-0.20190703171940-f639636d18e0/go.mod h1:vPAJM8Asw6u8LxC3eJCUZmRP/E4QmUGE1R7g7k8sG/8= github.com/hashicorp/raft-boltdb v0.0.0-20171010151810-6e5ba93211ea/go.mod h1:pNv7Wc3ycL6F5oOWn+tPGo2gWD4a5X+yp/ntwdKLjRk= github.com/hashicorp/vault/api v1.0.4 h1:j08Or/wryXT4AcHj1oCbMd7IijXcKzYUGw59LGu9onU= @@ -174,10 +201,12 @@ github.com/jcmturner/gofork v0.0.0-20190328161633-dc7c13fece03 h1:FUwcHNlEqkqLjL github.com/jcmturner/gofork v0.0.0-20190328161633-dc7c13fece03/go.mod h1:MK8+TM0La+2rjBD4jE12Kj1pCCxK7d2LK/UM3ncEo0o= github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM= github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k= +github.com/jonboulle/clockwork v0.1.0 h1:VKV+ZcuP6l3yW9doeqz6ziZGgcynBVQO+obU0+0hcPo= github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo= github.com/json-iterator/go v1.1.9 h1:9yzud/Ht36ygwatGx56VwCZtlI/2AD15T1X2sjSuGns= github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= +github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo= github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= @@ -195,13 +224,14 @@ github.com/klauspost/readahead v1.3.1 h1:QqXNYvm+VvqYcbrRT4LojUciM0XrznFRIDrbHiJ github.com/klauspost/readahead v1.3.1/go.mod h1:AH9juHzNH7xqdqFHrMRSHeH2Ps+vFf+kblDqzPFiLJg= github.com/klauspost/reedsolomon v1.9.3 h1:N/VzgeMfHmLc+KHMD1UL/tNkfXAt8FnUqlgXGIduwAY= github.com/klauspost/reedsolomon v1.9.3/go.mod h1:CwCi+NUr9pqSVktrkN+Ondf06rkhYZ/pcNv7fu+8Un4= +github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= +github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= -github.com/kurin/blazer v0.5.4-0.20200327014341-8f90a40f8af7 h1:smZXPopqRVVywwzou4WYWvUbJvSAzIDFizfWElpmAqY= -github.com/kurin/blazer v0.5.4-0.20200327014341-8f90a40f8af7/go.mod h1:4FCXMUWo9DllR2Do4TtBd377ezyAJ51vB5uTBjt0pGU= github.com/lib/pq v1.1.1 h1:sJZmqHoEaY7f+NPP8pgLB/WxulyR3fewgCM2qaSlBb4= github.com/lib/pq v1.1.1/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo= github.com/mailru/easyjson v0.0.0-20180730094502-03f2033d19d5 h1:0x4qcEHDpruK6ML/m/YSlFUUu0UpRD3I2PHsNCuGnyA= @@ -233,8 +263,6 @@ github.com/minio/highwayhash v1.0.0/go.mod h1:xQboMTeM9nY9v/LlAOxFctujiv5+Aq2hR5 github.com/minio/lsync v1.0.1 h1:AVvILxA976xc27hstd1oR+X9PQG0sPSom1MNb1ImfUs= github.com/minio/lsync v1.0.1/go.mod h1:tCFzfo0dlvdGl70IT4IAK/5Wtgb0/BrTmo/jE8pArKA= github.com/minio/minio-go/v6 v6.0.53/go.mod h1:DIvC/IApeHX8q1BAMVCXSXwpmrmM+I+iBvhvztQorfI= -github.com/minio/minio-go/v6 v6.0.55-0.20200424204115-7506d2996b22 h1:nZEve4vdUhwHBoV18zRvPDgjL6NYyDJE5QJvz3l9bRs= -github.com/minio/minio-go/v6 v6.0.55-0.20200424204115-7506d2996b22/go.mod h1:KQMM+/44DSlSGSQWSfRrAZ12FVMmpWNuX37i2AX0jfI= github.com/minio/minio-go/v6 v6.0.55-0.20200425081427-89eebdef2af0 h1:PdHKpM9h2vqCDr1AjJdK8e/6tRdOSjUNzIqeNmxu7ak= github.com/minio/minio-go/v6 v6.0.55-0.20200425081427-89eebdef2af0/go.mod h1:KQMM+/44DSlSGSQWSfRrAZ12FVMmpWNuX37i2AX0jfI= github.com/minio/parquet-go v0.0.0-20200414234858-838cfa8aae61 h1:pUSI/WKPdd77gcuoJkSzhJ4wdS8OMDOsOu99MtpXEQA= @@ -255,6 +283,7 @@ github.com/mitchellh/go-wordwrap v1.0.0/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUb github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw= +github.com/mmcloughlin/avo v0.0.0-20200303042253-6df701fe672f h1:4a8aNS/kCsM3TZnLO0jUzMAch5+XYh5Poe74nqatJcw= github.com/mmcloughlin/avo v0.0.0-20200303042253-6df701fe672f/go.mod h1:L0u9qfRMLNBO97u6pPukRp6ncoQz0Q25W69fvtht3vA= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= @@ -265,8 +294,11 @@ github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3Rllmb github.com/montanaflynn/stats v0.5.0 h1:2EkzeTSqBB4V4bJwWrt5gIIrZmpJBcoIRGS2kWLgzmk= github.com/montanaflynn/stats v0.5.0/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc= github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= +github.com/nats-io/gnatsd v1.4.1 h1:RconcfDeWpKCD6QIIwiVFcvForlXpWeJP7i5/lDLy44= github.com/nats-io/gnatsd v1.4.1/go.mod h1:nqco77VO78hLCJpIcVfygDP2rPGfsEHkGTUk94uh5DQ= +github.com/nats-io/go-nats v1.7.2 h1:cJujlwCYR8iMz5ofZSD/p2WLW8FabhkQ2lIEVbSvNSA= github.com/nats-io/go-nats v1.7.2/go.mod h1:+t7RHT5ApZebkrQdnn6AhQJmhJJiKAvJUio1PiiCtj0= +github.com/nats-io/go-nats-streaming v0.4.4 h1:1I3lkZDRdQYXb+holjdqZ2J6xyekrD06o9Fd8rWlgP4= github.com/nats-io/go-nats-streaming v0.4.4/go.mod h1:gfq4R3c9sKAINOpelo0gn/b9QDMBZnmrttcsNF+lqyo= github.com/nats-io/jwt v0.3.0/go.mod h1:fRYCDE99xlTsqUzISS1Bi75UBJ6ljOJQOAAu5VglpSg= github.com/nats-io/jwt v0.3.2 h1:+RB5hMpXUUA2dfxuhBTEkMOrYmM+gKIZYS1KjSostMI= @@ -275,6 +307,7 @@ github.com/nats-io/nats-server v1.4.1 h1:Ul1oSOGNV/L8kjr4v6l2f9Yet6WY+LevH1/7cRZ github.com/nats-io/nats-server v1.4.1/go.mod h1:c8f/fHd2B6Hgms3LtCaI7y6pC4WD1f4SUxcCud5vhBc= github.com/nats-io/nats-server/v2 v2.1.2 h1:i2Ly0B+1+rzNZHHWtD4ZwKi+OU5l+uQo1iDHZ2PmiIc= github.com/nats-io/nats-server/v2 v2.1.2/go.mod h1:Afk+wRZqkMQs/p45uXdrVLuab3gwv3Z8C4HTBu8GD/k= +github.com/nats-io/nats-streaming-server v0.14.2 h1:WjQMDqVOwsI0Nb0E+XmEs1LY17CwHRbTCSTWKhw9fXs= github.com/nats-io/nats-streaming-server v0.14.2/go.mod h1:RyqtDJZvMZO66YmyjIYdIvS69zu/wDAkyNWa8PIUa5c= github.com/nats-io/nats.go v1.9.1 h1:ik3HbLhZ0YABLto7iX80pZLPw/6dx3T+++MZJwLnMrQ= github.com/nats-io/nats.go v1.9.1/go.mod h1:ZjDU1L/7fJ09jvUSRVBR2e7+RnLiiIQyqyzEE/Zbp4w= @@ -290,6 +323,7 @@ github.com/ncw/directio v1.0.5/go.mod h1:rX/pKEYkOXBGOggmcyJeJGloCkleSvphPx2eV3t github.com/nsqio/go-nsq v1.0.7 h1:O0pIZJYTf+x7cZBA0UMY8WxFG79lYTURmWzAAh48ljY= github.com/nsqio/go-nsq v1.0.7/go.mod h1:XP5zaUs3pqf+Q71EqUJs3HYfBIqfK6G83WQMdNN+Ito= github.com/oklog/run v1.0.0/go.mod h1:dlhp/R75TPv97u0XWUtDeV/lRKWPKSdTuV0TZvrmrQA= +github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY= github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= github.com/pborman/getopt v0.0.0-20180729010549-6fdd0a2c7117/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o= github.com/philhofer/fwd v1.0.0 h1:UbZqGr5Y38ApvM/V/jEljVxwocdweyH+vmYvRPBnbqQ= @@ -301,6 +335,7 @@ github.com/pierrec/lz4 v2.4.0+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI= github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= @@ -338,9 +373,12 @@ github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPx github.com/sirupsen/logrus v1.5.0 h1:1N5EYkVAPEywqZRJd7cwnRtCb6xJx7NH3T3WUTF980Q= github.com/sirupsen/logrus v1.5.0/go.mod h1:+F7Ogzej0PZc/94MaYx/nvG9jOFMD2osvC3s+Squfpo= github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= +github.com/smartystreets/assertions v0.0.0-20190401211740-f487f9de1cd3 h1:hBSHahWMEgzwRyS6dRpxY0XyjZsHyQ61s084wo5PJe0= github.com/smartystreets/assertions v0.0.0-20190401211740-f487f9de1cd3/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= github.com/smartystreets/go-aws-auth v0.0.0-20180515143844-0c1422d1fdb9/go.mod h1:SnhjPscd9TpLiy1LpzGSKh3bXCfxxXuqd9xmQJy3slM= +github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a h1:pa8hGb/2YqsZKovtsgrwcDH1RZhVbTKCjLp47XpqCDs= github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= +github.com/soheilhy/cmux v0.1.4 h1:0HKaf1o97UwFjHH9o5XsHUOF+tqmdA7KEzXLpiyaw0E= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= @@ -350,6 +388,7 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+ github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/tidwall/gjson v1.3.5 h1:2oW9FBNu8qt9jy5URgrzsVx/T/KSn3qn/smJQ0crlDQ= github.com/tidwall/gjson v1.3.5/go.mod h1:P256ACg0Mn+j1RXIDXoss50DeIABTYK1PULOJHhxOls= @@ -361,9 +400,12 @@ github.com/tidwall/sjson v1.0.4 h1:UcdIRXff12Lpnu3OLtZvnc03g4vH2suXDXhBwBqmzYg= github.com/tidwall/sjson v1.0.4/go.mod h1:bURseu1nuBkFpIES5cz6zBtjmYeOQmEESshn7VpF15Y= github.com/tinylib/msgp v1.1.1 h1:TnCZ3FIuKeaIy+F45+Cnp+caqdXGy4z74HvwXN+570Y= github.com/tinylib/msgp v1.1.1/go.mod h1:+d+yLhGm8mzTaHzB+wgMYrodPfmZrzkirds8fDWklFE= +github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5 h1:LnC5Kc/wtumK+WB441p7ynQJzVuNRJiqddSIE3IlSEQ= github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926/go.mod h1:9ESjWnEqriFuLhtthL60Sar/7RFoluCcXsuvEwTV5KM= +github.com/ugorji/go v1.1.5-pre h1:jyJKFOSEbdOc2HODrf2qcCkYOdq7zzXqA9bhW5oV4fM= github.com/ugorji/go v1.1.5-pre/go.mod h1:FwP/aQVg39TXzItUBMwnWp9T9gPQnXw4Poh4/oBQZ/0= +github.com/ugorji/go/codec v1.1.5-pre h1:5YV9PsFAN+ndcCtTM7s60no7nY7eTG3LPtxhSwuxzCs= github.com/ugorji/go/codec v1.1.5-pre/go.mod h1:tULtS6Gy1AE1yCENaw4Vb//HLH5njI2tfCQDUqRd8fI= github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a h1:0R4NLDRDZX6JcmhJgXi5E4b8Wg84ihbmUKp/GvSPEzc= github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV9WzVtRmSR+PDvWpU/qWl4Wa5LApYYX4ZtKbio= @@ -375,14 +417,20 @@ github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c h1:u40Z8hqBAAQyv+vATcGgV github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c/go.mod h1:lB8K/P019DLNhemzwFU4jHLhdvlE6uDZjXFejJXr49I= github.com/xdg/stringprep v1.0.0 h1:d9X0esnoa3dFsV0FG35rAT0RIhYFlPq7MiP+DW89La0= github.com/xdg/stringprep v1.0.0/go.mod h1:Jhud4/sHMO4oL310DaZAKk9ZaJ08SJfe+sJh0HrGL1Y= +github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 h1:eY9dn8+vbi4tKz5Qo6v2eYzo7kUS51QINcR5jNpbZS8= github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +go.etcd.io/bbolt v1.3.3 h1:MUGmc65QhB3pIlaQ5bB4LwqSj6GIonVJXpZiaKNyaKk= go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= +go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk= go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= +go.uber.org/multierr v1.5.0 h1:KCa4XfM8CWFCpxXRGok+Q0SS/0XBhMDbHHGABQLvD2A= go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= +go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4= go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= +go.uber.org/zap v1.15.0 h1:ZZCA22JRF2gQE5FoNmhmrf7jeJJ2uhqDUNRYKm8dvmM= go.uber.org/zap v1.15.0/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc= golang.org/x/arch v0.0.0-20190909030613-46d78d1859ac/go.mod h1:flIaEI6LNU6xOCD5PaJvn9wGP0agmIOqjrtsKGRguv4= golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= @@ -401,8 +449,10 @@ golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTk golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs= golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= +golang.org/x/mod v0.2.0 h1:KU7oHjnv3XNWfa5COkzUifxZmxp1TyI7ImMXqFxLwvQ= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -428,6 +478,7 @@ golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e h1:vcxGaoTs7kV8m5Np9uUNQin4BrLOthgV7252N8V+FwY= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -467,15 +518,18 @@ golang.org/x/tools v0.0.0-20190914235951-31e00f45c22e/go.mod h1:b+2E5dAYhXwXZwtn golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20200428211428-0c9eba77bc32 h1:Xvf3ZQTm5bjXPxhI7g+dwqsCqadK1rcNtwtszuatetk= golang.org/x/tools v0.0.0-20200428211428-0c9eba77bc32/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= google.golang.org/api v0.5.0 h1:lj9SyhMzyoa38fgFF0oO2T6pjs5IzkLPKfVtxpyCRMM= google.golang.org/api v0.5.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/appengine v1.6.0 h1:Tfd7cKwKbFRsI8RMAD3oqqw7JPFRrvFlOsfbgVkjOOw= google.golang.org/appengine v1.6.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= @@ -492,7 +546,9 @@ gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLks gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d h1:TxyelI5cVkbREznMhfzycHdkp5cLA7DpE+GKjSslYhM= gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d/go.mod h1:cuepJuh7vyXfUyUwEgHQXw849cJrilpS5NeIjOWESAw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/cheggaaa/pb.v1 v1.0.28 h1:n1tBJnnK2r7g9OW2btFH91V92STTUevLXYFb8gy9EMk= gopkg.in/cheggaaa/pb.v1 v1.0.28/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/ini.v1 v1.42.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= @@ -524,5 +580,6 @@ gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM= honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=