Commit Graph

7922 Commits

Author SHA1 Message Date
Harshavardhana 85e939636f Fix JSON parser handling for certain objects (#7162)
This PR also adds some comments and simplifies
the code. Primary handling is done to ensure
that we make sure to honor cached buffer.

Added unit tests as well

Fixes #7141
2019-02-07 08:04:42 +05:30
poornas d203e7e1cc azure gateway: return MD5Sum as ETag for S3 API compatibility (#6884)
Fixes #6872.

This PR refactors multipart upload implementation to use a per
part metadata file which is cleaned up at the end of the upload
2019-02-06 16:58:43 -08:00
Aditya Manthramurthy 4aa9ee153b Fix S3 Select request XML parsing (#7202) 2019-02-06 13:25:52 -08:00
Minio Trusted 5fb813a5cc Update yaml files to latest version RELEASE.2019-02-06T21-16-36Z 2019-02-06 21:24:08 +00:00
Harshavardhana 817269475f Make sure to drain body upon an error (#7197)
Also cleanup redundant code and use it at a common place
2019-02-06 12:07:03 -08:00
Krishna Srinivas 2d168b532b Allow format.json healing on dev/test setup (single node XL, all root disks) (#7170) 2019-02-06 11:44:19 -08:00
Aditya Manthramurthy fd4e15c116 Flush the records staging buffer periodically (#7193)
- Staging buffer is flushed every 500ms. In cases where the result
  records are slowly generated (e.g. when a where condition
  matches very few records), this change causes the server to send
  results even though the staging buffer is not full.

- Refactor messageWriter code to use simpler channel based
  co-ordination instead of atomic variables.
2019-02-06 16:03:05 +05:30
Krishna Srinivas 3dfbe0f68c Send white spaces to client till completeMultipart() process completes (#7198) 2019-02-05 20:58:09 -08:00
Harshavardhana 30135eed86 Redo how to handle stale dangling files (#7171)
foo.CORRUPTED should never be created because when
multiple sets are involved we would hash the file
to wrong a location, this PR removes the code.

But allows DeleteBucket() to work properly to delete
dangling buckets/objects. Also adds another option
to Healing where a user needs to specify `--remove`
such that all dangling objects will be deleted with
user confirmation.
2019-02-05 17:58:48 -08:00
Harshavardhana e4081aee62 Added support for reading body in STS API (#7188)
STS API supports both URL query params and reading
from a body.
2019-02-05 15:47:11 -08:00
kannappanr df418a2783
Create Cors handler with permissive configuration (#7186)
Create new Cors handler allowing all origins with all standard
methods with any header and credentials.

Fixes #7181
2019-02-05 14:06:52 -08:00
kannappanr 9a65f6dc97 Remove duplicate code in object-handlers.go (#7176)
removed duplicate code in CompleteMultipartUploadHandler
and CopyObjectPartHandler.
2019-02-05 13:36:38 -08:00
Aditya Manthramurthy f04f8bbc78 Add support for Timestamp data type in SQL Select (#7185)
This change adds support for casting strings to Timestamp via CAST:
`CAST('2010T' AS TIMESTAMP)`

It also implements the following date-time functions:
  - UTCNOW()
  - DATE_ADD()
  - DATE_DIFF()
  - EXTRACT()

For values passed to these functions, date-types are automatically
inferred.
2019-02-04 20:54:45 -08:00
Harshavardhana ea6d61ab1f Use loadCachedConfigs appropriately to load ENVs (#7187) 2019-02-04 10:31:11 +05:30
Krishna Srinivas 6f08edfb36 Use O_EXCL when creating file as we never overwrite an existing file (#7189) 2019-02-01 19:01:06 -08:00
Sidhartha Mani e9fdea05c6 Enable CI control from repository: Add Dockerfile.simpleci (#7122) 2019-02-01 12:04:28 -08:00
Harshavardhana e005910051 Add more information in our select docs (#7177) 2019-02-01 11:34:56 -08:00
Anis Elleuch de2c106386 xl: ListObjectParts uses the latest valid xl meta (#7184)
ListObjectParts is using xl.readXLMetaParts which picks the first
xl meta found in any disk, which is an inconsistent information.

E.g.: In a middle of a multipart upload, one node can go offline
and get back later with an outdated multipart information.
2019-02-01 08:58:41 -08:00
Harshavardhana 32a6dd1dd6 Remove sporadic tests which fail on windows (#7178) 2019-01-31 16:48:47 -08:00
Harshavardhana 432aec73d9 Return proper errors for invalid bodies (#7179) 2019-01-31 07:19:09 -08:00
Anis Elleuch 36dae04671 CopyObjectPart: remove duplicated etag decryption (#7174) 2019-01-30 19:33:31 -08:00
Minio Trusted 9dc9f03c02 Update yaml files to latest version RELEASE.2019-01-31T00-31-19Z 2019-01-31 00:37:43 +00:00
Krishna Srinivas b18c0478e7 Only heal on disks where we are sure that healing is needed (#7148) 2019-01-30 10:53:57 -08:00
Anis Elleuch 2d9860e875 heal: Fix healing empty directories (#7154)
This commit fixes the computation of Before/After healing state
for empty directories.

Issues before the commit:
- Before state doesn't reflect the real status (no StatVol() called)
- For any MakeVol() error, healObjectDir is exited directly, which is
  wrong.
2019-01-30 10:51:56 -08:00
kannappanr d3553f8dfc
Bucket Heal: Do not add empty endpoint entry (#7172)
Currently during a heal of a bucket, if one disk is offline an empty endpoint entry is added.
Then another entry with the missing endpoint is also added.

This results in more entries than disks being added.

Code that adds empty endpoint has been removed.
2019-01-30 10:40:43 -08:00
Harshavardhana e1ae90c12b Make sure to pass the right username for correct ConditionValues (#7169)
Without passing proper username value would result in AccessDenied
errors when policies with `{aws:username}` substitutions are used.

Fixes #7165
2019-01-30 14:21:09 +05:30
Sidhartha Mani 34e7259f95 Add Historic CPU and memory stats (#7136)
Collect historic cpu and mem stats.  Also, use actual values 
instead of formatted strings while returning to the client. The string 
formatting prevents values from being processed by the server or 
by the client without parsing it. 

This change will allow the values to be processed (eg. 
compute rolling-average over the lifetime of the minio server)
and offloads the formatting to the client.
2019-01-30 12:47:32 +05:30
poornas d0015b4d66 update kms docs example to set a longer period for token renewal (#7149) 2019-01-29 08:04:07 -08:00
poornas 3467460456 Fix vault client to autorenew or reauthenticate (#7161)
Switch to Vault API's Renewer for token renewal.If
token can no longer be renewed, reauthenticate to
get a fresh token.
2019-01-29 16:57:23 +05:30
Harshavardhana 64b5701971 Support AWS envs creds for non-aws endpoints in S3 gateway (#7156)
We made a change previously in #7111 which moved support
for AWS envs only for AWS S3 endpoint. Some users requested
that this be added back to Non-AWS endpoints as well as
they require separate credentials for backend authentication
from security point of view.
2019-01-29 16:05:20 +05:30
Praveen raj Mani fad59da29d `clientID` removed in the MQTT config (#7157)
More than one client can't use the same clientID for MQTT connection. 
This causes problem in distributed deployments where config is shared 
across nodes, as each Minio instance tries to connect to MQTT using the
same clientID.

This commit removes the clientID field in config, and allows
MQTT client to create random clientID for each node.
2019-01-29 15:00:15 +05:30
Aditya Manthramurthy 91c839ad28 Use a buffer to collect SQL Select result rows (#7158)
Batching records into a single SQL Select message in the response
leads to significant speed up as the message header overhead is made
negligible.

This change leads to a speed up of 3-5x for queries that select many
small records.
2019-01-28 20:00:18 -08:00
Aditya Manthramurthy 2786055df4 Add new SQL parser to support S3 Select syntax (#7102)
- New parser written from scratch, allows easier and complete parsing
  of the full S3 Select SQL syntax. Parser definition is directly
  provided by the AST defined for the SQL grammar.

- Bring support to parse and interpret SQL involving JSON path
  expressions; evaluation of JSON path expressions will be
  subsequently added.

- Bring automatic type inference and conversion for untyped
  values (e.g. CSV data).
2019-01-28 17:59:48 -08:00
Harshavardhana 0a28c28a8c Avoid code which looks at local files when etcd is configured (#7144)
This situation happens only in gateway nas which supports
etcd based `config.json` to support all FS mode features.

The issue was we would try to migrate something which doesn't
exist when etcd is configured which leads to inconsistent
server configs in memory.

This PR fixes this situation by properly loading config after
initialization, avoiding backend disk config migration to be
done only if etcd is not configured.
2019-01-28 13:31:35 -08:00
Harshavardhana 526546d588 Remove '.minio.sys/tmp' files in background (#7124)
If it does happen that we have a lot files in '.minio.sys/tmp',
minio startup might block deleting this folder. Rename and
delete in background instead to allow Minio to start serving
requests.
2019-01-25 13:33:28 -08:00
Aditya Manthramurthy 2053b3414f Reduce heal parallelism (#7155)
To avoid a large number of concurrent connections between minio
servers and to reduce CPU pressure, it is better to limit the number
of objects healed in parallel to number_of_CPUs.
2019-01-25 13:11:17 -08:00
kannappanr ce870466ff
Top Locks command implementation (#7052)
API to list locks used in distributed XL mode
2019-01-24 07:22:14 -08:00
Harshavardhana 964e354d06 Fix liveness check for NAS gateway (#7142)
Current master throws '503' unavailable for liveness check
```
~ curl -v http://localhost:9000/minio/health/live
> GET /minio/health/live HTTP/1.1
...
...
< HTTP/1.1 503 Service Unavailable
```

With this fix liveness check returns error appropriately
```
~ curl -v http://localhost:9000/minio/health/live
> GET /minio/health/live HTTP/1.1
...
...
< HTTP/1.1 200 OK
```
2019-01-24 19:14:05 +05:30
kannappanr 8ee8ad777c
logger: do not interpret encoded url as format string (#7110)
Error logger currently interprets encoded url in the error as a format string.
2019-01-24 00:30:00 -08:00
Krishna Srinivas 82af0be1aa Healing process should not heal root disk (#7089) 2019-01-23 15:29:29 -08:00
Minio Trusted e8c18bc145 Update yaml files to latest version RELEASE.2019-01-23T23-18-58Z 2019-01-23 23:25:02 +00:00
Harshavardhana bd25f31100 Use IAM creds only if endpoint is S3 (#7111)
Requirements like being able to run minio gateway in ec2
pointing to a Minio deployment wouldn't work properly
because IAM creds take precendence on ec2.

Add checks such that we only enable AWS specific features
if our backend URL points to actual AWS S3 not S3 compatible
endpoints.
2019-01-23 11:12:33 -08:00
Harshavardhana ee7dcc2903 Handle errs returned with etcd properly for config init and migration (#7134)
Returning unexpected errors can cause problems for config handling,
which is what led gateway deployments with etcd to misbehave and
had stopped working properly
2019-01-23 11:10:59 -08:00
Harshavardhana 55ef51a99d Vendorize all recent changes to minio-go (#7135)
- Default support for S3 dualstack endpoints (IPv6 support)
- Support granular policy conditionals in List operations
- Support proxy cookies for stickiness
2019-01-23 19:22:09 +05:30
Anis Elleuch dc2348daa5 heal: Preserve deployment ID from reference format.json (#7126)
Deployment ID is not copied into new formats after healing format. Although,
this is not critical since a new deployment ID will be generated and set in the
next cluster restart, it is still much better if we don't change the deployment
id of a cluster for a better tracking.
2019-01-22 18:32:06 -08:00
Aditya Manthramurthy 042d7f25e4 Fix regexp matcher of browser assets and paths (#7083)
Fix regexp matcher for special assets for the browser to clash with
less of the object namespace.

Assets should now be loaded with the /minio/ prefix. Previously,
favicon.ico (and others) could be loaded at any path matching
/minio/*/favicon.ico. This clashes with a large part of the object
namespace. With this change, /minio/favicon.ico will serve the favicon
but not /minio/mybucket/favicon.ico

Fixes #7077
2019-01-22 10:58:28 -08:00
Andreas Auernhammer 8c1b649b2d load system CAs before trying to load custom CAs (#7133)
This changes causes `getRootCAs` to always load system-wide CAs.
Any additional custom CAs (at `certs/CA/`) are added to the certificate pool
of system CAs.

The previous behavior was incorrect since all no system-wide CAs were
loaded if either there were CAs under `certs/CA` or the `certs/CA`
directory didn't exist at all.
2019-01-22 09:18:06 -08:00
Kumar Sukhani f03ccec912 Fix slack Link (#7131) 2019-01-22 19:53:50 +05:30
Nitish Tiwari 0bb65f84bb
Add example for IPv6 for address flag (#7127) 2019-01-22 15:55:27 +05:30
Harshavardhana 8e0910ab3e Fix build issues on BSDs in pkg/cpu (#7116)
Also add a cross compile script to test always cross
compilation for some well known platforms and architectures
, we support out of box compilation of these platforms even
if we don't make an official release build.

This script is to avoid regressions in this area when we
add platform dependent code.
2019-01-22 09:27:23 +05:30