Commit Graph

1461 Commits

Author SHA1 Message Date
Praveen raj Mani
322385f1b6
fix: only show active/available ARNs in server startup banner (#9392) 2020-04-21 09:38:32 -07:00
Praveen raj Mani
c79358c67e
notification queue limit has no maxLimit (#9380)
New value defaults to 100K events by default,
but users can tune this value upto any value
they seem necessary.

* increase the limit to maxint64 while validating
2020-04-18 01:20:56 -07:00
Harshavardhana
75107d7698
fix: remove any duplicate statements in policy input (#9385)
Add support for removing duplicate statements automatically
2020-04-17 21:26:42 -07:00
Harshavardhana
d92db198d1
Add target parsing code for config (#9375)
This code is helper for mcs project
2020-04-16 17:43:14 -07:00
Harshavardhana
c82fa2c829
fix: load LDAP users appropriately (#9360)
This PR also fixes issues when

deletePolicy, deleteUser is idempotent so can lead to
issues when client can prematurely timeout, so a retry
call error response should be ignored when call returns
http.StatusNotFound

Fixes #9347
2020-04-16 16:22:34 -07:00
Harshavardhana
69fb68ef0b
fix simplify code to start using context (#9350) 2020-04-16 10:56:18 -07:00
Harshavardhana
bde0f444db
fix support OBDAdminAction is valid action (#9354) 2020-04-15 12:16:40 -07:00
Klaus Post
6a8298b137
Reduce Mutex test runs (#9345)
Some tests take a long time on CI:

* `--- PASS: TestRWMutex (226.49s)`
* ` --- PASS: TestRWMutex (7.13s)`

Reduce the number of runs.

Before/after locally:

```
--- PASS: TestRWMutex (20.95s)
--- PASS: TestRWMutex (7.13s)

--- PASS: TestMutex (3.01s)
--- PASS: TestMutex (1.65s)
```
2020-04-14 18:39:03 -07:00
Harshavardhana
5c11a46412 update minio-go/parquet-go to latest 2020-04-14 16:53:29 -07:00
Sidhartha Mani
ec11e99667
implement configurable timeout for OBD tests (#9324) 2020-04-14 11:48:32 -07:00
Harshavardhana
37d066b563
fix: deprecate requirement of session token for service accounts (#9320)
This PR fixes couple of behaviors with service accounts

- not need to have session token for service accounts
- service accounts can be generated by any user for themselves
  implicitly, with a valid signature.
- policy input for AddNewServiceAccount API is not fully typed
  allowing for validation before it is sent to the server.
- also bring in additional context for admin API errors if any
  when replying back to client.
- deprecate GetServiceAccount API as we do not need to reply
  back session tokens
2020-04-14 11:28:56 -07:00
Praveen raj Mani
bfec5fe200
fix: fetchLambdaInfo should return consistent results (#9332)
- Introduced a function `FetchRegisteredTargets` which will return
  a complete set of registered targets irrespective to their states,
  if the `returnOnTargetError` flag is set to `False`
- Refactor NewTarget functions to return non-nil targets
- Refactor GetARNList() to return a complete list of configured targets
2020-04-14 11:19:25 -07:00
Harshavardhana
9054ce73b2
fix: deprecate skyring/uuid and use maintained google/uuid (#9340) 2020-04-14 02:40:05 -07:00
kannappanr
1fa65c7f2f
fix: object lock behavior when default lock config is enabled (#9305) 2020-04-13 14:03:23 -07:00
Harshavardhana
7e12eab3ad
fix: cleanup madmin docs (#9330) 2020-04-13 10:30:41 +05:30
Harshavardhana
4314ee1670
fix: remove unusued PerfInfoHandler code (#9328)
- Removes PerfInfo admin API as its not OBDInfo
- Keep the drive path without the metaBucket in OBD
  global latency map.
- Remove all the unused code related to PerfInfo API
- Do not redefined global mib,gib constants use
  humanize.MiByte and humanize.GiByte instead always
2020-04-12 19:37:09 -07:00
Harshavardhana
bf9d51cf14
fix: add missing copyright headers in some files (#9321) 2020-04-12 13:55:22 -07:00
Harshavardhana
3184205519
fix: config to support keys with special values (#9304)
This PR adds context-based `k=v` splits based
on the sub-system which was obtained, if the
keys are not provided an error will be thrown
during parsing, if keys are provided with wrong
values an error will be thrown. Keys can now
have values which are of a much more complex
form such as `k="v=v"` or `k=" v = v"`
and other variations.

additionally, deprecate unnecessary postgres/mysql
configuration styles, support only

- connection_string for Postgres
- dsn_string for MySQL

All other parameters are removed.
2020-04-09 21:45:17 -07:00
Praveen raj Mani
2054ca5c9a
fix: honor token based authentication in NATS streaming (#9296)
fixes #9148
2020-04-08 12:45:24 -07:00
Sidhartha Mani
44decbeae0
increase drive OBD blocksize to 4MB (#9258) 2020-04-08 06:04:27 -07:00
Harshavardhana
2642e12d14
fix: change policies API to return and take struct (#9181)
This allows for order guarantees in returned values
can be consumed safely by the caller to avoid any
additional parsing and validation.

Fixes #9171
2020-04-07 19:30:59 -07:00
Harshavardhana
2c20716f37
fix: Avoid force delete in compliance/worm mode (#9276)
also, bring in an additional policy to ensure that
force delete bucket is only allowed with the right
policy for the user, just DeleteBucketAction
policy action is not enough.
2020-04-06 17:51:05 -07:00
Harshavardhana
928f5b0564
fix: Quit when the context is canceled in madmin (#9264) 2020-04-06 17:50:14 -07:00
Harshavardhana
43a3778b45
fix: support object-remaining-retention-days policy condition (#9259)
This PR also tries to simplify the approach taken in
object-locking implementation by preferential treatment
given towards full validation.

This in-turn has fixed couple of bugs related to
how policy should have been honored when ByPassGovernance
is provided.

Simplifies code a bit, but also duplicates code intentionally
for clarity due to complex nature of object locking
implementation.
2020-04-06 13:44:16 -07:00
Anis Elleuch
9902c9baaa
sql: Add support of escape quote in CSV (#9231)
This commit modifies csv parser, a fork of golang csv
parser to support a custom quote escape character.

The quote escape character is used to escape the quote
character when a csv field contains a quote character
as part of data.
2020-04-01 15:39:34 -07:00
Harshavardhana
7de29e6e6b
Add rotating token support for admin API (#9244)
Use the *credentials.Credentials implementation method *Get*

```
func (c *Credentials) Get() (Value, error) {
```

which also handles auto-refresh, this allows for chaining
of various implementations together if necessary or simply
initialize with credentials.NewStaticV4(access, secret, token)

Co-authored-by: Klaus Post <klauspost@gmail.com>
2020-04-01 13:34:20 -07:00
Harshavardhana
d8af244708
Add numeric/date policy conditions (#9233)
add new policy conditions

- NumericEquals
- NumericNotEquals
- NumericLessThan
- NumericLessThanEquals
- NumericGreaterThan
- NumericGreaterThanEquals
- DateEquals
- DateNotEquals
- DateLessThan
- DateLessThanEquals
- DateGreaterThan
- DateGreaterThanEquals
2020-04-01 00:04:25 -07:00
Sidhartha Mani
c8243706b4
Add Parallel NetOBD tests to saturate all nodes at once (#9241) 2020-03-31 17:08:28 -07:00
Harshavardhana
30707659b5
[feature] allow for an odd number of erasure packs (#9221)
Too many deployments come up with an odd number
of hosts or drives, to facilitate even distribution
among those setups allow for odd and prime numbers
based packs.
2020-03-31 09:32:16 -07:00
Harshavardhana
6f992134a2
fix: startup load time by reusing storageDisks (#9210) 2020-03-27 14:48:30 -07:00
Sidhartha Mani
0c80bf45d0
Implement oboard diagnostics admin API (#9024)
- Implement a graph algorithm to test network bandwidth from every 
  node to every other node
- Saturate any network bandwidth adaptively, accounting for slow 
  and fast network capacity
- Implement parallel drive OBD tests
- Implement a paging mechanism for OBD test to provide periodic updates to client
- Implement Sys, Process, Host, Mem OBD Infos
2020-03-26 21:07:39 -07:00
Anis Elleuch
b207520d98
Fix lifecycle GET: AWS SDK complaints on empty config (#9201) 2020-03-25 21:06:03 -07:00
Praveen raj Mani
e7a0be5bd3
fix: throttling of events during their replay (#9188) 2020-03-23 12:34:39 -07:00
stefan-work
f001e99fcd
create the final file with mode 0666 for multipart-uploads (#9173)
NAS gateway creates non-multipart-uploads with mode 0666.
But multipart-uploads are created with a differing mode of 0644.

Both modes should be equal! Else it leads to files with different
permissions based on its file-size. This patch solves that by
using 0666 for both cases.
2020-03-20 15:32:15 -07:00
Harshavardhana
b4bfdc92cc fix: admin console logger changes to log.Info 2020-03-20 15:14:14 -07:00
Harshavardhana
ae654831aa
Add madmin package context support (#9172)
This is to improve responsiveness for all
admin API operations and allowing callers
to cancel any on-going admin operations,
if they happen to be waiting too long.
2020-03-20 15:00:44 -07:00
Stephen N
1ffa983a9d
added support for SASL/SCRAM on Kafka bucket notifications. (#9168)
fixes #9167
2020-03-20 11:10:27 -07:00
Klaus Post
8d98662633
re-implement data usage crawler to be more efficient (#9075)
Implementation overview: 

https://gist.github.com/klauspost/1801c858d5e0df391114436fdad6987b
2020-03-18 16:19:29 -07:00
Anis Elleuch
496f4a7dc7
Add service account type in IAM (#9029) 2020-03-17 10:36:13 -07:00
Anis Elleuch
35ecc04223
Support configurable quote character parameter in Select (#8955) 2020-03-13 22:09:34 -07:00
Kody A Kantor
06e30b5aa1
Skip building directio on platforms that don't support Direct IO (#9059) 2020-03-12 18:57:41 -07:00
Harshavardhana
603cf2a8bb
fix: broken gzip handling with Select API (#9128)
This PR fixes a regression introduced in a1c7c9ea73
2020-03-12 15:34:11 -07:00
Anis Elleuch
fdf65aa9b9
heal: Add info about the next background healing round (#9122)
- avoid setting last heal activity when starting self-healing

This can be confusing to users thinking that the self healing
cycle was already performed.

- add info about the next background healing round
2020-03-11 23:00:31 -07:00
Aditya Manthramurthy
cec8cdb35e
S3Select: Handle array selection in from clause (#9076) 2020-03-10 22:34:58 -07:00
kannappanr
33edb072a3
Add TopLocksAdminAction to diagsnostics canned policy (#9104) 2020-03-08 18:32:39 -07:00
ebozduman
a1c7c9ea73
Matches s3 invalid compression format error for 'mc sql' (#9067) 2020-03-05 19:34:04 -08:00
kannappanr
2f2c7d91a8
Add new extended list of JWT keys from OpenID group (#9087)
https://www.iana.org/assignments/jwt/jwt.xhtml#claims
2020-03-05 05:05:36 -08:00
Klaus Post
f1b2462193
Add goroutine profiles (#9078)
Allow downloading goroutine dump to help detect leaks
or overuse of goroutines.

Extensions are now type dependent.

Change `profiling` -> `profile` prefix, since that is what they are 
not the abstract concept.
2020-03-04 06:58:12 -08:00
Hamid
5b8975bf4b
Simplify redis access event format to faciliate parsing (#9046) 2020-02-26 09:23:32 -08:00
Kody A Kantor
deb3911f5e
cpu package fails to build on illumos (#9036) 2020-02-25 10:58:18 -08:00