config: Check for duplicated entries in all scopes (#3872)

Validate Minio config by checking if there is double json key
in any scope level. The returned error contains the json path
to the duplicated key.
This commit is contained in:
Anis Elleuch
2017-03-16 00:30:34 +01:00
committed by Harshavardhana
parent cad0d0eb7a
commit ae4361cc45
19 changed files with 1261 additions and 55 deletions

View File

@@ -11,7 +11,7 @@
<p align="center">get a json value quickly</a></p>
GJSON is a Go package the provides a [very fast](#performance) and simple way to get a value from a json document. The reason for this library it to give efficient json indexing for the [BuntDB](https://github.com/tidwall/buntdb) project.
GJSON is a Go package that provides a [very fast](#performance) and simple way to get a value from a json document. The purpose for this library it to give efficient json indexing for the [BuntDB](https://github.com/tidwall/buntdb) project.
Getting Started
===============
@@ -27,7 +27,7 @@ $ go get -u github.com/tidwall/gjson
This will retrieve the library.
## Get a value
Get searches json for the specified path. A path is in dot syntax, such as "name.last" or "age". This function expects that the json is well-formed and validates. Invalid json will not panic, but it may return back unexpected results. When the value is found it's returned immediately.
Get searches json for the specified path. A path is in dot syntax, such as "name.last" or "age". This function expects that the json is well-formed and validates. Invalid json will not panic, but it may return back unexpected results. When the value is found it's returned immediately.
```go
package main
@@ -47,6 +47,7 @@ This will print:
```
Prichard
```
*There's also the [GetMany](#get-multiple-values-at-once) function to get multiple values at once, and [GetBytes](#working-with-bytes) for working with JSON byte slices.*
## Path Syntax
@@ -63,25 +64,33 @@ The dot and wildcard characters can be escaped with '\'.
"children": ["Sara","Alex","Jack"],
"fav.movie": "Deer Hunter",
"friends": [
{"first": "James", "last": "Murphy"},
{"first": "Roger", "last": "Craig"}
{"first": "Dale", "last": "Murphy", "age": 44},
{"first": "Roger", "last": "Craig", "age": 68},
{"first": "Jane", "last": "Murphy", "age": 47}
]
}
```
```
"name.last" >> "Anderson"
"age" >> 37
"children" >> ["Sara","Alex","Jack"]
"children.#" >> 3
"children.1" >> "Alex"
"child*.2" >> "Jack"
"c?ildren.0" >> "Sara"
"fav\.movie" >> "Deer Hunter"
"friends.#.first" >> [ "James", "Roger" ]
"friends.#.first" >> ["Dale","Roger","Jane"]
"friends.1.last" >> "Craig"
```
To query an array:
You can also query an array for the first match by using `#[...]`, or find all matches with `#[...]#`.
Queries support the `==`, `!=`, `<`, `<=`, `>`, `>=` comparison operators and the simple pattern matching `%` operator.
```
`friends.#[last="Murphy"].first` >> "James"
friends.#[last=="Murphy"].first >> "Dale"
friends.#[last=="Murphy"]#.first >> ["Dale","Jane"]
friends.#[age>45]#.last >> ["Craig","Murphy"]
friends.#[first%"D*"].last >> "Murphy"
```
## Result Type
@@ -105,7 +114,7 @@ result.Type // can be String, Number, True, False, Null, or JSON
result.Str // holds the string
result.Num // holds the float64 number
result.Raw // holds the raw json
result.Multi // holds nested array values
result.Index // index of raw value in original json, zero means index unknown
```
There are a variety of handy functions that work on a result:
@@ -113,16 +122,25 @@ There are a variety of handy functions that work on a result:
```go
result.Value() interface{}
result.Int() int64
result.Uint() uint64
result.Float() float64
result.String() string
result.Bool() bool
result.Array() []gjson.Result
result.Map() map[string]gjson.Result
result.Get(path string) Result
result.ForEach(iterator func(key, value Result) bool)
result.Less(token Result, caseSensitive bool) bool
```
The `result.Value()` function returns an `interface{}` which requires type assertion and is one of the following Go types:
The `result.Array()` function returns back an array of values.
If the result represents a non-existent value, then an empty array will be returned.
If the result is not a JSON array, the return value will be an array containing one result.
```go
boolean >> bool
number >> float64
@@ -169,6 +187,20 @@ name := gjson.Get(json, `programmers.#[lastName="Hunter"].firstName`)
println(name.String()) // prints "Elliotte"
```
## Iterate through an object or array
The `ForEach` function allows for quickly iterating through an object or array.
The key and value are passed to the iterator function for objects.
Only the value is passed for arrays.
Returning `false` from an iterator will stop iteration.
```go
result := gjson.Get(json, "programmers")
result.ForEach(func(key, value gjson.Result) bool{
println(value.String())
return true // keep iterating
})
```
## Simple Parse and Get
@@ -184,7 +216,7 @@ gjson.Get(json, "name.last")
## Check for the existence of a value
Sometimes you just want to know you if a value exists.
Sometimes you just want to know if a value exists.
```go
value := gjson.Get(json, "name.last")
@@ -211,6 +243,40 @@ if !ok{
}
```
## Working with Bytes
If your JSON is contained in a `[]byte` slice, there's the [GetBytes](https://godoc.org/github.com/tidwall/gjson#GetBytes) function. This is preferred over `Get(string(data), path)`.
```go
var json []byte = ...
result := gjson.GetBytes(json, path)
```
If you are using the `gjson.GetBytes(json, path)` function and you want to avoid converting `result.Raw` to a `[]byte`, then you can use this pattern:
```go
var json []byte = ...
result := gjson.GetBytes(json, path)
var raw []byte
if result.Index > 0 {
raw = json[result.Index:result.Index+len(result.Raw)]
} else {
raw = []byte(result.Raw)
}
```
This is a best-effort no allocation sub slice of the original json. This method utilizes the `result.Index` field, which is the position of the raw data in the original json. It's possible that the value of `result.Index` equals zero, in which case the `result.Raw` is converted to a `[]byte`.
## Get multiple values at once
The `GetMany` function can be used to get multiple values at the same time, and is optimized to scan over a JSON payload once.
```go
results := gjson.GetMany(json, "name.first", "name.last", "age")
```
The return value is a `[]Result`, which will always contain exactly the same number of items as the input paths.
## Performance
Benchmarks of GJSON alongside [encoding/json](https://golang.org/pkg/encoding/json/),
@@ -229,6 +295,17 @@ BenchmarkEasyJSONLexer-8 3000000 938 ns/op 613 B/op
BenchmarkJSONParserGet-8 3000000 442 ns/op 21 B/op 0 allocs/op
```
Benchmarks for the `GetMany` function:
```
BenchmarkGJSONGetMany4Paths-8 4000000 319 ns/op 112 B/op 0 allocs/op
BenchmarkGJSONGetMany8Paths-8 8000000 218 ns/op 56 B/op 0 allocs/op
BenchmarkGJSONGetMany16Paths-8 16000000 160 ns/op 56 B/op 0 allocs/op
BenchmarkGJSONGetMany32Paths-8 32000000 130 ns/op 64 B/op 0 allocs/op
BenchmarkGJSONGetMany64Paths-8 64000000 117 ns/op 64 B/op 0 allocs/op
BenchmarkGJSONGetMany128Paths-8 128000000 109 ns/op 64 B/op 0 allocs/op
```
JSON document used:
```json
@@ -267,6 +344,20 @@ widget.image.hOffset
widget.text.onMouseUp
```
For the `GetMany` benchmarks these paths are used:
```
widget.window.name
widget.image.hOffset
widget.text.onMouseUp
widget.window.title
widget.image.alignment
widget.text.style
widget.window.height
widget.image.src
widget.text.data
widget.text.size
```
*These benchmarks were run on a MacBook Pro 15" 2.8 GHz Intel Core i7 using Go 1.7.*

View File

@@ -4,6 +4,10 @@ package gjson
import (
"reflect"
"strconv"
// It's totally safe to use this package, but in case your
// project or organization restricts the use of 'unsafe',
// there's the "github.com/tidwall/gjson-safe" package.
"unsafe"
"github.com/tidwall/match"
@@ -27,6 +31,26 @@ const (
JSON
)
// String returns a string representation of the type.
func (t Type) String() string {
switch t {
default:
return ""
case Null:
return "Null"
case False:
return "False"
case Number:
return "Number"
case String:
return "String"
case True:
return "True"
case JSON:
return "JSON"
}
}
// Result represents a json value that is returned from Get().
type Result struct {
// Type is the json type
@@ -37,6 +61,8 @@ type Result struct {
Str string
// Num is the json number
Num float64
// Index of raw value in original json, zero means index unknown
Index int
}
// String returns a string representation of the value.
@@ -86,6 +112,21 @@ func (t Result) Int() int64 {
}
}
// Uint returns an unsigned integer representation.
func (t Result) Uint() uint64 {
switch t.Type {
default:
return 0
case True:
return 1
case String:
n, _ := strconv.ParseUint(t.Str, 10, 64)
return n
case Number:
return uint64(t.Num)
}
}
// Float returns an float64 representation.
func (t Result) Float() float64 {
switch t.Type {
@@ -101,16 +142,91 @@ func (t Result) Float() float64 {
}
}
// Array returns back an array of children. The result must be a JSON array.
// Array returns back an array of values.
// If the result represents a non-existent value, then an empty array will be returned.
// If the result is not a JSON array, the return value will be an array containing one result.
func (t Result) Array() []Result {
if t.Type != JSON {
if !t.Exists() {
return nil
}
if t.Type != JSON {
return []Result{t}
}
r := t.arrayOrMap('[', false)
return r.a
}
// Map returns back an map of children. The result should be a JSON array.
// ForEach iterates through values.
// If the result represents a non-existent value, then no values will be iterated.
// If the result is an Object, the iterator will pass the key and value of each item.
// If the result is an Array, the iterator will only pass the value of each item.
// If the result is not a JSON array or object, the iterator will pass back one value equal to the result.
func (t Result) ForEach(iterator func(key, value Result) bool) {
if !t.Exists() {
return
}
if t.Type != JSON {
iterator(Result{}, t)
return
}
json := t.Raw
var keys bool
var i int
var key, value Result
for ; i < len(json); i++ {
if json[i] == '{' {
i++
key.Type = String
keys = true
break
} else if json[i] == '[' {
i++
break
}
if json[i] > ' ' {
return
}
}
var str string
var vesc bool
var ok bool
for ; i < len(json); i++ {
if keys {
if json[i] != '"' {
continue
}
s := i
i, str, vesc, ok = parseString(json, i+1)
if !ok {
return
}
if vesc {
key.Str = unescape(str[1 : len(str)-1])
} else {
key.Str = str[1 : len(str)-1]
}
key.Raw = str
key.Index = s
}
for ; i < len(json); i++ {
if json[i] <= ' ' || json[i] == ',' || json[i] == ':' {
continue
}
break
}
s := i
i, value, ok = parseAny(json, i, true)
if !ok {
return
}
value.Index = s
if !iterator(key, value) {
return
}
}
}
// Map returns back an map of values. The result should be a JSON array.
func (t Result) Map() map[string]Result {
if t.Type != JSON {
return map[string]Result{}
@@ -232,7 +348,7 @@ end:
return
}
// Parse parses the json and returns a result
// Parse parses the json and returns a result.
func Parse(json string) Result {
var value Result
for i := 0; i < len(json); i++ {
@@ -270,6 +386,12 @@ func Parse(json string) Result {
return value
}
// ParseBytes parses the json and returns a result.
// If working with bytes, this method preferred over Parse(string(data))
func ParseBytes(json []byte) Result {
return Parse(string(json))
}
func squash(json string) string {
// expects that the lead character is a '[' or '{'
// squash the value, ignoring all nested arrays and objects.
@@ -387,7 +509,13 @@ func tostr(json string) (raw string, str string) {
break
}
}
return json[:i+1], unescape(json[1:i])
var ret string
if i+1 < len(json) {
ret = json[:i+1]
} else {
ret = json[:i]
}
return ret, unescape(json[1:i])
}
}
return json, json[1:]
@@ -506,6 +634,7 @@ type arrayPathResult struct {
path string
op string
value string
all bool
}
}
@@ -536,8 +665,12 @@ func parseArrayPath(path string) (r arrayPathResult) {
}
s := i
for ; i < len(path); i++ {
if path[i] <= ' ' || path[i] == '=' ||
path[i] == '<' || path[i] == '>' ||
if path[i] <= ' ' ||
path[i] == '!' ||
path[i] == '=' ||
path[i] == '<' ||
path[i] == '>' ||
path[i] == '%' ||
path[i] == ']' {
break
}
@@ -551,7 +684,11 @@ func parseArrayPath(path string) (r arrayPathResult) {
}
if i < len(path) {
s = i
if path[i] == '<' || path[i] == '>' {
if path[i] == '!' {
if i < len(path)-1 && path[i+1] == '=' {
i++
}
} else if path[i] == '<' || path[i] == '>' {
if i < len(path)-1 && path[i+1] == '=' {
i++
}
@@ -596,6 +733,9 @@ func parseArrayPath(path string) (r arrayPathResult) {
}
}
} else if path[i] == ']' {
if i+1 < len(path) && path[i+1] == '#' {
r.query.all = true
}
break
}
}
@@ -877,6 +1017,8 @@ func queryMatches(rp *arrayPathResult, value Result) bool {
switch rp.query.op {
case "=":
return value.Str == rpv
case "!=":
return value.Str != rpv
case "<":
return value.Str < rpv
case "<=":
@@ -885,12 +1027,16 @@ func queryMatches(rp *arrayPathResult, value Result) bool {
return value.Str > rpv
case ">=":
return value.Str >= rpv
case "%":
return match.Match(value.Str, rpv)
}
case Number:
rpvn, _ := strconv.ParseFloat(rpv, 64)
switch rp.query.op {
case "=":
return value.Num == rpvn
case "!=":
return value.Num == rpvn
case "<":
return value.Num < rpvn
case "<=":
@@ -904,6 +1050,8 @@ func queryMatches(rp *arrayPathResult, value Result) bool {
switch rp.query.op {
case "=":
return rpv == "true"
case "!=":
return rpv != "true"
case ">":
return rpv == "false"
case ">=":
@@ -913,6 +1061,8 @@ func queryMatches(rp *arrayPathResult, value Result) bool {
switch rp.query.op {
case "=":
return rpv == "false"
case "!=":
return rpv != "false"
case "<":
return rpv == "true"
case "<=":
@@ -927,6 +1077,7 @@ func parseArray(c *parseContext, i int, path string) (int, bool) {
var h int
var alog []int
var partidx int
var multires []byte
rp := parseArrayPath(path)
if !rp.arrch {
n, err := strconv.ParseUint(rp.part, 10, 64)
@@ -983,12 +1134,21 @@ func parseArray(c *parseContext, i int, path string) (int, bool) {
res := Get(val, rp.query.path)
if queryMatches(&rp, res) {
if rp.more {
c.value = Get(val, rp.path)
res = Get(val, rp.path)
} else {
c.value.Raw = val
c.value.Type = JSON
res = Result{Raw: val, Type: JSON}
}
if rp.query.all {
if len(multires) == 0 {
multires = append(multires, '[')
} else {
multires = append(multires, ',')
}
multires = append(multires, res.Raw...)
} else {
c.value = res
return i, true
}
return i, true
}
} else if hit {
if rp.alogok {
@@ -1051,13 +1211,14 @@ func parseArray(c *parseContext, i int, path string) (int, bool) {
if rp.alogok {
var jsons = make([]byte, 0, 64)
jsons = append(jsons, '[')
for j := 0; j < len(alog); j++ {
for j, k := 0, 0; j < len(alog); j++ {
res := Get(c.json[alog[j]:], rp.alogkey)
if res.Exists() {
if j > 0 {
if k > 0 {
jsons = append(jsons, ',')
}
jsons = append(jsons, []byte(res.Raw)...)
k++
}
}
jsons = append(jsons, ']')
@@ -1071,9 +1232,16 @@ func parseArray(c *parseContext, i int, path string) (int, bool) {
c.value.Raw = val
c.value.Type = Number
c.value.Num = float64(h - 1)
c.calcd = true
return i + 1, true
}
}
if len(multires) > 0 && !c.value.Exists() {
c.value = Result{
Raw: string(append(multires, ']')),
Type: JSON,
}
}
return i + 1, false
}
break
@@ -1085,6 +1253,7 @@ func parseArray(c *parseContext, i int, path string) (int, bool) {
type parseContext struct {
json string
value Result
calcd bool
}
// Get searches json for the specified path.
@@ -1093,7 +1262,7 @@ type parseContext struct {
// Invalid json will not panic, but it may return back unexpected results.
// When the value is found it's returned immediately.
//
// A path is a series of keys seperated by a dot.
// A path is a series of keys searated by a dot.
// A key may contain special wildcard characters '*' and '?'.
// To access an array value use the index as the key.
// To get the number of elements in an array or to access a child path, use the '#' character.
@@ -1110,11 +1279,12 @@ type parseContext struct {
// }
// "name.last" >> "Anderson"
// "age" >> 37
// "children" >> ["Sara","Alex","Jack"]
// "children.#" >> 3
// "children.1" >> "Alex"
// "child*.2" >> "Jack"
// "c?ildren.0" >> "Sara"
// "friends.#.first" >> [ "James", "Roger" ]
// "friends.#.first" >> ["James","Roger"]
//
func Get(json, path string) Result {
var i int
@@ -1131,8 +1301,53 @@ func Get(json, path string) Result {
break
}
}
if len(c.value.Raw) > 0 && !c.calcd {
jhdr := *(*reflect.StringHeader)(unsafe.Pointer(&json))
rhdr := *(*reflect.StringHeader)(unsafe.Pointer(&(c.value.Raw)))
c.value.Index = int(rhdr.Data - jhdr.Data)
if c.value.Index < 0 || c.value.Index >= len(json) {
c.value.Index = 0
}
}
return c.value
}
func fromBytesGet(result Result) Result {
// safely get the string headers
rawhi := *(*reflect.StringHeader)(unsafe.Pointer(&result.Raw))
strhi := *(*reflect.StringHeader)(unsafe.Pointer(&result.Str))
// create byte slice headers
rawh := reflect.SliceHeader{Data: rawhi.Data, Len: rawhi.Len}
strh := reflect.SliceHeader{Data: strhi.Data, Len: strhi.Len}
if strh.Data == 0 {
// str is nil
if rawh.Data == 0 {
// raw is nil
result.Raw = ""
} else {
// raw has data, safely copy the slice header to a string
result.Raw = string(*(*[]byte)(unsafe.Pointer(&rawh)))
}
result.Str = ""
} else if rawh.Data == 0 {
// raw is nil
result.Raw = ""
// str has data, safely copy the slice header to a string
result.Str = string(*(*[]byte)(unsafe.Pointer(&strh)))
} else if strh.Data >= rawh.Data &&
int(strh.Data)+strh.Len <= int(rawh.Data)+rawh.Len {
// Str is a substring of Raw.
start := int(strh.Data - rawh.Data)
// safely copy the raw slice header
result.Raw = string(*(*[]byte)(unsafe.Pointer(&rawh)))
// substring the raw
result.Str = result.Raw[start : start+strh.Len]
} else {
// safely copy both the raw and str slice headers to strings
result.Raw = string(*(*[]byte)(unsafe.Pointer(&rawh)))
result.Str = string(*(*[]byte)(unsafe.Pointer(&strh)))
}
return result
}
// GetBytes searches json for the specified path.
// If working with bytes, this method preferred over Get(string(data), path)
@@ -1141,29 +1356,7 @@ func GetBytes(json []byte, path string) Result {
if json != nil {
// unsafe cast to string
result = Get(*(*string)(unsafe.Pointer(&json)), path)
// copy of string data for safety.
rawh := *(*reflect.SliceHeader)(unsafe.Pointer(&result.Raw))
strh := *(*reflect.SliceHeader)(unsafe.Pointer(&result.Str))
if strh.Data == 0 {
if rawh.Data == 0 {
result.Raw = ""
} else {
result.Raw = string(*(*[]byte)(unsafe.Pointer(&result.Raw)))
}
result.Str = ""
} else if rawh.Data == 0 {
result.Raw = ""
result.Str = string(*(*[]byte)(unsafe.Pointer(&result.Str)))
} else if strh.Data >= rawh.Data &&
int(strh.Data)+strh.Len <= int(rawh.Data)+rawh.Len {
// Str is a substring of Raw.
start := int(strh.Data - rawh.Data)
result.Raw = string(*(*[]byte)(unsafe.Pointer(&result.Raw)))
result.Str = result.Raw[start : start+strh.Len]
} else {
result.Raw = string(*(*[]byte)(unsafe.Pointer(&result.Raw)))
result.Str = string(*(*[]byte)(unsafe.Pointer(&result.Str)))
}
result = fromBytesGet(result)
}
return result
}
@@ -1289,3 +1482,465 @@ func stringLessInsensitive(a, b string) bool {
}
return len(a) < len(b)
}
// parseAny parses the next value from a json string.
// A Result is returned when the hit param is set.
// The return values are (i int, res Result, ok bool)
func parseAny(json string, i int, hit bool) (int, Result, bool) {
var res Result
var val string
for ; i < len(json); i++ {
if json[i] == '{' || json[i] == '[' {
i, val = parseSquash(json, i)
if hit {
res.Raw = val
res.Type = JSON
}
return i, res, true
}
if json[i] <= ' ' {
continue
}
switch json[i] {
case '"':
i++
var vesc bool
var ok bool
i, val, vesc, ok = parseString(json, i)
if !ok {
return i, res, false
}
if hit {
res.Type = String
res.Raw = val
if vesc {
res.Str = unescape(val[1 : len(val)-1])
} else {
res.Str = val[1 : len(val)-1]
}
}
return i, res, true
case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
i, val = parseNumber(json, i)
if hit {
res.Raw = val
res.Type = Number
res.Num, _ = strconv.ParseFloat(val, 64)
}
return i, res, true
case 't', 'f', 'n':
vc := json[i]
i, val = parseLiteral(json, i)
if hit {
res.Raw = val
switch vc {
case 't':
res.Type = True
case 'f':
res.Type = False
}
return i, res, true
}
}
}
return i, res, false
}
var ( // used for testing
testWatchForFallback bool
testLastWasFallback bool
)
// areSimplePaths returns true if all the paths are simple enough
// to parse quickly for GetMany(). Allows alpha-numeric, dots,
// underscores, and the dollar sign. It does not allow non-alnum,
// escape characters, or keys which start with a numbers.
// For example:
// "name.last" == OK
// "user.id0" == OK
// "user.ID" == OK
// "user.first_name" == OK
// "user.firstName" == OK
// "user.0item" == BAD
// "user.#id" == BAD
// "user\.name" == BAD
func areSimplePaths(paths []string) bool {
for _, path := range paths {
var fi int // first key index, for keys with numeric prefix
for i := 0; i < len(path); i++ {
if path[i] >= 'a' && path[i] <= 'z' {
// a-z is likely to be the highest frequency charater.
continue
}
if path[i] == '.' {
fi = i + 1
continue
}
if path[i] >= 'A' && path[i] <= 'Z' {
continue
}
if path[i] == '_' || path[i] == '$' {
continue
}
if i > fi && path[i] >= '0' && path[i] <= '9' {
continue
}
return false
}
}
return true
}
// GetMany searches json for the multiple paths.
// The return value is a Result array where the number of items
// will be equal to the number of input paths.
func GetMany(json string, paths ...string) []Result {
if len(paths) < 4 {
if testWatchForFallback {
testLastWasFallback = false
}
switch len(paths) {
case 0:
// return nil when no paths are specified.
return nil
case 1:
return []Result{Get(json, paths[0])}
case 2:
return []Result{Get(json, paths[0]), Get(json, paths[1])}
case 3:
return []Result{Get(json, paths[0]), Get(json, paths[1]), Get(json, paths[2])}
}
}
var results []Result
var ok bool
var i int
if len(paths) > 512 {
// we can only support up to 512 paths. Is that too many?
goto fallback
}
if !areSimplePaths(paths) {
// If there is even one path that is not considered "simple" then
// we need to use the fallback method.
goto fallback
}
// locate the object token.
for ; i < len(json); i++ {
if json[i] == '{' {
i++
break
}
if json[i] <= ' ' {
continue
}
goto fallback
}
// use the call function table.
if len(paths) <= 8 {
results, ok = getMany8(json, i, paths)
} else if len(paths) <= 16 {
results, ok = getMany16(json, i, paths)
} else if len(paths) <= 32 {
results, ok = getMany32(json, i, paths)
} else if len(paths) <= 64 {
results, ok = getMany64(json, i, paths)
} else if len(paths) <= 128 {
results, ok = getMany128(json, i, paths)
} else if len(paths) <= 256 {
results, ok = getMany256(json, i, paths)
} else if len(paths) <= 512 {
results, ok = getMany512(json, i, paths)
}
if !ok {
// there was some fault while parsing. we should try the
// fallback method. This could result in performance
// degregation in some cases.
goto fallback
}
if testWatchForFallback {
testLastWasFallback = false
}
return results
fallback:
results = results[:0]
for i := 0; i < len(paths); i++ {
results = append(results, Get(json, paths[i]))
}
if testWatchForFallback {
testLastWasFallback = true
}
return results
}
// GetManyBytes searches json for the specified path.
// If working with bytes, this method preferred over
// GetMany(string(data), paths...)
func GetManyBytes(json []byte, paths ...string) []Result {
if json == nil {
return GetMany("", paths...)
}
results := GetMany(*(*string)(unsafe.Pointer(&json)), paths...)
for i := range results {
results[i] = fromBytesGet(results[i])
}
return results
}
// parseGetMany parses a json object for keys that match against the callers
// paths. It's a best-effort attempt and quickly locating and assigning the
// values to the []Result array. If there are failures such as bad json, or
// invalid input paths, or too much recursion, the function will exit with a
// return value of 'false'.
func parseGetMany(
json string, i int,
level uint, kplen int,
paths []string, completed []bool, matches []uint64, results []Result,
) (int, bool) {
if level > 62 {
// The recursion level is limited because the matches []uint64
// array cannot handle more the 64-bits.
return i, false
}
// At this point the last character read was a '{'.
// Read all object keys and try to match against the paths.
var key string
var val string
var vesc, ok bool
next_key:
for ; i < len(json); i++ {
if json[i] == '"' {
// read the key
i, val, vesc, ok = parseString(json, i+1)
if !ok {
return i, false
}
if vesc {
// the value is escaped
key = unescape(val[1 : len(val)-1])
} else {
// just a plain old ascii key
key = val[1 : len(val)-1]
}
var hasMatch bool
var parsedVal bool
var valOrgIndex int
var valPathIndex int
for j := 0; j < len(key); j++ {
if key[j] == '.' {
// we need to look for keys with dot and ignore them.
if i, _, ok = parseAny(json, i, false); !ok {
return i, false
}
continue next_key
}
}
var usedPaths int
// loop through paths and look for matches
for j := 0; j < len(paths); j++ {
if completed[j] {
usedPaths++
// ignore completed paths
continue
}
if level > 0 && (matches[j]>>(level-1))&1 == 0 {
// ignore unmatched paths
usedPaths++
continue
}
// try to match the key to the path
// this is spaghetti code but the idea is to minimize
// calls and variable assignments when comparing the
// key to paths
if len(paths[j])-kplen >= len(key) {
i, k := kplen, 0
for ; k < len(key); k, i = k+1, i+1 {
if key[k] != paths[j][i] {
// no match
goto nomatch
}
}
if i < len(paths[j]) {
if paths[j][i] == '.' {
// matched, but there still more keys in the path
goto match_not_atend
}
}
// matched and at the end of the path
goto match_atend
}
// no match, jump to the nomatch label
goto nomatch
match_atend:
// found a match
// at the end of the path. we must take the value.
usedPaths++
if !parsedVal {
// the value has not been parsed yet. let's do so.
valOrgIndex = i // keep track of the current position.
i, results[j], ok = parseAny(json, i, true)
if !ok {
return i, false
}
parsedVal = true
valPathIndex = j
} else {
results[j] = results[valPathIndex]
}
// mark as complete
completed[j] = true
// jump over the match_not_atend label
goto nomatch
match_not_atend:
// found a match
// still in the middle of the path.
usedPaths++
// mark the path as matched
matches[j] |= 1 << level
if !hasMatch {
hasMatch = true
}
nomatch: // noop label
}
if !parsedVal {
if hasMatch {
// we found a match and the value has not been parsed yet.
// let's find out if the next value type is an object.
for ; i < len(json); i++ {
if json[i] <= ' ' || json[i] == ':' {
continue
}
break
}
if i < len(json) {
if json[i] == '{' {
// it's an object. let's go deeper
i, ok = parseGetMany(json, i+1, level+1, kplen+len(key)+1, paths, completed, matches, results)
if !ok {
return i, false
}
} else {
// not an object. just parse and ignore.
if i, _, ok = parseAny(json, i, false); !ok {
return i, false
}
}
}
} else {
// Since there was no matches we can just parse the value and
// ignore the result.
if i, _, ok = parseAny(json, i, false); !ok {
return i, false
}
}
} else if hasMatch && len(results[valPathIndex].Raw) > 0 && results[valPathIndex].Raw[0] == '{' {
// The value was already parsed and the value type is an object.
// Rewind the json index and let's parse deeper.
i = valOrgIndex
for ; i < len(json); i++ {
if json[i] == '{' {
break
}
}
i, ok = parseGetMany(json, i+1, level+1, kplen+len(key)+1, paths, completed, matches, results)
if !ok {
return i, false
}
}
if usedPaths == len(paths) {
// all paths have been used, either completed or matched.
// we should stop parsing this object to save CPU cycles.
if level > 0 && i < len(json) {
i, _ = parseSquash(json, i)
}
return i, true
}
} else if json[i] == '}' {
// reached the end of the object. end it here.
return i + 1, true
}
}
return i, true
}
// Call table for GetMany. Using an isolated function allows for allocating
// arrays with know capacities on the stack, as opposed to dynamically
// allocating on the heap. This can provide a tremendous performance boost
// by avoiding the GC.
func getMany8(json string, i int, paths []string) ([]Result, bool) {
const max = 8
var completed = make([]bool, 0, max)
var matches = make([]uint64, 0, max)
var results = make([]Result, 0, max)
completed = completed[0:len(paths):max]
matches = matches[0:len(paths):max]
results = results[0:len(paths):max]
_, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results)
return results, ok
}
func getMany16(json string, i int, paths []string) ([]Result, bool) {
const max = 16
var completed = make([]bool, 0, max)
var matches = make([]uint64, 0, max)
var results = make([]Result, 0, max)
completed = completed[0:len(paths):max]
matches = matches[0:len(paths):max]
results = results[0:len(paths):max]
_, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results)
return results, ok
}
func getMany32(json string, i int, paths []string) ([]Result, bool) {
const max = 32
var completed = make([]bool, 0, max)
var matches = make([]uint64, 0, max)
var results = make([]Result, 0, max)
completed = completed[0:len(paths):max]
matches = matches[0:len(paths):max]
results = results[0:len(paths):max]
_, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results)
return results, ok
}
func getMany64(json string, i int, paths []string) ([]Result, bool) {
const max = 64
var completed = make([]bool, 0, max)
var matches = make([]uint64, 0, max)
var results = make([]Result, 0, max)
completed = completed[0:len(paths):max]
matches = matches[0:len(paths):max]
results = results[0:len(paths):max]
_, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results)
return results, ok
}
func getMany128(json string, i int, paths []string) ([]Result, bool) {
const max = 128
var completed = make([]bool, 0, max)
var matches = make([]uint64, 0, max)
var results = make([]Result, 0, max)
completed = completed[0:len(paths):max]
matches = matches[0:len(paths):max]
results = results[0:len(paths):max]
_, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results)
return results, ok
}
func getMany256(json string, i int, paths []string) ([]Result, bool) {
const max = 256
var completed = make([]bool, 0, max)
var matches = make([]uint64, 0, max)
var results = make([]Result, 0, max)
completed = completed[0:len(paths):max]
matches = matches[0:len(paths):max]
results = results[0:len(paths):max]
_, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results)
return results, ok
}
func getMany512(json string, i int, paths []string) ([]Result, bool) {
const max = 512
var completed = make([]bool, 0, max)
var matches = make([]uint64, 0, max)
var results = make([]Result, 0, max)
completed = completed[0:len(paths):max]
matches = matches[0:len(paths):max]
results = results[0:len(paths):max]
_, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results)
return results, ok
}

6
vendor/vendor.json vendored
View File

@@ -296,10 +296,10 @@
"revisionTime": "2016-03-11T21:55:03Z"
},
{
"checksumSHA1": "+Pcohsuq0Mi/y8bgaDFjb/CGzkk=",
"checksumSHA1": "k/Xh0p5L7+tBCXAL2dOCwUf9J3Y=",
"path": "github.com/tidwall/gjson",
"revision": "7c631e98686a791e5fc60ff099512968122afb52",
"revisionTime": "2016-09-08T16:02:40Z"
"revision": "09d1c5c5bc64e094394dfe2150220d906c55ac37",
"revisionTime": "2017-02-05T16:10:42Z"
},
{
"checksumSHA1": "qmePMXEDYGwkAfT9QvtMC58JN/E=",