when parsing a json array, [] object yielded at end

[ilackarms]

i've noticed that when i parse an array, after yielding each of the objects in the array, an empty array object [] is yielded at the end

reproduce:

input:

{
    "kind": "PodList",
    "apiVersion": "v1",
    "metadata": {
        "selfLink": "/api/v1/pods",
        "resourceVersion": "1315"
    },
    "items": [
        {
            "metadata": {
                "name": "redis-master3",
                "namespace": "default",
                "selfLink": "/api/v1/pods/redis-master3?namespace=default",
                "uid": "1da148b4-cef5-11e4-ac24-3c970e4a436a",
                "resourceVersion": "1301",
                "creationTimestamp": "2015-03-20T13:34:48+02:00",
                "labels": {
                    "mylabel": "mylabelvalue",
                    "role": "pod"
                }
            },
            "spec": {
                "volumes": null,
                "containers": [
                    {
                        "name": "master",
                        "image": "dockerfile/redis",
                        "ports": [
                            {
                                "hostPort": 6379,
                                "containerPort": 6379,
                                "protocol": "TCP"
                            }
                        ],
                        "resources": {
                            "limits": {
                                "cpu": "100m"
                            }
                        },
                        "terminationMessagePath": "/dev/termination-log",
                        "imagePullPolicy": "IfNotPresent",
                        "securityContext": {
                            "capabilities": {}
                        }
                    },
                    {
                        "name": "php-redis",
                        "image": "kubernetes/example-guestbook-php-redis",
                        "ports": [
                            {
                                "hostPort": 8000,
                                "containerPort": 80,
                                "protocol": "TCP"
                            }
                        ],
                        "resources": {
                            "limits": {
                                "cpu": "100m",
                                "memory": "50000000"
                            }
                        },
                        "terminationMessagePath": "/dev/termination-log",
                        "imagePullPolicy": "IfNotPresent",
                        "securityContext": {
                            "capabilities": {}
                        }
                    }
                ],
                "restartPolicy": {
                    "always": {}
                },
                "dnsPolicy": "ClusterFirst"
            },
            "status": {
                "phase": "Pending"
            }
        }
    ]
}

parsing code:

streamer.get(key: 'items') do |object|
  p object
end

result:

{"metadata"=>{"name"=>"redis-master3", "namespace"=>"default", "selfLink"=>"/api/v1/pods/redis-master3?namespace=default", "uid"=>"1da148b4-cef5-11e4-ac24-3c970e4a436a", "resourceVersion"=>"1301", "creationTimestamp"=>"2015-03-20T13:34:48+02:00", "labels"=>{"mylabel"=>"mylabelvalue", "role"=>"pod"}}, "spec"=>{"volumes"=>nil, "containers"=>[{"name"=>"master", "image"=>"dockerfile/redis", "ports"=>[{"hostPort"=>6379, "containerPort"=>6379, "protocol"=>"TCP"}], "resources"=>{"limits"=>{"cpu"=>"100m"}}, "terminationMessagePath"=>"/dev/termination-log", "imagePullPolicy"=>"IfNotPresent", "securityContext"=>{"capabilities"=>{}}}, {"name"=>"php-redis", "image"=>"kubernetes/example-guestbook-php-redis", "ports"=>[{"hostPort"=>8000, "containerPort"=>80, "protocol"=>"TCP"}], "resources"=>{"limits"=>{"cpu"=>"100m", "memory"=>"50000000"}}, "terminationMessagePath"=>"/dev/termination-log", "imagePullPolicy"=>"IfNotPresent", "securityContext"=>{"capabilities"=>{}}}], "restartPolicy"=>{"always"=>{}}, "dnsPolicy"=>"ClusterFirst"}, "status"=>{"phase"=>"Pending"}}
[]

[thisismydesign]

Issue is caused by a bug that values within an array were handled as if they have keys (namely the previous key in the JSON object) while they should not have keys at all.

[thisismydesign]

Issue is fixed in v1.1.1. Please verify and close this issue if you're satisfied.

[ilackarms]

now I'm only getting the first object in the array

[thisismydesign]

What do you mean?

[thisismydesign]

I see what you mean, hang on..

[ilackarms]

input:

{
    "kind": "ServiceList",
    "apiVersion": "v1",
    "metadata": {
        "selfLink": "/api/v1/services",
        "resourceVersion": "59"
    },
    "items": [
        {
            "metadata": {
                "name": "kubernetes",
                "namespace": "default",
                "selfLink": "/api/v1/services/kubernetes?namespace=default",
                "uid": "016e9dcd-ce39-11e4-ac24-3c970e4a436a",
                "resourceVersion": "6",
                "creationTimestamp": "2015-03-19T15:08:16+02:00",
                "labels": {
                    "component": "apiserver",
                    "provider": "kubernetes"
                }
            },
            "spec": {
                "port": 443,
                "protocol": "TCP",
                "selector": null,
                "clusterIP": "10.0.0.2",
                "containerPort": 0,
                "sessionAffinity": "None"
            },
            "status": {}
        },
        {
            "metadata": {
                "name": "kubernetes-ro",
                "namespace": "default",
                "selfLink": "/api/v1/services/kubernetes-ro?namespace=default",
                "uid": "015b78bf-ce39-11e4-ac24-3c970e4a436a",
                "resourceVersion": "5",
                "creationTimestamp": "2015-03-19T15:08:15+02:00",
                "labels": {
                    "component": "apiserver",
                    "provider": "kubernetes"
                }
            },
            "spec": {
                "port": 80,
                "protocol": "TCP",
                "selector": null,
                "clusterIP": "10.0.0.1",
                "containerPort": 0,
                "sessionAffinity": "None"
            },
            "status": {}
        }
    ]
}

output:

{"metadata"=>{"name"=>"kubernetes", "namespace"=>"default", "selfLink"=>"/api/v1/services/kubernetes?namespace=default", "uid"=>"016e9dcd-ce39-11e4-ac24-3c970e4a436a", "resourceVersion"=>"6", "creationTimestamp"=>"2015-03-19T15:08:16+02:00", "labels"=>{"component"=>"apiserver", "provider"=>"kubernetes"}}, "spec"=>{"port"=>443, "protocol"=>"TCP", "selector"=>nil, "clusterIP"=>"10.0.0.2", "containerPort"=>0, "sessionAffinity"=>"None"}, "status"=>{}}

only the first object is yielded

[ilackarms]

i am also experiencing another new bug: certain keys are rendered as nil

input:

{
    "kind": "PodList",
    "apiVersion": "v1",
    "metadata": {
        "selfLink": "/api/v1/pods",
        "resourceVersion": "1315"
    },
    "items": [
        {
            "metadata": {
                "name": "redis-master3",
                "namespace": "default",
                "selfLink": "/api/v1/pods/redis-master3?namespace=default",
                "uid": "1da148b4-cef5-11e4-ac24-3c970e4a436a",
                "resourceVersion": "1301",
                "creationTimestamp": "2015-03-20T13:34:48+02:00",
                "labels": {
                    "mylabel": "mylabelvalue",
                    "role": "pod"
                }
            },
            "spec": {
                "volumes": null,
                "containers": [
                    {
                        "name": "master",
                        "image": "dockerfile/redis",
                        "ports": [
                            {
                                "hostPort": 6379,
                                "containerPort": 6379,
                                "protocol": "TCP"
                            }
                        ],
                        "resources": {
                            "limits": {
                                "cpu": "100m"
                            }
                        },
                        "terminationMessagePath": "/dev/termination-log",
                        "imagePullPolicy": "IfNotPresent",
                        "securityContext": {
                            "capabilities": {}
                        }
                    },
                    {
                        "name": "php-redis",
                        "image": "kubernetes/example-guestbook-php-redis",
                        "ports": [
                            {
                                "hostPort": 8000,
                                "containerPort": 80,
                                "protocol": "TCP"
                            }
                        ],
                        "resources": {
                            "limits": {
                                "cpu": "100m",
                                "memory": "50000000"
                            }
                        },
                        "terminationMessagePath": "/dev/termination-log",
                        "imagePullPolicy": "IfNotPresent",
                        "securityContext": {
                            "capabilities": {}
                        }
                    }
                ],
                "restartPolicy": {
                    "always": {}
                },
                "dnsPolicy": "ClusterFirst"
            },
            "status": {
                "phase": "Pending"
            }
        }
    ]
}

output:

{"metadata"=>{"name"=>"redis-master3", "namespace"=>"default", "selfLink"=>"/api/v1/pods/redis-master3?namespace=default", "uid"=>"1da148b4-cef5-11e4-ac24-3c970e4a436a", "resourceVersion"=>"1301", "creationTimestamp"=>"2015-03-20T13:34:48+02:00", "labels"=>{"mylabel"=>"mylabelvalue", "role"=>"pod"}}, "spec"=>{"volumes"=>nil, "labels"=>[{"name"=>"master", "image"=>"dockerfile/redis", nil=>[{"hostPort"=>6379, "containerPort"=>6379, "protocol"=>"TCP"}], "resources"=>{"limits"=>{"cpu"=>"100m"}}, "terminationMessagePath"=>"/dev/termination-log", "imagePullPolicy"=>"IfNotPresent", "securityContext"=>{"capabilities"=>{}}}, {"name"=>"php-redis", "image"=>"kubernetes/example-guestbook-php-redis", "securityContext"=>{"capabilities"=>{}}, "resources"=>{"limits"=>{"cpu"=>"100m", "memory"=>"50000000"}}, "terminationMessagePath"=>"/dev/termination-log", "imagePullPolicy"=>"IfNotPresent"}], "restartPolicy"=>{"always"=>{}}, "dnsPolicy"=>"ClusterFirst"}, "status"=>{"phase"=>"Pending"}}

(notice the line nil=>[{"hostPort"=>6379, "containerPort"=>6379, "protocol"=>"TCP"}]) parsed from the "ports" key above

[thisismydesign]

Turns out the cause was identified correctly (values within an array were handled as if they have keys) but I made wrong assumptions regarding fixing it. v1.1.2 should be fine, I also added more tests covering handling of arrays. Please verify again.

[ilackarms]

now i'm getting the whole object back as a single array. what I want to do is yield each object within the array one-by-one. is this possible with json-streamer? I've tried playing with combinations of key: 'items' and nesting_level: X`, but so far nothing has worked.

[thisismydesign]

Since the items key points to an array it will return an array, there's no way around that using the key matcher.

However..

Input: #7 (comment)
The following parameters {nesting_level: 2, yield_values: false}
Result:

{"metadata"=>{"name"=>"kubernetes", "namespace"=>"default", "selfLink"=>"/api/v1/services/kubernetes?namespace=default", "uid"=>"016e9dcd-ce39-11e4-ac24-3c970e4a436a", "resourceVersion"=>"6", "creationTimestamp"=>"2015-03-19T15:08:16+02:00", "labels"=>{"component"=>"apiserver", "provider"=>"kubernetes"}}, "spec"=>{"port"=>443, "protocol"=>"TCP", "selector"=>"null", "clusterIP"=>"10.0.0.2", "containerPort"=>0, "sessionAffinity"=>"None"}, "status"=>{}}

{"metadata"=>{"name"=>"kubernetes-ro", "namespace"=>"default", "selfLink"=>"/api/v1/services/kubernetes-ro?namespace=default", "uid"=>"015b78bf-ce39-11e4-ac24-3c970e4a436a", "resourceVersion"=>"5", "creationTimestamp"=>"2015-03-19T15:08:15+02:00", "labels"=>{"component"=>"apiserver", "provider"=>"kubernetes"}}, "spec"=>{"port"=>80, "protocol"=>"TCP", "selector"=>"null", "clusterIP"=>"10.0.0.1", "containerPort"=>0, "sessionAffinity"=>"None"}, "status"=>{}}

Is this what you're looking for?

Using v1.3.0 (latest).

[ilackarms]

@thisismydesign I'm looking for the same output, but with the parameters {key: 'items', nesting_level: 2} however i notice that everything with nesting_level 2 gets printed. I'd only like to access the contents of the array "items". is that possible?