Skip to main content

Posts about json

json in python

dictionary

add key:value pair to a dictionary

>>> newdict = {}
>>> newdict['foo'] = "bar"
>>> newdict
{'foo': 'bar'}

get all keys or values of a dictionary

>>> newdict.keys()
['foo']
>>> newdict.values()
['bar']

get a value for a key from a dictionary

>>> newdict['foo']
'bar'
>>> newdict.get("foo")
'bar'

add a dictionary as a value

>>> newdict0 = {}
>>> newdict0["mykey"] = newdict
>>> newdict0
{'mykey': {'foo': 'bar'}}

get a value

>>> newdict0['mykey']['foo']
'bar'

add a list as a value

>>> newdict0["mykey"] = [newdict, {'hoge': 'piyo'}]
>>> newdict0
{'mykey': [{'foo': 'bar'}, {'hoge': 'piyo'}]}

get a list

>>> for dict in newdict0.get('mykey'):
...     print dict
...
{'foo': 'bar'}
{'hoge': 'piyo'}

json

dump dictionary as json format string

>>> import json
>>> mystr = json.dumps(newdict)
>>> mystr
'{"foo": "bar"}'

load json format string to dictionary

>>> mydict = json.loads(mystr)
>>> mydict
{u'foo': u'bar'}
>>> mydict.get("foo")
u'bar'

To show non-ASCII characters, we can use ensure_ascii=False option, otherwise the output are escaped with \uXXXX sequences

$ python26
>>> import json
>>> mystr = '{"foo":"ばぁ"}'
>>> mydict = json.loads(mystr)
>>>
>>> print(json.dumps(mydict))
{"foo": "\u3070\u3041"}
>>>
>>> print(json.dumps(mydict, ensure_ascii=False))
{"foo": "ばぁ"}
>>>

If locale is not UTF-8, we will met UnicodeEncodeError when it prints UTF-8 strings to stdout. In this case, we can use encode() to avoid the error.

$ LANG=C python26
>>> import json
>>> mystr = '{"foo":"ばぁ"}'
>>> mydict = json.loads(mystr)
>>>
>>> print(json.dumps(mydict))
{"foo": "\u3070\u3041"}
>>>
>>> print(json.dumps(mydict, ensure_ascii=False))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 14-15: ordinal not in range(128)
>>>
>>> print(json.dumps(mydict, ensure_ascii=False).encode('utf-8'))
{"foo": "ばぁ"}
>>>

write json to a file

mystr = '{"foo":"ばぁ"}
mydict = json.loads(mystr)

with open('new.json', 'w') as f:
    json.dump(mydict, f)

read json from a file

with open('new.json') as f:
    myjson = json.load(f)
    print(json.dumps(myjson, ensure_ascii=False).encode('utf-8'))

sample script

using urllib2 library

similar as above but using reauests library

aws

query with JMESPath

projection

$ aws ec2 describe-vpcs --query 'Vpcs[].VpcId'
$ aws ec2 describe-vpcs --query 'Vpcs[].{VpcId:VpcId, IsDefault:IsDefault}'

selection (if the targe command has filters option, it would be faster than query.)

$ aws ec2 describe-vpcs --query 'Vpcs[?IsDefault == `true`]'
$ aws ec2 describe-vpcs --filters 'Name=isDefault,Values=true'

$ aws ec2 describe-vpcs --query 'Vpcs[?Tags[?Key == `Name` && Value == `test`]]'
$ aws ec2 describe-vpcs --filters 'Name=tag:Name,Values=test'

function

$ aws ec2 describe-vpcs --query 'Vpcs[?contains(VpcId, `vpc`)].VpcId'

$ aws ec2 describe-vpcs --query 'sort_by(Vpcs[?contains(VpcId, `vpc`)].VpcId, &VpcId)'
$ aws ec2 describe-images --filters "Name=owner-id,Values=<id>" --query "sort_by(Images[].{Name:Name, ImageId:ImageId}, &Name)"

$ aws ec2 describe-vpcs --query 'length(Vpcs[?contains(VpcId, `vpc`)])'

generate-cli-skeleton output

$ aws ec2 describe-vpcs --generate-cli-skeleton output
$ aws ec2 describe-vpcs --generate-cli-skeleton output --query 'Vpcs[].{CidrBlock:CidrBlock, VpcId:VpcId}'
$ aws ec2 describe-vpcs --query 'Vpcs[].{CidrBlock:CidrBlock, VpcId:VpcId}'

generate-cli-skeleton input

$ aws ec2 describe-vpcs --generate-cli-skeleton input | tee describe-vpcs.json
$ vi describe-vpcs.json
$ jq . describe-vpcs.json
{
  "Filters": [
    {
      "Name": "tag:Name",
      "Values": [
        "test"
      ]
    }
  ]
}
$ aws ec2 describe-vpcs --cli-input-json file://describe-vpcs.json

samples of filter

$ aws ec2 describe-vpcs --filters '["Name":"isDefault","Values":["true"]]'
$ aws ec2 describe-vpcs --filters 'Name=isDefault,Values=true'

$ aws ec2 describe-vpcs --filters '[{"Name":"tag-key","Values":["aws:cloudformation:stack-name"]}]'
$ aws ec2 describe-vpcs --filters 'Name=tag-key,Values=aws:cloudformation:stack-name'

$ aws ec2 describe-vpcs --filters '[{"Name":"tag:Name","Values":["terraform_test","mystack-VPC"]}]'
$ aws ec2 describe-vpcs --filters 'Name=tag:Name,Values=terraform_test,mystack-VPC'

$ aws ec2 describe-vpcs --filters '[{"Name":"isDefault","Values":["false"]}, {"Name":"state","Values":["available"]}]'
$ aws ec2 describe-vpcs --filters 'Name=isDefault,Values=true','Name=state,Values=available'

json

json(JavaScript Object Notation)

  • object is unordered set of name/value pair.
  • array is ordered set of values
  • value can be number, string, boolean(true or false), null, object or array.

    { "number": 128, "string": "128", "boolean": true, "null": null, "object": { "key0": "value0" }, "array": [ 128, "128", true, null, { "key1": "value1" } ] }

jq

jq is command line json processor. How to install jq in debian.

apt-get install jq

Here we use result of below command as a sample json file. Of cource I know awscli has convinience options like filter and query.

aws ec2 describe-images \
--filters '[{"Name":"owner-id","Values":["136693071363"]},{"Name":"architecture","Values":["x86_64"]}]' > ami.json

内容全てを整形して出力

$ jq '.' ami.json

Images の子要素(配列)を全て出力

$ jq '.Images[]' amis.json

Images の子要素のうち Description が Test のものの内容を出力

$ jq '.Images[] | select(.Description == "Test")' ami.json

Not condition

$ jq '.Images[] | select(.Description == "Test" | not)' ami.json

Startswith

$ jq '.Images[] | select(.Description | startswith("Debian 10"))' ami.json

contains

$ jq '.Images[] | select(.Description | contains("202005"))' ami.json

Images の子要素のうち Description が Test のもののNameの値を出力

$ jq '.Images[] | select(.Description == "Test") | .Name' ami.json

Images の子要素のうち Description が Test のものの内容を整形して出力

$ jq '.Images[] | select(.Description == "Test") | {"MyName":.Name, "MyDate":.CreationDate}' ami.json

jqのコマンドをファイルから読み込むことができる。

$ cat ami.jq
.Images[] |
select(.Description == "Test") |
{
  "MyName":.Name,
  "MyDate":.CreationDate
}

$ jq -f ami.jq ami.json

シェバンを書けば、sed や awk のようにスクリプトとして実行することができる。

$ cat ami.jq
#! /usr/bin/jq -f
.Images[] |
select(.Description == "Test") |
{
  "MyName":.Name,
  "MyDate":.CreationDate
}

$ chmod +x ami.jq

$ ./ami.jq ami.json

adding value of add.json to ResourceRecordSet part of orig.json

$ jq . orig.json
{
  "Comment": "delete after certbot validation",
  "Changes": [
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "Name": "_acme-challenge.'${CERTBOT_DOMAIN}'.",
        "Type": "TXT"
      }
    }
  ]
}
$ jq . add.json 
{
  "ResourceRecords": [
    {
      "Value": "\"abcd1234\""
    }
  ],
  "TTL": 1800
}
$ jq '.Changes[0].ResourceRecordSet |= .+ '$(jq -c . add.json)'' orig.json
{
  "Comment": "delete after certbot validation",
  "Changes": [
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "Name": "_acme-challenge.'${CERTBOT_DOMAIN}'.",
        "Type": "TXT",
        "ResourceRecords": [
          {
            "Value": "\"abcd1234\""
          }
        ],
        "TTL": 1800
      }
    }
  ]
}