パンダ：JSONファイルからネストされたデータを取得する

Question

hereからネストされたJSONデータを解析しています。このファイル内の一部のファイルには、複数のファイルが関連付けられています。各ファイルに関連するすべての委員会が必要です。私は確信していませんが、それはそれぞれcommittee_idの新しい行を書くことを意味すると思います。私のコードは次のとおりです。

import os.path 
import csv 
import json 

path = '/home/jayaramdas/anaconda3/Thesis/govtrack/bills109/hr' 
dirs = os.listdir(path) 
outputfile = open('df/h109_s_b', 'w', newline='')        
outputwriter = csv.writer(outputfile) 

for dir in dirs: 
    with open(path + "/" + dir + "/data.json", "r") as f: 
     data = json.load(f) 

     a = data['introduced_at'] 
     b = data['bill_id'] 
     c = data['sponsor']['thomas_id'] 
     d = data['sponsor']['state'] 
     e = data['sponsor']['name'] 
     f = data['sponsor']['type'] 
     i = data['subjects_top_term'] 
     j = data['official_title']    

     if data['committees']: 
      g = data['committees'][0]['committee_id'] 
     else: 
      g = "None"      
    outputwriter.writerow([a, b, c, d, e, f, g, i, j]) 
outputfile.close()  

を私が午前問題は、私のコードのみ記載されている最初のcommittee_idを収集しているということです。たとえば、ファイルは次のようになります。

bill_iid committee subcommittee introduced at Thomas_id state name 
hr145-109 HSPW   na    "2005-01-4"   73  NY "McHugh, John M." 
hr145-109 HSPW   13    "2005-01-4"   73  NY "McHugh, John M." 
hr145-109 HSBA   na    "2005-01-4"   73  NY "McHugh, John M." 
hr145-109 HSBA   19    "2005-01-4"   73  NY "McHugh, John M." 

：法案は、小委員会に渡されるとき、私も committee_idに関連付けられている subcommittee_idをしたいので、それは少しトリッキーです

"committees": [ 
{ 
    "activity": [ 
    "referral", 
    "in committee" 
    ], 
    "committee": "House Transportation and Infrastructure", 
    "committee_id": "HSPW" 
}, 
{ 
    "activity": [ 
    "referral" 
    ], 
    "committee": "House Transportation and Infrastructure", 
    "committee_id": "HSPW", 
    "subcommittee": "Subcommittee on Economic Development, Public Buildings and Emergency Management", 
    "subcommittee_id": "13" 
}, 
{ 
    "activity": [ 
    "referral", 
    "in committee" 
    ], 
    "committee": "House Financial Services", 
    "committee_id": "HSBA" 
}, 
{ 
    "activity": [ 
    "referral" 
    ], 
    "committee": "House Financial Services", 


    "committee_id": "HSBA", 
    "subcommittee": "Subcommittee on Domestic and International Monetary Policy, Trade, and Technology", 
    "subcommittee_id": "19" 
} 

これは、

アイデア？

Answer 1

あなたはこのようにそれを行うことができます。

In [111]: with open(fn) as f: 
    .....:  data = ujson.load(f) 
    .....: 

In [112]: committees = pd.io.json.json_normalize(data, 'committees') 

In [113]: committees 
Out[113]: 
      activity        committee committee_id       subcommittee subcommittee_id 
0   [referral]    House Energy and Commerce   HSIF          NaN    NaN 
1   [referral]    House Energy and Commerce   HSIF Subcommittee on Energy and Air Quality    03 
2   [referral]  House Education and the Workforce   HSED          NaN    NaN 
3   [referral]     House Financial Services   HSBA          NaN    NaN 
4   [referral]      House Agriculture   HSAG          NaN    NaN 
5 [referral, markup]       House Resources   HSII          NaN    NaN 
6   [referral]       House Science   HSSY          NaN    NaN 
7   [referral]      House Ways and Means   HSWM          NaN    NaN 
8   [referral] House Transportation and Infrastructure   HSPW          NaN    NaN 

UPDATE：

あなたは1 DF内のすべてのデータを持っているしたい場合は、このようにそれを行うことができます。

import os 
import ujson 
import pandas as pd 

start_path = '/home/jayaramdas/anaconda3/Thesis/govtrack/bills109/hr' 

def get_merged_json(start_path): 
    return [ujson.load(open(os.path.join(path, f))) 
      for p, _, files in os.walk(start_path) 
      for f in files 
      if f.endswith('.json') 
      ] 

df = pd.read_json(ujson.dumps(data)) 

PS committeesを1列にJSONデータとして入れます。