spark sqlでjavaを使用して複雑なjsonを読み取る

Question

私のjsonファイルは以下のようなものです。以下のコードを使用してmajorsector_percentの下のすべてのNameを読み込もうとしています。

コード：

JavaSQLContext sQLContext = new JavaSQLContext(sc); 
    sQLContext.jsonFile("C:/Users/HimanshuK/Downloads/world_bank/world_bank.json").registerTempTable("logs"); 
    sQLContext.sqlContext().cacheTable("logs"); 
    List s = sQLContext.sql("select majorsector_percent from logs limit 1 ").map(row -> new Tuple2<>(row.getString(0), row.getString(1))).collect(); 


    JSON FIle 

    { "_id" : { "$oid" : "52b213b38594d8a2be17c780" }, "approvalfy" : 1999, "board_approval_month" : "November", "boardapprovaldate" : "2013-11-12T00:00:00Z", "borrower" : "FEDERAL DEMOCRATIC REPUBLIC OF ETHIOPIA", "closingdate" : "2018-07-07T00:00:00Z", "country_namecode" : "Federal Democratic Republic of Ethiopia!$!ET", "countrycode" : "ET", "countryname" : "Federal Democratic Republic of Ethiopia", "countryshortname" : "Ethiopia", "docty" : "Project Information Document,Indigenous Peoples Plan,Project Information Document", "envassesmentcategorycode" : "C", "grantamt" : 0, "ibrdcommamt" : 0, "id" : "P129828", "idacommamt" : 130000000, "impagency" : "MINISTRY OF EDUCATION", "lendinginstr" : "Investment Project Financing", "lendinginstrtype" : "IN", "lendprojectcost" : 550000000, "majorsector_percent" : [ { "Name" : "Education", "Percent" : 46 }, { "Name" : "Education", "Percent" : 26 }, { "Name" : "Public Administration, Law, and Justice", "Percent" : 16 }, { "Name" : "Education", "Percent" : 12 } ], "mjtheme" : [ "Human development" ], "mjtheme_namecode" : [ { "name" : "Human development", "code" : "8" }, { "name" : "", "code" : "11" } ], "mjthemecode" : "8,11", "prodline" : "PE", "prodlinetext" : "IBRD/IDA", "productlinetype" : "L", "project_abstract" : { "cdata" : "The development }, "project_name" : "Ethiopia General Education Quality Improvement Project II", "projectfinancialtype" : "IDA", "projectstatusdisplay" : "Active", "regionname" : "Africa", "sector1" : { "Name" : "Primary education", "Percent" : 46 }, "sector2" : { "Name" : "Secondary education", "Percent" : 26 }, "sector3" : { "Name" : "Public administration- Other social services", "Percent" : 16 }, "sector4" : { "Name" : "Tertiary education", "Percent" : 12 }, "sectorcode" : "ET,BS,ES,EP", "source" : "IBRD", "status" : "Active", "supplementprojectflg" : "N", "theme1" : { "Name" : "Education for all", "Percent" : 100 }, "themecode" : "65", "totalamt" : 130000000, "totalcommamt" : 130000000, "url" : "http://www.worldbank.org/projects/P129828/ethiopia-general-education-quality-improvement-project-ii?lang=en" } 

が、私はこのような場合の処理​​方法とスキーマを知っているか、理由は型キャストのこのエラーを取得しています：

とjava.lang.ClassCastException：scala.collection.mutableを.ArrayBufferはjava.lang.Stringにキャストできません

Answer 1

問題は、そのクエリの結果が配列を含む構造体であることです。配列にrow.getString(1)を使用して結果をマップしようとすると、対応するオブジェクトが文字列ではないため、CastExceptionで失敗します。

SQLクエリの結果がデータフレームである、あなたは（あなたは、Java APIで同じコマンドを使用することができます）このようなスキーマを求めることができます：

scala> val data = sqlContext.sql("select majorsector_percent from logs limit 1 ") 
data: org.apache.spark.sql.DataFrame = [majorsector_percent: array<struct<Name:string,Percent:bigint>>] 

scala> data.printSchema 
root 
|-- majorsector_percent: array (nullable = true) 
| |-- element: struct (containsNull = true) 
| | |-- Name: string (nullable = true) 
| | |-- Percent: long (nullable = true) 

あなたはから必要な情報を抽出することができそのような結果のデータフレーム：

data.select("majorsector_percent.Name","majorsector_percent.Percent") 

scala> data.select("majorsector_percent.Name","majorsector_percent.Percent").collect 
res4: Array[org.apache.spark.sql.Row] = Array([WrappedArray(Education, Education, Public Administration, Law, and Justice, Education),WrappedArray(46, 26, 16, 12)]) 

それとも、で開始する、より具体的なクエリを使用することにより、プロセスを簡素化することができます：

val directQuery = sqlContext.sql("select majorsector_percent.Name, majorsector_percent.Percent from logs limit 1 ") 
directQuery: org.apache.spark.sql.DataFrame = [Name: array<string>, Percent: array<bigint>] 

scala> directQuery.collect 
res5: Array[org.apache.spark.sql.Row] = Array([WrappedArray(Education, Education, Public Administration, Law, and Justice, Education),WrappedArray(46, 26, 16, 12)])