解決できなかったPigのUNIONに問題があります。同じスクリプトでいくつかの関係についてUNIONを実行するとうまくいくが、それ以外の関係ではうまくいきません。私はUNIONを2つの関係、すなわちone
とtwo
で実行しています。そして、これらの2つの関係と他の多くの関係の多くでは機能しません。ここにスクリプトがあります:Apache Pigの特定の関係でのみUNION操作を実行できません
A = LOAD '/home/biadmin/datasets/Datasets/HomeA/2014/homeA2.csv' USING PigStorage(',') AS (Date:chararray ,use:float, gen:float, FurnaceHRV:float, CellarOutlets:float, WashingMachine:float, FridgeRange:float, DisposalDishwasher:float, KitchenLights:float, BedroomOutlets:float, BedroomLights:float, MasterOutlets:float, MasterLights:float, DuctHeaterHRV:float);
C = FOREACH A GENERATE Date,SUM(TOBAG(use..)) AS total:float, '1388552400' AS unixconst1:chararray, '86400' AS dayconst1:chararray;
hourly = FILTER C BY ENDSWITH(Date, '00');
hourlymd = FOREACH hourly GENERATE *, SUBSTRING(Date,0,INDEXOF(Date,'/',0)) as month1:chararray, SUBSTRING(Date,INDEXOF(Date,'/',0)+1,LAST_INDEX_OF(Date,'/')) as day1:chararray, SUBSTRING(Date,LAST_INDEX_OF(Date,'/')+1,INDEXOF(Date,' ',0)) as year1:chararray, SUBSTRING(Date,INDEXOF(Date,' ',0)+1,INDEXOF(Date,':',0)) as hour1:chararray;
hourlymdB = FOREACH hourlymd GENERATE (int)(hour1) AS hour:int, (int)(day1) AS day:int, (int)(month1) AS month:int, (int)(year1) AS year:int, (int)(unixconst1) AS unixconst:int, (int)(dayconst1) AS dayconst:int;
SPLIT hourlymdB INTO
one IF(month==1),
two IF(month==2),
three IF(month==3),
four IF(month==4),
five IF(month==5),
six IF(month==6),
seven IF(month==7),
eight IF(month==8),
nine IF(month==9),
ten IF(month==10),
eleven IF(month==11),
twelve IF(month==12),
rest OTHERWISE;
yearone = UNION one, two;
STORE yearone INTO '/home/biadmin/datasets/output/yearone';
このコードは機能しません、エラーはありません。ここで私はそれを実行するときの出力です。しかし
Pig Stack Trace
---------------
ERROR 0: java.lang.NullPointerException
org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.lang.NullPointerException
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:283)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
at org.apache.pig.PigServer.execute(PigServer.java:1341)
at org.apache.pig.PigServer.executeBatch(PigServer.java:392)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:608)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.Job$4.run(Job.java:963)
at org.apache.hadoop.mapreduce.Job$4.run(Job.java:961)
at java.security.AccessController.doPrivileged(AccessController.java:366)
at javax.security.auth.Subject.doAs(Subject.java:572)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
at org.apache.hadoop.mapreduce.Job.getTaskReports(Job.java:961)
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getTaskReports(HadoopShims.java:218)
at org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addMapReduceStatistics(MRJobStats.java:353)
at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:233)
at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:165)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:364)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277)
... 16 more
================================================================================
、A
とC
にUNIONを実行し、書く場合:
yearone = UNION A, C;
そして、それは問題なく正常に動作し、移動先での関係を格納します。なぜ私は理解できませんか?私はそれをコマンドを使ってgrunt>
(ローカルモード)のシェルと外部の両方で実行しました。
ご協力いただきまして誠にありがとうございます。
あなたは正しいです。これは誤った構文です。私は 'UNION one、two;'に変更しました。繰り返しますが、UNIONを行うことができる唯一の関係は、「A」、「C」、「時間単位」です。 –
他の関係にレコードがあるかどうか確認してください。 –