糸クラスター（Linux）のクライアント（ウィンドウ）からスパークを実行する：Filesystem "C"のエラーはありません

Question

スパークアプリケーション（単純なHello Worldアプリケーション）をhadoopクラスターにデプロイします。 私のWindowsマシンでspark submitを使う私は、 - master糸でクライアントモードでアプリケーションを実行します。 hadoopクラスタへの接続は、クラスタ上のログファイルに表示されるため、成功しています。 （hadoopのconfファイルはクラスタからダウンロードされ、クライアントのWindowsマシンに保存され、環境変数が設定されています）。

16/08/10 10:52:45 ERROR SparkContext: Error initializing SparkContext. 
java.io.IOException: No FileSystem for scheme: C 

完全なエラーメッセージ：

>spark-submit --master yarn --class "SimpleApp" ..\..\SimpleApp\target\scala-2.11\simple-project_2.11-1.0.jar --deploy-mode client 

これは、返されたエラーメッセージがある：Hadoopの2.7と火花を使用

は

1.6

これは、使用されるスパーク送信コマンドであります

Warning: Ignoring non-spark config property: fs.hdfs.impl=org.apache.hadoop.hdfs.DistributedFileSystem 
16/08/10 10:52:32 INFO SparkContext: Running Spark version 1.6.2 
16/08/10 10:52:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/08/10 10:52:32 INFO SecurityManager: Changing view acls to: sensored,sensored 
16/08/10 10:52:32 INFO SecurityManager: Changing modify acls to: sensored,sensored 
16/08/10 10:52:32 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sensored,sensored); users with modify permissions: Set(sensored,sensored) 
16/08/10 10:52:33 INFO Utils: Successfully started service 'sparkDriver' on port 52340. 
16/08/10 10:52:33 INFO Slf4jLogger: Slf4jLogger started 
16/08/10 10:52:33 INFO Remoting: Starting remoting 
16/08/10 10:52:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@sensored,sensored] 
16/08/10 10:52:34 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 52353. 
16/08/10 10:52:34 INFO SparkEnv: Registering MapOutputTracker 
16/08/10 10:52:34 INFO SparkEnv: Registering BlockManagerMaster 
16/08/10 10:52:34 INFO DiskBlockManager: Created local directory at C:\Users\sensored,sensored\AppData\Local\Temp\blockmgr-35b65f9a-47a3-4d1e-940f-be8bc9c78b8f 
16/08/10 10:52:34 INFO MemoryStore: MemoryStore started with capacity 511.1 MB 
16/08/10 10:52:34 INFO SparkEnv: Registering OutputCommitCoordinator 
16/08/10 10:52:34 INFO Utils: Successfully started service 'SparkUI' on port 4040. 
16/08/10 10:52:34 INFO SparkUI: Started SparkUI at http://10.61.156.198:4040 
16/08/10 10:52:34 INFO HttpFileServer: HTTP File server directory is C:\Users\sensored,sensored\AppData\Local\Temp\spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e\httpd-e39c2eda-b9aa-40df-95c9-eae44b53c4ee 
16/08/10 10:52:34 INFO HttpServer: Starting HTTP Server 
16/08/10 10:52:34 INFO Utils: Successfully started service 'HTTP file server' on port 52356. 
16/08/10 10:52:34 INFO SparkContext: Added JAR file:/C:/Users/sensored,sensored/Documents/spark-1.6.2-bin-hadoop2.6/bin/../../SimpleApp/target/scala-2.11/simple-project_2.11-1.0.jar at http://sensored,sensored:52356/jars 
/simple-project_2.11-1.0.jar with timestamp 1470819154823 
16/08/10 10:52:35 INFO TimelineClientImpl: Timeline service address: http://sensored,sensored:8188/ws/v1/timeline/ 
16/08/10 10:52:35 INFO RMProxy: Connecting to ResourceManager at sensored,sensored/sensored,sensored.188:8050 
16/08/10 10:52:36 INFO Client: Requesting a new application from cluster with 2 NodeManagers 
16/08/10 10:52:36 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (221184 MB per container) 
16/08/10 10:52:36 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 
16/08/10 10:52:36 INFO Client: Setting up container launch context for our AM 
16/08/10 10:52:36 INFO Client: Setting up the launch environment for our AM container 
16/08/10 10:52:36 INFO Client: Preparing resources for our AM container 
16/08/10 10:52:36 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows. 
16/08/10 10:52:36 INFO Client: Uploading resource file:/C:/Users/sensored,sensored/Documents/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar -> hdfs://sensored,sensored:8020/user/sensored,sensored/.sparkSt 
aging/application_1468399950865_0375/spark-assembly-1.6.2-hadoop2.6.0.jar 
16/08/10 10:52:41 INFO Client: Uploading resource file:/C:/Users/sensored,sensored/AppData/Local/Temp/spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e/__spark_conf__7713144618669536696.zip -> hdfs://sensored,sensored:80 
20/user/sensored,sensored/.sparkStaging/application_1468399950865_0375/__spark_conf__7713144618669536696.zip 
16/08/10 10:52:41 INFO SecurityManager: Changing view acls to: sensored,sensored 
16/08/10 10:52:41 INFO SecurityManager: Changing modify acls to: sensored,sensored 
16/08/10 10:52:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sensored,sensored); users with modify permissions: Set(sensored,sensored) 
16/08/10 10:52:41 INFO Client: Submitting application 375 to ResourceManager 
16/08/10 10:52:41 INFO YarnClientImpl: Submitted application application_1468399950865_0375 
16/08/10 10:52:42 INFO Client: Application report for application_1468399950865_0375 (state: ACCEPTED) 
16/08/10 10:52:42 INFO Client: 
     client token: N/A 
     diagnostics: N/A 
     ApplicationMaster host: N/A 
     ApplicationMaster RPC port: -1 
     queue: default 
     start time: 1470819161610 
     final status: UNDEFINED 
     tracking URL: http://sensored,sensored:8088/proxy/application_1468399950865_0375/ 
     user: sensored,sensored 
16/08/10 10:52:43 INFO Client: Application report for application_1468399950865_0375 (state: ACCEPTED) 
16/08/10 10:52:44 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null) 
16/08/10 10:52:44 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> sensored,sensored, PROXY_URI_BASES -> http://sensored,sensored:8088/proxy/application_1468399950865_0375), /proxy/application_1468399950865_0375 
16/08/10 10:52:44 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 
16/08/10 10:52:44 INFO Client: Application report for application_1468399950865_0375 (state: RUNNING) 
16/08/10 10:52:44 INFO Client: 
     client token: N/A 
     diagnostics: N/A 
     ApplicationMaster host: 10.3.131.188 
     ApplicationMaster RPC port: 0 
     queue: default 
     start time: 1470819161610 
     final status: UNDEFINED 
     tracking URL: http://sensored,sensored:8088/proxy/application_1468399950865_0375/ 
     user: sensored,sensored 
16/08/10 10:52:44 INFO YarnClientSchedulerBackend: Application application_1468399950865_0375 has started running. 
16/08/10 10:52:44 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52383. 
16/08/10 10:52:44 INFO NettyBlockTransferService: Server created on 52383 
16/08/10 10:52:44 INFO BlockManagerMaster: Trying to register BlockManager 
16/08/10 10:52:44 INFO BlockManagerMasterEndpoint: Registering block manager 10.61.156.198:52383 with 511.1 MB RAM, BlockManagerId(driver, sensored,sensored, 52383) 
16/08/10 10:52:44 INFO BlockManagerMaster: Registered BlockManager 
16/08/10 10:52:45 ERROR SparkContext: Error initializing SparkContext. 
java.io.IOException: No FileSystem for scheme: C 
     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) 
     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) 
     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) 
     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) 
     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) 
     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) 
     at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1686) 
     at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:66) 
     at org.apache.spark.SparkContext.<init>(SparkContext.scala:547) 
     at SimpleApp$.main(SimpleApp.scala:10) 
     at SimpleApp.main(SimpleApp.scala) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:498) 
     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) 
     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) 
     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) 
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) 
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
16/08/10 10:52:45 INFO SparkUI: Stopped Spark web UI at http://10.61.156.198:4040 
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Shutting down all executors 
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Interrupting monitor thread 
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Asking each executor to shut down 
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Stopped 
16/08/10 10:52:45 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 
16/08/10 10:52:45 INFO MemoryStore: MemoryStore cleared 
16/08/10 10:52:45 INFO BlockManager: BlockManager stopped 
16/08/10 10:52:45 INFO BlockManagerMaster: BlockManagerMaster stopped 
16/08/10 10:52:45 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 
16/08/10 10:52:45 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 
16/08/10 10:52:45 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 
16/08/10 10:52:45 INFO SparkContext: Successfully stopped SparkContext 
Exception in thread "main" java.io.IOException: No FileSystem for scheme: C 
     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) 
     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) 
     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) 
     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) 
     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) 
     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) 
     at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1686) 
     at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:66) 
     at org.apache.spark.SparkContext.<init>(SparkContext.scala:547) 
     at SimpleApp$.main(SimpleApp.scala:10) 
     at SimpleApp.main(SimpleApp.scala) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:498) 
     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) 
     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) 
     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) 
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) 
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
16/08/10 10:52:45 INFO ShutdownHookManager: Shutdown hook called 
16/08/10 10:52:45 INFO ShutdownHookManager: Deleting directory C:\Users\sensored,sensored\AppData\Local\Temp\spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e\httpd-e39c2eda-b9aa-40df-95c9-eae44b53c4ee 
16/08/10 10:52:45 INFO ShutdownHookManager: Deleting directory C:\Users\sensored,sensored\AppData\Local\Temp\spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e 

Answer 1

スパーク2.0に切り替えた後、問題は発生しなくなりました！