2
私の仕事が失敗した理由を理解するために助けが必要です。私はそれを試してみるためにただ一つの ノードクラスタを構築しました。私は例hereに従った。Hadoopの単語カウントが実行できません - hadoopエラーメッセージのデコードに助けが必要です
すべて正常に動作しているようです。私はnamenodeをフォーマットし、ウェブインターフェースを介してjobtracker、datanode、namenodeに接続することができました。私はすべてのhadoopサービスを開始して停止することができます。私はwordcount例を実行しようとすると、
はしかし、私はこれを取得：
Error initializing attempt_201105161023_0002_m_000011_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
私は、ローカルファイルシステム上のディレクトリを作成しました。
$ sudo mkdir /app/hadoop/tmp
$ sudo chown hadoop:hadoop /app/hadoop/tmp
私もcore-site.xmlを修正：
<!-- In: conf/core-site.xml -->
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
私は名前ノードをフォーマットすると、それは地元の に成功HDFSの両方にサブディレクトリを作成します。私は失敗したwordcount出力結果を見てみると
、エラー メッセージがI/Oエラーを訴え、ファイル /app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/ job_201105161023_0002/jobToken
に私はいくつかのトラブルシューティングを行なったし、 にこのjobTokenファイルを参照することができますローカルファイルシステム、問題ありません。内容は次のようなものですHDTS MapReduce.job 201105161023_0002
許可の問題ですか？私はhadoopプロセスのユーザーにすべてのサブディレクトリに書き込むことができ、ファイルを作成することができました。だから何が間違っているのでしょうか？
コマンドラインからいくつかのより詳細な出力：
[email protected]:/usr/local/hadoop/hadoop-0.20.203.0$ bin/hadoop jar hadoop-examples-0.20.203.0.jar wordcount app/download app/output4
11/05/16 13:38:56 INFO input.FileInputFormat: Total input paths to process : 3
11/05/16 13:39:05 INFO mapred.JobClient: Running job: job_201105161222_0003
11/05/16 13:39:06 INFO mapred.JobClient: map 0% reduce 0%
11/05/16 13:39:17 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stdout
11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stderr
11/05/16 13:39:22 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stdout
11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stderr
11/05/16 13:39:23 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stdout
11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stderr
11/05/16 13:39:30 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stdout
11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stderr
11/05/16 13:39:33 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stdout
11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stderr
11/05/16 13:39:36 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stdout
11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stderr
11/05/16 13:39:39 INFO mapred.JobClient: Job complete: job_201105161222_0003
11/05/16 13:39:39 INFO mapred.JobClient: Counters: 4
11/05/16 13:39:39 INFO mapred.JobClient: Job Counters
11/05/16 13:39:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=0
11/05/16 13:39:39 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0