分散テンソルフローサポートフェールオーバーを作成するにはどうすればよいですか？

Question

私は4ノードのテンソルフロークラスタを作成します.2人の作業者は2 psです。ワーカーやpsが失敗すると、マシン上の同じ設定でそれを再起動します。ただし、チェックポイントから引き続き作業することはできません。 分散型テンソルフローはまだフェールオーバーをサポートしていませんか？

このファイル "model.ckpt-24"はマシン（作業者のタスク0）上にのみあり、トレースは次のとおりです。

私は、4ノードのテンソルフロークラスタ、2人の作業者、2psを作成します。ワーカーやpsが失敗すると、マシン上の同じ設定でそれを再起動します。ただし、チェックポイントから引き続き作業することはできません。 分散型テンソルフローはまだフェールオーバーをサポートしていませんか？

このファイル "model.ckpt-24"はマシン（作業者のタスク0）上にのみあり、トレースは次のとおりです。

Traceback (most recent call last): 
File "/dump/9/nm-local-dir/usercache/danrtsey.wy/appcache/application_1477899492621_0004/container_e07_1477899492621_0004_01_000003/app/install/trainer.py", line 107, in <module> 
tf.app.run() 
File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 30, in run 
sys.exit(main(sys.argv[:1] + flags_passthrough)) 
File "/dump/9/nm-local-dir/usercache/danrtsey.wy/appcache/application_1477899492621_0004/container_e07_1477899492621_0004_01_000003/app/install/trainer.py", line 87, in main 
with sv.managed_session(server.target) as sess: 
File "/usr/lib64/python2.7/contextlib.py", line 17, in enter 
return self.gen.next() 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 969, in managed_session 
self.stop(close_summary_writer=close_summary_writer) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 797, in stop 
stop_grace_period_secs=self._stop_grace_secs) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 386, in join 
six.reraise(*self._exc_info_to_raise) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 958, in managed_session 
start_standard_services=start_standard_services) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 715, in prepare_or_wait_for_session 
init_feed_dict=self._init_feed_dict, init_fn=self._init_fn) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 227, in prepare_session 
config=config) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 173, in _restore_checkpoint 
saver.restore(sess, ckpt.model_checkpoint_path) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1345, in restore 
{self.saver_def.filename_tensor_name: save_path}) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 717, in run 
run_metadata_ptr) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 915, in _run 
feed_dict_string, options, run_metadata) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _do_run 
target_list, options, run_metadata) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 985, in _do_call 
raise type(e)(node_def, op, message) 
tensorflow.python.framework.errors.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /dump/6/nm-logs/application_1477899492621_0004/container_e07_1477899492621_0004_01_000003/model.ckpt-24: Not found: /dump/6/nm-logs/application_1477899492621_0004/container_e07_1477899492621_0004_01_000003 
[[Node: save/restore_slice_1 = RestoreSlice[dt=DT_FLOAT, preferred_shard=-1, _device="/job:ps/replica:0/task:1/cpu:0"](_recv_save/Const_0_S3, save/restore_slice_1/tensor_name, save/restore_slice_1/shape_and_slice)]] 

Caused by op u'save/restore_slice_1', defined at: 
File "/dump/9/nm-local-dir/usercache/danrtsey.wy/appcache/application_1477899492621_0004/container_e07_1477899492621_0004_01_000003/app/install/trainer.py", line 107, in <module> 
tf.app.run() 
File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 30, in run 
sys.exit(main(sys.argv[:1] + flags_passthrough)) 
File "/dump/9/nm-local-dir/usercache/danrtsey.wy/appcache/application_1477899492621_0004/container_e07_1477899492621_0004_01_000003/app/install/trainer.py", line 71, in main 
saver = tf.train.Saver() 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 986, in init 
self.build() 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1015, in build 
restore_sequentially=self._restore_sequentially) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 620, in build 
restore_sequentially, reshape) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 357, in _AddRestoreOps 
tensors = self.restore_op(filename_tensor, saveable, preferred_shard) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 270, in restore_op 
preferred_shard=preferred_shard)) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/io_ops.py", line 204, in _restore_slice 
preferred_shard, name=name) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 359, in _restore_slice 
preferred_shard=preferred_shard, name=name) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op 
op_def=op_def) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2380, in create_op 
original_op=self._default_original_op, op_def=op_def) 
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1298, in init 
self._traceback = _extract_stack() 

InvalidArgumentError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to get matching files on /dump/6/nm-logs/application_1477899492621_0004/container_e07_1477899492621_0004_01_000003/model.ckpt-24: Not found: /dump/6/nm-logs/application_1477899492621_0004/container_e07_1477899492621_0004_01_000003 
[[Node: save/restore_slice_1 = RestoreSlice[dt=DT_FLOAT, preferred_shard=-1, _device="/job:ps/replica:0/task:1/cpu:0"](_recv_save/Const_0_S3, save/restore_slice_1/tensor_name, save/restore_slice_1/shape_and_slice)]] 

Answer 1

Tensorflowはチェックポイントから復元できます。 HDFSを使用してチェックポイントを保存するだけで、この問題が解決されます。