1
aws s3バケットにデータを書き込むためのflume sinkがあります。私は私の水路・プロセスを開始すると、私はまた、以下のjarファイルec2インスタンスIAMクレデンシャルを使用してflume sinkをs3に転送
commons-codec-1.4.jar
commons-httpclient-3.1.jar
hadoop-common-2.7.1.jar
commons-configuration-1.6.jar
hadoop-auth-2.7.1.jar
aws-java-sdk-1.10.64.jar
aws-java-sdk-s3-1.10.52.jar
hadoop-aws.jar
emr-core-2.4.0.jar
を含ま
aggregator.sinks.s3LogSink.type=hdfs
aggregator.sinks.s3LogSink.channel=flumeLogAgentFileChannel
aggregator.sinks.s3LogSink.hdfs.path=s3n://<aws bucket name>
aggregator.sinks.s3LogSink.hdfs.fileType=DataStream
aggregator.sinks.s3LogSink.hdfs.writeFormat=Text
aggregator.sinks.s3LogSink.hdfs.rollCount = 0
aggregator.sinks.s3LogSink.hdfs.rollSize = 67108864
aggregator.sinks.s3LogSink.hdfs.batchSize = 1000
aggregator.sinks.s3LogSink.hdfs.rollInterval = 0
以下のように水路S3シンクをサポートするために、 水路の設定がある..私は以下のエラーを取得
2016-04-18 18:03:06,326 (hdfs-s3LogSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-04-18 18:03:06,359 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:224)
at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2623)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/event/ProgressListener
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
at java.lang.Class.getConstructor0(Class.java:2895)
at java.lang.Class.newInstance(Class.java:354)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
... 18 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 23 more
何人かお手伝いできますか？