0
単一のマシン上の異なるディレクトリからログをローカルファイルシステムファイルまたはHDFSに収集しようとしています。Flume：単一のシンクにログを追加する複数のソース
私は2つのソースr1、r2を登録しました。 両方のソースが単一チャネルC1を指しています。 チャネルに1つのシンクが接続されています。 K1
以下の設定ファイルを見つけるしてください：
# Name the components on this agent
a1.sources = r1
a1.sources = r2
a1.sinks = k1
a1.channels = c1
a1.sources.r2.type = exec
a1.sources.r2.command = tail -f /PATH/bper-peg-pt-rest.log
a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /PATH/bper-peg-ejb.log
# Describe the sink
a1.sinks.k1.type = file_roll
a1.sinks.k1.sink.directory = /home/vbsc/Desktop/flume_project_logging/logs_aggregated
a1.sinks.k1.sink.rollInterval = 0
# Use file channel
a1.channels.c1.type = file
# Bind the source and sink to the channel
a1.sinks.k1.channel = c1
a1.sources.r2.channels = c1
a1.sources.r1.channels = c1
をしかし、私はエージェントA1と水路を起動したときに、唯一のソース（R2）が入門しています。 水路エージェントの起動ログ：私は以下のように2つのソースを宣言する必要があり
16/06/14 14:38:09 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
16/06/14 14:38:09 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/vbsc/Desktop/flume_project_logging/flume_tailSource.conf
16/06/14 14:38:09 INFO conf.FlumeConfiguration: Processing:k1
16/06/14 14:38:09 INFO conf.FlumeConfiguration: Added sinks: k1 Agent: a1
16/06/14 14:38:09 INFO conf.FlumeConfiguration: Processing:k1
16/06/14 14:38:09 INFO conf.FlumeConfiguration: Processing:k1
16/06/14 14:38:09 INFO conf.FlumeConfiguration: Processing:k1
16/06/14 14:38:09 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [a1]
16/06/14 14:38:09 INFO node.AbstractConfigurationProvider: Creating channels
16/06/14 14:38:09 INFO channel.DefaultChannelFactory: Creating instance of channel c1 type file
16/06/14 14:38:10 INFO node.AbstractConfigurationProvider: Created channel c1
16/06/14 14:38:10 INFO source.DefaultSourceFactory: Creating instance of source r2, type exec
16/06/14 14:38:10 INFO sink.DefaultSinkFactory: Creating instance of sink: k1, type: file_roll
16/06/14 14:38:10 INFO node.AbstractConfigurationProvider: Channel c1 connected to [r2, k1]
16/06/14 14:38:10 INFO node.Application: Starting new configuration:{ sourceRunners:{r2=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:r2,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:[email protected] counterGroup:{ name:null counters:{} } }} channels:{c1=FileChannel c1 { dataDirs: [/root/.flume/file-channel/data] }} }
16/06/14 14:38:10 INFO node.Application: Starting Channel c1
16/06/14 14:38:10 INFO file.FileChannel: Starting FileChannel c1 { dataDirs: [/root/.flume/file-channel/data] }...
16/06/14 14:38:10 INFO file.Log: Encryption is not enabled
16/06/14 14:38:10 INFO file.Log: Replay started
16/06/14 14:38:10 INFO file.Log: Found NextFileID 13, from [/root/.flume/file-channel/data/log-9, /root/.flume/file-channel/data/log-11, /root/.flume/file-channel/data/log-13, /root/.flume/file-channel/data/log-12, /root/.flume/file-channel/data/log-10]
16/06/14 14:38:10 INFO file.EventQueueBackingStoreFileV3: Starting up with /root/.flume/file-channel/checkpoint/checkpoint and /root/.flume/file-channel/checkpoint/checkpoint.meta
16/06/14 14:38:10 INFO file.EventQueueBackingStoreFileV3: Reading checkpoint metadata from /root/.flume/file-channel/checkpoint/checkpoint.meta
16/06/14 14:38:10 INFO file.FlumeEventQueue: QueueSet population inserting 0 took 0
16/06/14 14:38:10 INFO file.Log: Last Checkpoint Tue Jun 14 14:37:49 CEST 2016, queue depth = 0
16/06/14 14:38:10 INFO file.Log: Replaying logs with v2 replay logic
16/06/14 14:38:10 INFO file.ReplayHandler: Starting replay of [/root/.flume/file-channel/data/log-9, /root/.flume/file-channel/data/log-10, /root/.flume/file-channel/data/log-11, /root/.flume/file-channel/data/log-12, /root/.flume/file-channel/data/log-13]
16/06/14 14:38:10 INFO file.ReplayHandler: Replaying /root/.flume/file-channel/data/log-9
16/06/14 14:38:10 INFO tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
16/06/14 14:38:10 INFO tools.DirectMemoryUtils: Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 20316160, Remaining = 20316160
16/06/14 14:38:10 INFO file.LogFile: fast-forward to checkpoint position: 58602
16/06/14 14:38:10 INFO file.LogFile: Encountered EOF at 58602 in /root/.flume/file-channel/data/log-9
16/06/14 14:38:10 INFO file.ReplayHandler: Replaying /root/.flume/file-channel/data/log-10
16/06/14 14:38:10 INFO file.LogFile: fast-forward to checkpoint position: 20798
16/06/14 14:38:10 INFO file.LogFile: Encountered EOF at 20798 in /root/.flume/file-channel/data/log-10
16/06/14 14:38:10 INFO file.ReplayHandler: Replaying /root/.flume/file-channel/data/log-11
16/06/14 14:38:10 INFO file.LogFile: fast-forward to checkpoint position: 3178
16/06/14 14:38:10 INFO file.LogFile: Encountered EOF at 3178 in /root/.flume/file-channel/data/log-11
16/06/14 14:38:10 INFO file.ReplayHandler: Replaying /root/.flume/file-channel/data/log-12
16/06/14 14:38:10 INFO file.LogFile: fast-forward to checkpoint position: 3264
16/06/14 14:38:10 INFO file.LogFile: Encountered EOF at 3264 in /root/.flume/file-channel/data/log-12
16/06/14 14:38:10 INFO file.ReplayHandler: Replaying /root/.flume/file-channel/data/log-13
16/06/14 14:38:10 INFO file.LogFile: fast-forward to checkpoint position: 3264
16/06/14 14:38:10 INFO file.LogFile: Encountered EOF at 3264 in /root/.flume/file-channel/data/log-13
16/06/14 14:38:10 INFO file.ReplayHandler: read: 0, put: 0, take: 0, rollback: 0, commit: 0, skip: 0, eventCount:0
16/06/14 14:38:10 INFO file.FlumeEventQueue: Search Count = 0, Search Time = 0, Copy Count = 0, Copy Time = 0
16/06/14 14:38:10 INFO file.Log: Rolling /root/.flume/file-channel/data
16/06/14 14:38:10 INFO file.Log: Roll start /root/.flume/file-channel/data
16/06/14 14:38:10 INFO file.LogFile: Opened /root/.flume/file-channel/data/log-14
16/06/14 14:38:10 INFO file.Log: Roll end
16/06/14 14:38:10 INFO file.EventQueueBackingStoreFile: Start checkpoint for /root/.flume/file-channel/checkpoint/checkpoint, elements to sync = 0
16/06/14 14:38:10 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1465907890431, queueSize: 0, queueHead: 373
16/06/14 14:38:10 INFO file.Log: Updated checkpoint for file: /root/.flume/file-channel/data/log-14 position: 0 logWriteOrderID: 1465907890431
16/06/14 14:38:10 INFO file.FileChannel: Queue Size after replay: 0 [channel=c1]
16/06/14 14:38:11 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
16/06/14 14:38:11 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started
16/06/14 14:38:11 INFO node.Application: Starting Sink k1
16/06/14 14:38:11 INFO node.Application: Starting Source r2
16/06/14 14:38:11 INFO source.ExecSource: Exec source starting with command:tail -f /PATH/bper-peg-pt-rest.log
16/06/14 14:38:11 INFO sink.RollingFileSink: Starting org.apache.flume.sink.RollingFileSink{name:k1, channel:c1}...
16/06/14 14:38:11 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: k1: Successfully registered new MBean.
16/06/14 14:38:11 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: k1 started
16/06/14 14:38:11 INFO sink.RollingFileSink: RollInterval is not valid, file rolling will not happen.
16/06/14 14:38:11 INFO sink.RollingFileSink: RollingFileSink k1 started.
16/06/14 14:38:11 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: r2: Successfully registered new MBean.
16/06/14 14:38:11 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: r2 started
16/06/14 14:38:11 INFO source.ExecSource: Command [tail -f /PATH/bper-peg-pt-rest.log] exited with 1
おかげ