hadoop's attempt directory logs

[Abhijit Dhar]

Hi,

Could you please point me how to stream the attempt logs that hadoop generates to chukwa, while at the same time saving them the way hadoop saves by default under attempt directories/stdout ?

Thanks a lot

[Eric Yang-3]

This can be setup by using TaskLogAppender that comes with Chukwa.
In $HADOOP_CONF_DIR/log4j.properties, modify the TaskLogAppender configuration
to:

log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender

Make sure chukwa-0.x.0-client.jar is in Hadoop class path, and restart Hadoop.

This should enable Task Attempt Logs to be sent through Chukwa data
collection system, and also log a copy locally on the task tracker
node.  Hope this is what you are looking for.

regards,
Eric

On Mon, Jan 16, 2012 at 7:20 PM, Abhijit Dhar <[hidden email]> wrote:

> Hi,
>
> Could you please point me how to stream the attempt logs that hadoop
> generates to chukwa, while at the same time saving them the way hadoop saves
> by default under attempt directories/stdout ?
>
> Thanks a lot
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3665088.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

[Abhijit Dhar]

I added this in my $HADOOP_CONF_DIR/log4j.properties

log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender
log4j.appender.TLA.recordType=CrawlerLog
log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}
log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t] [%c{5}.%M()]:: %m%n

But, it looks like when I add this to the rootlogger  (log4j.rootLogger=ERROR, stdout, TLA), it stops going through even my stdout log appender.
Any idea, what might be going on? (chukwa jar is on my classpath, and I don't see any ClassNotFound exception, so I think that part is good)

Thanks,

On Tue, Jan 17, 2012 at 6:03 PM, Eric Yang <[hidden email]> wrote:

This can be setup by using TaskLogAppender that comes with Chukwa.
In $HADOOP_CONF_DIR/log4j.properties, modify the TaskLogAppender configuration
to:

log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender

Make sure chukwa-0.x.0-client.jar is in Hadoop class path, and restart Hadoop.

This should enable Task Attempt Logs to be sent through Chukwa data
collection system, and also log a copy locally on the task tracker
node.  Hope this is what you are looking for.

regards,
Eric

On Mon, Jan 16, 2012 at 7:20 PM, Abhijit Dhar <[hidden email]> wrote:
> Hi,
>
> Could you please point me how to stream the attempt logs that hadoop
> generates to chukwa, while at the same time saving them the way hadoop saves
> by default under attempt directories/stdout ?
>
> Thanks a lot
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3665088.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

--
Абижит Дар

[Eric Yang-3]

The logger should set to INFO because the information are logged at
INFO level.  The increased severity level to ERROR filtered out all
log entries from task attempts.
Hope this helps.

regards,
Eric


On Thu, Jan 19, 2012 at 6:13 PM, Abhijit Dhar <[hidden email]> wrote:

> I added this in my $HADOOP_CONF_DIR/log4j.properties
>
> log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender
> log4j.appender.TLA.recordType=CrawlerLog
> log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}
> log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
> log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
> [%c{5}.%M()]:: %m%n
>
> But, it looks like when I add this to the rootlogger
> (log4j.rootLogger=ERROR, stdout, TLA), it stops going through even my stdout
> log appender.
> Any idea, what might be going on? (chukwa jar is on my classpath, and I
> don't see any ClassNotFound exception, so I think that part is good)
>
> Thanks,
>
>
>
> On Tue, Jan 17, 2012 at 6:03 PM, Eric Yang <[hidden email]> wrote:
>>
>> This can be setup by using TaskLogAppender that comes with Chukwa.
>> In $HADOOP_CONF_DIR/log4j.properties, modify the TaskLogAppender
>> configuration
>> to:
>>
>>
>> log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.TaskLogAppender
>>
>> Make sure chukwa-0.x.0-client.jar is in Hadoop class path, and restart
>> Hadoop.
>>
>> This should enable Task Attempt Logs to be sent through Chukwa data
>> collection system, and also log a copy locally on the task tracker
>> node.  Hope this is what you are looking for.
>>
>> regards,
>> Eric
>>
>> On Mon, Jan 16, 2012 at 7:20 PM, Abhijit Dhar <[hidden email]>
>> wrote:
>> > Hi,
>> >
>> > Could you please point me how to stream the attempt logs that hadoop
>> > generates to chukwa, while at the same time saving them the way hadoop
>> > saves
>> > by default under attempt directories/stdout ?
>> >
>> > Thanks a lot
>> >
>> > --
>> > View this message in context:
>> > http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3665088.html
>> > Sent from the Chukwa - Users mailing list archive at Nabble.com.
>
>
>
>
> --
> Абижит Дар

[Abhijit Dhar]

I tried different loglevels too. But, I checked, and found that TaskLogAppender does not even register itself with the chuka agent as an adapter. I tried using ChukwaDailyRollingFileAppender with the below config, and it works flawlessly -

log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
log4j.appender.TLA.File=logs/jcrawler.log
log4j.appender.TLA.recordType=CrawlerLog
log4j.appender.TLA.chukwaClientHostname=localhost
log4j.appender.TLA.chukwaClientPortNum=9093
log4j.appender.TLA.DatePattern=.yyyy-MM-dd
log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t] [%c{5}.%M()]:: %m%n

But, with ChukwaDailyRollingFileAppender, it does not create the task log attempt directories(which I need too)

[Eric Yang-3]

Chukwa's own TaskLogAppender code seems to be using default
initialization parameters for contacting Chukwa Agent.  The code may
need some update for it to work right.  The subAppend function between
ChukwaDailyRollingFileAppender and TaskLogAppender should look very
similar.  Unfortunately, it isn't at the current state.  I think a
patch is required to get Chukwa's TaskLogAppender to work.

regards,
Eric

On Jan 19, 2012, at 6:49 PM, Abhijit Dhar <[hidden email]> wrote:

> I tried different loglevels too. But, I checked, and found that
> TaskLogAppender does not even register itself with the chuka agent as an
> adapter. I tried using ChukwaDailyRollingFileAppender with the below config,
> and it works flawlessly -
>
> log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
> log4j.appender.TLA.File=logs/jcrawler.log
> log4j.appender.TLA.recordType=CrawlerLog
> log4j.appender.TLA.chukwaClientHostname=localhost
> log4j.appender.TLA.chukwaClientPortNum=9093
> log4j.appender.TLA.DatePattern=.yyyy-MM-dd
> log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
> log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
> [%c{5}.%M()]:: %m%n
>
> But, with ChukwaDailyRollingFileAppender, it does not create the task log
> attempt directories(which I need too)
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3674393.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

[Abhijit Dhar]

Can you point me how to fix this thing? Or if there is a workaround? Getting this to work would be really helpful.

On Thu, Jan 19, 2012 at 8:59 PM, Eric Yang <[hidden email]> wrote:

Chukwa's own TaskLogAppender code seems to be using default
initialization parameters for contacting Chukwa Agent.  The code may
need some update for it to work right.  The subAppend function between
ChukwaDailyRollingFileAppender and TaskLogAppender should look very
similar.  Unfortunately, it isn't at the current state.  I think a
patch is required to get Chukwa's TaskLogAppender to work.

regards,
Eric

On Jan 19, 2012, at 6:49 PM, Abhijit Dhar <[hidden email]> wrote:

> I tried different loglevels too. But, I checked, and found that
> TaskLogAppender does not even register itself with the chuka agent as an
> adapter. I tried using ChukwaDailyRollingFileAppender with the below config,
> and it works flawlessly -
>
> log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
> log4j.appender.TLA.File=logs/jcrawler.log
> log4j.appender.TLA.recordType=CrawlerLog
> log4j.appender.TLA.chukwaClientHostname=localhost
> log4j.appender.TLA.chukwaClientPortNum=9093
> log4j.appender.TLA.DatePattern=.yyyy-MM-dd
> log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
> log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
> [%c{5}.%M()]:: %m%n
>
> But, with ChukwaDailyRollingFileAppender, it does not create the task log
> attempt directories(which I need too)
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3674393.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

--
Абижит Дар

[Eric Yang-3]

You can try to merge the code from ChukwaDailyRollingFileAppender with
TaskLogAppender to get a working TaskLogAppender.  In particular, the
subAppend function should be identical to
ChukwaDailyRollingFileAppender.  This should be a jira for Chukwa 0.6.
Hope this helps.

regards,
Eric

On Fri, Jan 20, 2012 at 8:16 PM, Abhijit Dhar <[hidden email]> wrote:

> Can you point me how to fix this thing? Or if there is a workaround? Getting
> this to work would be really helpful.
>
>
>
>
> On Thu, Jan 19, 2012 at 8:59 PM, Eric Yang <[hidden email]> wrote:
>>
>> Chukwa's own TaskLogAppender code seems to be using default
>> initialization parameters for contacting Chukwa Agent.  The code may
>> need some update for it to work right.  The subAppend function between
>> ChukwaDailyRollingFileAppender and TaskLogAppender should look very
>> similar.  Unfortunately, it isn't at the current state.  I think a
>> patch is required to get Chukwa's TaskLogAppender to work.
>>
>> regards,
>> Eric
>>
>> On Jan 19, 2012, at 6:49 PM, Abhijit Dhar <[hidden email]>
>> wrote:
>>
>> > I tried different loglevels too. But, I checked, and found that
>> > TaskLogAppender does not even register itself with the chuka agent as an
>> > adapter. I tried using ChukwaDailyRollingFileAppender with the below
>> > config,
>> > and it works flawlessly -
>> >
>> >
>> > log4j.appender.TLA=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
>> > log4j.appender.TLA.File=logs/jcrawler.log
>> > log4j.appender.TLA.recordType=CrawlerLog
>> > log4j.appender.TLA.chukwaClientHostname=localhost
>> > log4j.appender.TLA.chukwaClientPortNum=9093
>> > log4j.appender.TLA.DatePattern=.yyyy-MM-dd
>> > log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
>> > log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p [%t]
>> > [%c{5}.%M()]:: %m%n
>> >
>> > But, with ChukwaDailyRollingFileAppender, it does not create the task
>> > log
>> > attempt directories(which I need too)
>> >
>> > --
>> > View this message in context:
>> > http://apache-chukwa.679492.n3.nabble.com/hadoop-s-attempt-directory-logs-tp3665088p3674393.html
>> > Sent from the Chukwa - Users mailing list archive at Nabble.com.
>
>
>
>
> --
> Абижит Дар