Quantcast

Need help to configure chukwa to write logs file generated by log4j to hdfs

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Need help to configure chukwa to write logs file generated by log4j to hdfs

Neha Singh

Hi,
                I am using  chukwa0.5.I have followed the steps in http://wiki.apache.org/hadoop/Chukwa_Quick_Start site but still I cannot see logs files in hdfs.
Secondly demuxprocessing is blank .Also I m not able to write into hbase.
Please help

Regards,
Neha Singh


________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Need help to configure chukwa to write logs file generated by log4j to hdfs

Eric Yang-3
Hi Neha,

By default, the pipeline writer does not have seqFileWriter enabled, hence,
it does not write to Hadoop.  In addition, HADOOP_CONF_DIR and
HBASE_CONF_DIR should be configured in the classpath of the collector.
 This can be configured in chukwa-env.sh.
If this still doesn't work, check collector log file for actual error.

Hope this helps.

regards,
Eric

On Tue, Dec 25, 2012 at 10:19 PM, NehaS Singh
<[hidden email]>wrote:

>
> Hi,
>                 I am using  chukwa0.5.I have followed the steps in
> http://wiki.apache.org/hadoop/Chukwa_Quick_Start site but still I cannot
> see logs files in hdfs.
> Secondly demuxprocessing is blank .Also I m not able to write into hbase.
> Please help
>
> Regards,
> Neha Singh
>
>
> ________________________________
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Need help to configure chukwa to write logs file generated by log4j to hdfs

Neha Singh
I have added the HBASE_CONF_DIR as well as HADOOP_CONF_DIR to the classpath and my collector.og file looks as below


2012-12-20 16:43:43,919 INFO main ChukwaConfiguration - chukwaConf is /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/etc/chukwa
2012-12-20 16:43:44,244 INFO main log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-12-20 16:43:44,288 INFO main log - jetty-6.1.26
2012-12-20 16:43:44,331 INFO main root - initing servletCollector
2012-12-20 16:43:44,334 INFO main SeqFileWriter - rotateInterval is 300000
2012-12-20 16:43:44,334 INFO main SeqFileWriter - not using fixed time interval scheme
2012-12-20 16:43:44,334 INFO main SeqFileWriter - outputDir is /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs
2012-12-20 16:43:44,334 INFO main SeqFileWriter - fsname is hdfs://172.20.104.226:8020
2012-12-20 16:43:44,334 INFO main SeqFileWriter - filesystem type from core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
2012-12-20 16:43:44,816 INFO main log - Started SelectChannelConnector@0.0.0.0:9080
2012-12-20 16:43:45,331 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:43:45,805 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:44:15,805 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:44:45,331 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:44:45,805 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:45:15,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:45:45,331 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:45:45,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:46:15,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:46:45,331 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:46:45,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:47:15,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:47:45,331 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:47:45,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:48:15,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:48:44,805 INFO Timer-2 SeqFileWriter - closing sink file/home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de46d0$
2012-12-20 16:48:44,855 INFO Timer-2 SeqFileWriter - no chunks written to /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de$
2012-12-20 16:48:45,332 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:48:45,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:49:15,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:49:45,332 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:49:45,807 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:50:15,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:50:45,332 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:50:45,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:51:15,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:51:45,332 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2012-12-20 16:51:45,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:52:15,806 INFO Timer-3 SeqFileWriter - stat:datacollection.writer.hdfs dataSize=0 dataRate=0
2012-12-20 16:52:45,332 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0

I am not able to understand the error.could u please help.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Need help to configure chukwa to write logs file generated by log4j to hdfs

Eric Yang-3
There doesn't seem to be any errors, nor data received by the collectors.
 Check for errors in agent log file, and make sure there are adaptors
configured to collect data.

regards,
Eric

On Wed, Dec 26, 2012 at 10:58 PM, Neha Singh <[hidden email]>wrote:

> I have added the HBASE_CONF_DIR as well as HADOOP_CONF_DIR to the classpath
> and my collector.og file looks as below
>
>
> 2012-12-20 16:43:43,919 INFO main ChukwaConfiguration - chukwaConf is
> /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/etc/chukwa
> 2012-12-20 16:43:44,244 INFO main log - Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2012-12-20 16:43:44,288 INFO main log - jetty-6.1.26
> 2012-12-20 16:43:44,331 INFO main root - initing servletCollector
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - rotateInterval is 300000
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - not using fixed time
> interval scheme
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - outputDir is
> /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - fsname is
> hdfs://172.20.104.226:8020
> 2012-12-20 16:43:44,334 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2012-12-20 16:43:44,816 INFO main log - Started
> SelectChannelConnector@0.0.0.0:9080
> 2012-12-20 16:43:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:43:45,805 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:44:15,805 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:44:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:44:45,805 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:45:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:45:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:45:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:46:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:46:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:46:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:47:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:47:45,331 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:47:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:48:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:48:44,805 INFO Timer-2 SeqFileWriter - closing sink
>
> file/home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de46d0$
> 2012-12-20 16:48:44,855 INFO Timer-2 SeqFileWriter - no chunks written to
>
> /home/user/chukwa-0.5.0/chukwa-incubating-0.5.0/chukwa/logs/20121220164344752_INFPH01464U_1de$
> 2012-12-20 16:48:45,332 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:48:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:49:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:49:45,332 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:49:45,807 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:50:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:50:45,332 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:50:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:51:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:51:45,332 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2012-12-20 16:51:45,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:52:15,806 INFO Timer-3 SeqFileWriter -
> stat:datacollection.writer.hdfs dataSize=0 dataRate=0
> 2012-12-20 16:52:45,332 INFO Timer-1 root -
> stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
>
> I am not able to understand the error.could u please help.
>
>
>
> --
> View this message in context:
> http://apache-chukwa.679492.n3.nabble.com/Need-help-to-configure-chukwa-to-write-logs-file-generated-by-log4j-to-hdfs-tp4025511p4025515.html
> Sent from the Chukwa - Developers mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Need help to configure chukwa to write logs file generated by log4j to hdfs

panpan2523
This post has NOT been accepted by the mailing list yet.
In reply to this post by Neha Singh
Hollywood stars are crazy about red sole Christian Louboutin shoes the amount of which is cool expensive. However, it is said that the females who abreast accept added than 500 pairs of Christian Louboutin shoes are already added than 3000. It can be said that the account of admirers of air jordan 10 retro shoes includes every body. Which not alone includes Victoria Beckham, Nicole Richie, Kim Kardashian ect. abounding of acclaimed stars, it aswell accomplish the adolescent stars who are actual accepted not alone for their adolescence such as Leighton Meester, Blake Lively, Miley Cyrus, all of whom are aswell crazy about Christian Louboutin red sole top heels.
Loading...