Quantcast

chukwa for processing log4j logs

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

chukwa for processing log4j logs

sangeetha
Hi,

I wanted to aggregate log4j logs files from different systems and perform search on the logs.
Is it possible in chukwa? How can I use chukwa for this?

Let me know the configuration details for doing the same.

Thanks,
Sangeetha
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

Ariel Rabkin
Yes, this is a reasonable use for Chukwa.  You'll have an adaptor per
log file. Depending on your particular needs, you can either have a
custom log4j appender to set up the Chukwa side or else explicitly
configure adaptors on the log files you care about.

You will have to build the search part yourself.  Chukwa designed to
get stuff into HDFS; the rest is up to you. Of course, if you build
something generally usable and contribute it to the project, I'm sure
you'd get help with development and maintenance.

--Ari

On Mon, Jul 18, 2011 at 10:21 PM, sangeetha <[hidden email]> wrote:

> Hi,
>
> I wanted to aggregate log4j logs files from different systems and perform
> search on the logs.
> Is it possible in chukwa? How can I use chukwa for this?
>
> Let me know the configuration details for doing the same.
>
> Thanks,
> Sangeetha
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/chukwa-for-processing-log4j-logs-tp3181483p3181483.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.
>



--
Ari Rabkin [hidden email]
UC Berkeley Computer Science Department
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

Eric Yang-3
The easiest method to get going is to use trunk version of Chukwa, and setup your log4j configuration like:

log4j.appender.DRFA=org.apache.log4j.net.SocketAppender
log4j.appender.DRFA.RemoteHost=localhost
log4j.appender.DRFA.Port=9000
log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

This will setup log4j to stream data to localhost:9096
Then setup init_adaptors with:

add SocketAdaptor DataType 9000 0

This will start streaming data to HDFS.  You will need to build your own indexing of your log data on HDFS to provide search function.  Hadoop has a tutorial for running "grep", it may be a starting point to build a search service.  Hope this helps.

regards,
Eric
On Jul 18, 2011, at 10:26 PM, Ariel Rabkin wrote:

> Yes, this is a reasonable use for Chukwa.  You'll have an adaptor per
> log file. Depending on your particular needs, you can either have a
> custom log4j appender to set up the Chukwa side or else explicitly
> configure adaptors on the log files you care about.
>
> You will have to build the search part yourself.  Chukwa designed to
> get stuff into HDFS; the rest is up to you. Of course, if you build
> something generally usable and contribute it to the project, I'm sure
> you'd get help with development and maintenance.
>
> --Ari
>
> On Mon, Jul 18, 2011 at 10:21 PM, sangeetha <[hidden email]> wrote:
>> Hi,
>>
>> I wanted to aggregate log4j logs files from different systems and perform
>> search on the logs.
>> Is it possible in chukwa? How can I use chukwa for this?
>>
>> Let me know the configuration details for doing the same.
>>
>> Thanks,
>> Sangeetha
>>
>> --
>> View this message in context: http://apache-chukwa.679492.n3.nabble.com/chukwa-for-processing-log4j-logs-tp3181483p3181483.html
>> Sent from the Chukwa - Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Ari Rabkin [hidden email]
> UC Berkeley Computer Science Department

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

sangeetha
Hi,

Thanks for the response.

Need help on how to get the trunk version of Chukwa.

Thanks,
Sangeetha
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

sangeetha
In reply to this post by Eric Yang-3
Hi,

Thanks for the response.

Need help on how to get the trunk version of Chukwa.

Thanks,
Sangeetha
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

Eric Yang-3
You will need apache forrest and subversion.  Check out the source code and run:

svn co http://svn.apache.org/repos/asf/incubator/chukwa/trunk

Then run:

cd trunk
ant tar

The software will be built in trunk/build/chukwa-0.5.0-incubating.tar.gz

regards,
Eric

On Jul 19, 2011, at 5:12 AM, sangeetha wrote:

> Hi,
>
> Thanks for the response.
>
> Need help on how to get the trunk version of Chukwa.
>
> Thanks,
> Sangeetha
>
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/chukwa-for-processing-log4j-logs-tp3181483p3182297.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

Abhijit Dhar
I have added this to my HADOOP_CONF_DIR/log4j.properties
    log4j.appender.DRFA=org.apache.log4j.net.SocketAppender
    log4j.appender.DRFA.RemoteHost=localhost
    log4j.appender.DRFA.Port=9096
    log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
    log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

And this as adapter : add SocketAdaptor HadoopMetrics 9095 0
And, I have added CHUKWA_HOME/conf/hadoop-metrics.properties to HADOOP_CONF_DIR.

But, I still can't see anything being written to the agent logs. Any idea what else I'm missing?

Thanks,
Abhijit
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

Eric Yang-3
The log4j.properties is for streaming Hadoop logs into Chukwa.  It is running on port 9096.  You will need to add:

add SocketAdaptor Hadoop 9096 0

Make sure chukwa-hadoop-0.5.0-client.jar, json-simple-1.1jar are copied to HADOOP_HOME/lib, and hadoop is restarted.

Hope this helps.

regards,
Eric

On Jul 22, 2011, at 5:07 PM, abhijitdhar wrote:

> I have added this to my HADOOP_CONF_DIR/log4j.properties
>    log4j.appender.DRFA=org.apache.log4j.net.SocketAppender
>    log4j.appender.DRFA.RemoteHost=localhost
>    log4j.appender.DRFA.Port=9096
>    log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
>    log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
>
> And this as adapter : add SocketAdaptor HadoopMetrics 9095 0
> And, I have added CHUKWA_HOME/conf/hadoop-metrics.properties to
> HADOOP_CONF_DIR.
>
> But, I still can't see anything being written to the agent logs. Any idea
> what else I'm missing?
>
> Thanks,
> Abhijit
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/chukwa-for-processing-log4j-logs-tp3181483p3192681.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

Abhijit Dhar
I have added add SocketAdaptor Hadoop 9096 0 too. And have the jars added to HADOOP_HOME/lib as you mentioned. But, still dont see any logs in agent. I'm running hadoop 0.20.4. I'm able to see logs from SystemMetrics adapter correctly in hbase, but none of Hadoop logs. I'm using trunk chukwa.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

Eric Yang-3
Try restore the original Hadoop log4j.properties, and keep hadoop-metrics.properties in HADOOP_CONF_DIR, and jar files in the respected places.
Restart the cluster and check for exception in hadoop log and out files.

regards,
Eric

On Jul 22, 2011, at 5:36 PM, abhijitdhar wrote:

> I have added add SocketAdaptor Hadoop 9096 0 too. And have the jars added to
> HADOOP_HOME/lib as you mentioned. But, still dont see any logs in agent. I'm
> running hadoop 0.20.4. I'm able to see logs from SystemMetrics adapter
> correctly in hbase, but none of Hadoop logs. I'm using trunk chukwa.
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/chukwa-for-processing-log4j-logs-tp3181483p3192712.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

Abhijit Dhar
I was able to get the chukwa metrics fine. But, still something is not right with Hadoop metrics.

I get this error in agent.log :

[root@ip-10-176-22-21 chukwa-0.5.0]# log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at java.net.Socket.connect(Socket.java:478)
        at java.net.Socket.<init>(Socket.java:375)
        at java.net.Socket.<init>(Socket.java:218)
        at org.apache.log4j.net.SocketAppender.connect(SocketAppender.java:203)
        at org.apache.log4j.net.SocketAppender.<init>(SocketAppender.java:153)
        at org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext.emitRecord(Log4JMetricsContext.java:91)
        at org.apache.hadoop.metrics.spi.AbstractMetricsContext.emitRecords(AbstractMetricsContext.java:306)
        at org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:292)
        at org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:52)
        at org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:251)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)

And, this exception in hadoop's *.out logs for all three - jvm,rpc and dfs:

2011-07-25 20:16:50,319 ERROR [main] [hadoop.metrics.MetricsUtil.getContext()] - Unable to create metrics context jvm
java.lang.ClassNotFoundException: org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at org.apache.hadoop.metrics.ContextFactory.getContext(ContextFactory.java:132)
        at org.apache.hadoop.metrics.MetricsUtil.getContext(MetricsUtil.java:56)
        at org.apache.hadoop.metrics.MetricsUtil.getContext(MetricsUtil.java:45)
        at org.apache.hadoop.metrics.jvm.JvmMetrics.<init>(JvmMetrics.java:81)
        at org.apache.hadoop.metrics.jvm.JvmMetrics.init(JvmMetrics.java:73)
        at org.apache.hadoop.metrics.jvm.JvmMetrics.init(JvmMetrics.java:60)
        at org.apache.hadoop.hdfs.server.datanode.metrics.DataNodeMetrics.<init>(DataNodeMetrics.java:98)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)

Any idea with the ConnectException and which classpath I'm missing?

Thanks,
Abhijit

On Fri, Jul 22, 2011 at 6:14 PM, Eric Yang <[hidden email]> wrote:
Try restore the original Hadoop log4j.properties, and keep hadoop-metrics.properties in HADOOP_CONF_DIR, and jar files in the respected places.
Restart the cluster and check for exception in hadoop log and out files.

regards,
Eric

On Jul 22, 2011, at 5:36 PM, abhijitdhar wrote:

> I have added add SocketAdaptor Hadoop 9096 0 too. And have the jars added to
> HADOOP_HOME/lib as you mentioned. But, still dont see any logs in agent. I'm
> running hadoop 0.20.4. I'm able to see logs from SystemMetrics adapter
> correctly in hbase, but none of Hadoop logs. I'm using trunk chukwa.
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/chukwa-for-processing-log4j-logs-tp3181483p3192712.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.




--
Абижит Дар
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: chukwa for processing log4j logs

Abhijit Dhar
I fixed the ClassNotFoundException (was missing jars), but now get
SocketAdaptor - Caught java.io.EOFException closing conneciton
in agent.log
I don't see any errors in hadoop *.out logs.
Loading...