|
Hello list,
I have used Chukwa, without any modification till now. Now I want to collect data that is in form of "xml" and store it in Hbase tables and for that I have to write the parser.Since it is my first attempt of doing such thing, I would like to have some guidance from the experts. I am using this link for getting started "http://wiki.apache.org/hadoop/DemuxModification".If anyone is aware of some other place that can help me, please let me know.Many thanks. Regards, Mohammad Tariq |
|
To store data to HBase, the demux parser must be implanted with Chukwa
HBase annotation. For more details, check out this document: http://people.apache.org/~eyang/chukwa-0.5.0-docs/programming.html Section: Demux Data To HBase Hope this helps. regards, Eric On Mon, Dec 26, 2011 at 5:47 AM, Mohammad Tariq <[hidden email]> wrote: > Hello list, > > I have used Chukwa, without any modification till now. Now I want > to collect data that is in form of "xml" and store it in Hbase tables > and for that I have to write the parser.Since it is my first attempt > of doing such thing, I would like to have some guidance from the > experts. I am using this link for getting started > "http://wiki.apache.org/hadoop/DemuxModification".If anyone is aware > of some other place that can help me, please let me know.Many thanks. > > Regards, > Mohammad Tariq |
|
Also I wrote my own processor by just mimicking one of the existing like TSProcessor. Its pretty simple you only need to implement a couple methods in a new classfile. Here is a massively oversimplified example.
package org.apache.hadoop.chukwa.extraction.demux.processor.mapper; public class MyCustomProcessor extends AbstractProcessor { @Override protected void parse(String recordEntry,OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter) {
ChukwaRecord record = new ChukwaRecord(); this.buildGenericRecord(record, recordEntry, archiveKey .getTimePartition(), chunk.getDataType()); // Do all your parsing here then add whatever fields you want to the record record.add("some_field","some_value"); output.collect(key,record);
} } On Mon, Dec 26, 2011 at 12:24 PM, Eric Yang <[hidden email]> wrote: To store data to HBase, the demux parser must be implanted with Chukwa |
|
Thanks a lot Eric and AD.
Regards, Mohammad Tariq On Mon, Dec 26, 2011 at 11:25 PM, AD [via Apache Chukwa] <[hidden email]> wrote: > Also I wrote my own processor by just mimicking one of the existing like > TSProcessor. Its pretty simple you only need to implement a couple methods > in a new classfile. Here is a massively oversimplified example. > > package org.apache.hadoop.chukwa.extraction.demux.processor.mapper; > > public class MyCustomProcessor extends AbstractProcessor { > > @Override > protected void parse(String recordEntry,OutputCollector<ChukwaRecordKey, > ChukwaRecord> output, Reporter reporter) { > > ChukwaRecord record = new ChukwaRecord(); > this.buildGenericRecord(record, recordEntry, archiveKey > .getTimePartition(), chunk.getDataType()); > > // Do all your parsing here then add whatever fields you want to the > record > > record.add("some_field","some_value"); > output.collect(key,record); > } > } > > > On Mon, Dec 26, 2011 at 12:24 PM, Eric Yang <[hidden email]> wrote: >> >> To store data to HBase, the demux parser must be implanted with Chukwa >> HBase annotation. >> For more details, check out this document: >> >> http://people.apache.org/~eyang/chukwa-0.5.0-docs/programming.html >> >> Section: Demux Data To HBase >> >> Hope this helps. >> >> regards, >> Eric >> >> On Mon, Dec 26, 2011 at 5:47 AM, Mohammad Tariq <[hidden email]> wrote: >> > Hello list, >> > >> > I have used Chukwa, without any modification till now. Now I want >> > to collect data that is in form of "xml" and store it in Hbase tables >> > and for that I have to write the parser.Since it is my first attempt >> > of doing such thing, I would like to have some guidance from the >> > experts. I am using this link for getting started >> > "http://wiki.apache.org/hadoop/DemuxModification".If anyone is aware >> > of some other place that can help me, please let me know.Many thanks. >> > >> > Regards, >> > Mohammad Tariq > > > > > ________________________________ > If you reply to this email, your message will be added to the discussion > below: > http://apache-chukwa.679492.n3.nabble.com/Writing-custom-parser-tp3612782p3613116.html > To unsubscribe from Apache Chukwa, click here. > NAML |
|
Hi,
I wrote my custom parser too (in my separate project and built a jar). But where should I specify the path to chukwa-demux-conf.xml so that it picks up my parser from my jar? Thanks, Abhijit |
|
Hi Abhijit,
The directory hosting chukwa-demux-conf.xml shoudl be on the class path. This is typically CHUKWA_CONF_DIR, which is exported as part of the class path. regards, Eric On Fri, Jan 13, 2012 at 3:52 PM, Abhijit Dhar <[hidden email]> wrote: > Hi, > > I wrote my custom parser too (in my separate project and built a jar). But > where should I specify the path to chukwa-demux-conf.xml so that it picks up > my parser from my jar? > > Thanks, > Abhijit > > -- > View this message in context: http://apache-chukwa.679492.n3.nabble.com/Writing-custom-parser-tp3612782p3658036.html > Sent from the Chukwa - Users mailing list archive at Nabble.com. |
| Powered by Nabble | Edit this page |
