Quantcast

Writing custom parser

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Writing custom parser

TARIQ
Hello list,

    I have used Chukwa, without any modification till now. Now I want
to collect data that is in form of "xml" and store it in Hbase tables
and for that I have to write the parser.Since it is my first attempt
of doing such thing, I would like to have some guidance from the
experts. I am using this link for getting started
"http://wiki.apache.org/hadoop/DemuxModification".If anyone is aware
of some other place that can help me, please let me know.Many thanks.

Regards,
    Mohammad Tariq
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Writing custom parser

Eric Yang-3
To store data to HBase, the demux parser must be implanted with Chukwa
HBase annotation.
For more details, check out this document:

http://people.apache.org/~eyang/chukwa-0.5.0-docs/programming.html

Section: Demux Data To HBase

Hope this helps.

regards,
Eric

On Mon, Dec 26, 2011 at 5:47 AM, Mohammad Tariq <[hidden email]> wrote:

> Hello list,
>
>    I have used Chukwa, without any modification till now. Now I want
> to collect data that is in form of "xml" and store it in Hbase tables
> and for that I have to write the parser.Since it is my first attempt
> of doing such thing, I would like to have some guidance from the
> experts. I am using this link for getting started
> "http://wiki.apache.org/hadoop/DemuxModification".If anyone is aware
> of some other place that can help me, please let me know.Many thanks.
>
> Regards,
>     Mohammad Tariq
AD
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Writing custom parser

AD
Also I wrote my own processor by just mimicking one of the existing like TSProcessor.  Its pretty simple you only need to implement a couple methods in a new classfile.  Here is a massively oversimplified example.

package org.apache.hadoop.chukwa.extraction.demux.processor.mapper;

public class MyCustomProcessor extends AbstractProcessor {

@Override
  protected void parse(String recordEntry,OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter) {

  ChukwaRecord record = new ChukwaRecord();
        this.buildGenericRecord(record, recordEntry, archiveKey
            .getTimePartition(), chunk.getDataType());

   // Do all your parsing here then add whatever fields you want to the record

   record.add("some_field","some_value");
   output.collect(key,record);
 }
}
   

On Mon, Dec 26, 2011 at 12:24 PM, Eric Yang <[hidden email]> wrote:
To store data to HBase, the demux parser must be implanted with Chukwa
HBase annotation.
For more details, check out this document:

http://people.apache.org/~eyang/chukwa-0.5.0-docs/programming.html

Section: Demux Data To HBase

Hope this helps.

regards,
Eric

On Mon, Dec 26, 2011 at 5:47 AM, Mohammad Tariq <[hidden email]> wrote:
> Hello list,
>
>    I have used Chukwa, without any modification till now. Now I want
> to collect data that is in form of "xml" and store it in Hbase tables
> and for that I have to write the parser.Since it is my first attempt
> of doing such thing, I would like to have some guidance from the
> experts. I am using this link for getting started
> "http://wiki.apache.org/hadoop/DemuxModification".If anyone is aware
> of some other place that can help me, please let me know.Many thanks.
>
> Regards,
>     Mohammad Tariq

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Writing custom parser

TARIQ
Thanks a lot Eric and AD.

Regards,
    Mohammad Tariq



On Mon, Dec 26, 2011 at 11:25 PM, AD [via Apache Chukwa]
<[hidden email]> wrote:

> Also I wrote my own processor by just mimicking one of the existing like
> TSProcessor.  Its pretty simple you only need to implement a couple methods
> in a new classfile.  Here is a massively oversimplified example.
>
> package org.apache.hadoop.chukwa.extraction.demux.processor.mapper;
>
> public class MyCustomProcessor extends AbstractProcessor {
>
> @Override
>   protected void parse(String recordEntry,OutputCollector<ChukwaRecordKey,
> ChukwaRecord> output, Reporter reporter) {
>
>   ChukwaRecord record = new ChukwaRecord();
>         this.buildGenericRecord(record, recordEntry, archiveKey
>             .getTimePartition(), chunk.getDataType());
>
>    // Do all your parsing here then add whatever fields you want to the
> record
>
>    record.add("some_field","some_value");
>    output.collect(key,record);
>  }
> }
>
>
> On Mon, Dec 26, 2011 at 12:24 PM, Eric Yang <[hidden email]> wrote:
>>
>> To store data to HBase, the demux parser must be implanted with Chukwa
>> HBase annotation.
>> For more details, check out this document:
>>
>> http://people.apache.org/~eyang/chukwa-0.5.0-docs/programming.html
>>
>> Section: Demux Data To HBase
>>
>> Hope this helps.
>>
>> regards,
>> Eric
>>
>> On Mon, Dec 26, 2011 at 5:47 AM, Mohammad Tariq <[hidden email]> wrote:
>> > Hello list,
>> >
>> >    I have used Chukwa, without any modification till now. Now I want
>> > to collect data that is in form of "xml" and store it in Hbase tables
>> > and for that I have to write the parser.Since it is my first attempt
>> > of doing such thing, I would like to have some guidance from the
>> > experts. I am using this link for getting started
>> > "http://wiki.apache.org/hadoop/DemuxModification".If anyone is aware
>> > of some other place that can help me, please let me know.Many thanks.
>> >
>> > Regards,
>> >     Mohammad Tariq
>
>
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-chukwa.679492.n3.nabble.com/Writing-custom-parser-tp3612782p3613116.html
> To unsubscribe from Apache Chukwa, click here.
> NAML
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Writing custom parser

Abhijit Dhar
Hi,

I wrote my custom parser too (in my separate project and built a jar). But where should I specify the path to chukwa-demux-conf.xml so that it picks up my parser from my jar?

Thanks,
Abhijit
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Writing custom parser

Eric Yang-3
Hi Abhijit,

The directory hosting chukwa-demux-conf.xml shoudl be on the class
path.  This is typically CHUKWA_CONF_DIR, which is exported as part of
the class path.

regards,
Eric

On Fri, Jan 13, 2012 at 3:52 PM, Abhijit Dhar <[hidden email]> wrote:

> Hi,
>
> I wrote my custom parser too (in my separate project and built a jar). But
> where should I specify the path to chukwa-demux-conf.xml so that it picks up
> my parser from my jar?
>
> Thanks,
> Abhijit
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/Writing-custom-parser-tp3612782p3658036.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.
Loading...