Quantcast

Query regarding adaptor for binary files

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Query regarding adaptor for binary files

Arati Mahimane
Hi All,

I want to send Binary files (larger than 10 MB) from Chukwa agent to collector.
I have the following queries:

1) I read that the filetailer.LWFTAdaptor  repeatedly tails a file, treating the file as a sequence of bytes, ignoring the content. But the Chunk boundaries are arbitrary. Is there any adaptor in which we can specify the chunk size ? If yes, how can the chunk size be configured?
(e.g. I have a 10 MB file and need to divide it into 10 chunks of 1 MB each and send each chunk from agent to collector)

2) When binary files are sent, does chukwa ensure that they are not corrupted in anyway (i.e. Do the contents of the file remain unaltered) ?

Any pointers ?

Thanks,
Arati Mahimane
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Query regarding adaptor for binary files

Ariel Rabkin
Howdy.

Yes, the chunk size is configurable. I think
chukwaAgent.fileTailingAdaptor.maxReadSize is the option you want to
change.

Chukwa should not alter any of your data and should send every byte at
least once, appropriately tagged so that you can recover the exact
original file.

--Ari

On Fri, Feb 4, 2011 at 6:28 AM, Arati Mahimane
<[hidden email]> wrote:

>
> Hi All,
>
> I want to send Binary files (larger than 10 MB) from Chukwa agent to
> collector.
> I have the following queries:
>
> 1) I read that the filetailer.LWFTAdaptor  repeatedly tails a file, treating
> the file as a sequence of bytes, ignoring the content. But the Chunk
> boundaries are arbitrary. Is there any adaptor in which we can specify the
> chunk size ? If yes, how can the chunk size be configured?
> (e.g. I have a 10 MB file and need to divide it into 10 chunks of 1 MB each
> and send each chunk from agent to collector)
>
> 2) When binary files are sent, does chukwa ensure that they are not
> corrupted in anyway (i.e. Do the contents of the file remain unaltered) ?
>
> Any pointers ?
>
> Thanks,
> Arati Mahimane
>
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/Query-regarding-adaptor-for-binary-files-tp2424540p2424540.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.
>



--
Ari Rabkin [hidden email]
UC Berkeley Computer Science Department
Loading...