An approach I've taken in .NET is to write a class that implements IDisposable and wraps a byte buffer of a standardized size. Behind the scenes it used an array of stacks for recycling the byte arrays. Normal use case would be something like:
using (ByteBuffer buffer = pool.AllocateBuffer(128 * 1024))
{
// use buffer.array
}
The pool used weak references to the buffers so they could eventually be GC'd. Dead weak references were cleaned out when found during allocation. Typically the byte arrays needed were big enough to need to go into the large object heap, the cutoff for which was 80k IIRC. The LOH was not collected until gen2 collection.
What you should do: allocate one read buffer with a capacity > expected length, overwrite-read into it, copy the data into a fresh capacity == actual length byte array, parse the copy, empty and reuse the read buffer.
What you definitely shouldn't do, despite it being tempting: allocate a read buffer each time and pass the whole thing around.