Out Of Memory When Converting A Large Stream To String

I am trying to convert a large stream (4mb) to a string which i eventually convert it to a JSON Array. when the stream size is small ( in KB ) every thing works fine, the minute it

Solution 1:

StringWriter writes to a StringBuffer internally. A StringBuffer is basically a wrapper round a char array. That array has a certain capacity. When that capacity is insufficient, StringBuffer will allocate a new larger char array and copy the contents of the previous one. At the end you call toString() on the StringWriter, which will again copy the contents of the char array into the char array of the resulting String.

If you have any means of knowing beforehand what the needed capacity is, you should use StringWriter's contructor that sets the initial capacity. That would avoid needlessly copying arrays to increase the buffer.

Yet that doesn't avoid the final copy that happens in toString(). If you're dealing with streams that can be large, you may need to reconsider whether you really need that inputstream as a String. Using a sufficiently large char array directly would avoid all the copying around, and would greatly reduce memory usage.

The ultimate solution would be to do some of the processing of the input, before all of the input has come in, so the characters that have been processed can be discarded. This way you'll only need to hold as much in memory as what is needed for a processing step.