如何在Java中无内存问题地使用REST模板Spring MVC获取大型JSON

当我从REST服务获得大型JSON时，我收到了java.lang.OutOfMemoyError：Java堆空间...

这发生在~1000次调用REST服务之后

    HttpEntity<ProductRequest> requestUpdate = new HttpEntity<>(request, httpHeaders);

    ResponseEntity<String> messageEntity = restTemplate.exchange(getEndPointAddress(), HttpMethod.POST, requestUpdate, String.class);

    Map<String, Object> map = gson.fromJson(Helper.cleanJson(messageEntity.getBody()), Map.class);

<pre>
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:2694) ~[?:1.7.0_181]
    at java.lang.String.<init>(String.java:203) ~[?:1.7.0_181]
    at java.lang.StringBuilder.toString(StringBuilder.java:405) ~[?:1.7.0_181]
    at org.springframework.util.StreamUtils.copyToString(StreamUtils.java:80) ~[stormjar.jar:4.0.3]
    at org.springframework.http.converter.StringHttpMessageConverter.readInternal(StringHttpMessageConverter.java:86) ~[stonjar.jar:4.0.3]
    at org.springframework.http.converter.StringHttpMessageConverter.readInternal(StringHttpMessageConverter.java:41) ~[stonjar.jar:4.0.3]
    at org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:193) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:104) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:884) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:868) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:622) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) ~[stonjar.jar:4.0.3]
    at com.myorg.wcf.PMiddle.retrieveP(PMiddle.java:95) ~[ssar.jar:4.0.3]
    at com.myorg.RetrieveP.getPJsonFromService(RetrieveP.java:188) ~[ssar.jar:4.0.3]
    at com.myorg.RetrieveP.execute(RetrieveP.java:107) ~[ssar.jar:4.0.3]
    at org.apache.storm.daemon.executor$fn__5044$tuple_action_fn__5046.invoke(executor.clj:727) ~[storm-core-1.1.0.jar:1.1.0]
    at org.apache.storm.daemon.executor$mk_task_receiver$fn__4965.invoke(executor.clj:459) ~[storm-core-1.1.0.jar:1.1.0]
    at org.apache.storm.disruptor$clojure_handler$reify__4480.onEvent(disruptor.clj:40) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.daemon.executor$fn__5044$fn__5057$fn__5110.invoke(executor.clj:846) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) [ss-core-1.1.0.jar:1.1.0]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.7.0_181]</pre>

---

解决方案

RestTemplate和底层的SpringBean不是为流处理而设计的，它们在内存中构建整个ResponsEntity。根据您的示例，您没有按预期使用RestTemplate。您正在读取String，而不是将响应对象Map<String, Object>反序列化委托给基础转换器Bean。

如果JSON响应实在太大，无法放入堆中，您可以将RestTemplate替换为URLConnection或任何其他允许您直接访问响应的HTTP客户端库。这将允许您手动读取和解析响应，例如by using com.google.gson.stream.JsonReader：

URLConnection conn = // create and open
JsonReader reader = new JsonReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
// read manually using reader

通过避免包含整个JSON正文的一个大String对象，您将能够以这种方式节省内存。