Camel JDBC StreamList 查询似乎在拆分之前加载整个结果集

2021-11-11 00:00:00 sql-server jdbc apache-camel

我正在运行一个 SQL 使用者来读取表中的更改，这一切都很好.但是，有时会发生大量更改，然后我的查询因内存不足错误而中断，如您所料.

I'm running a SQL consumer to read changes from a table, which is all well and good. However, there are occasions where changes happen on mass, and then my query breaks with out of memory error, as you might expect.

不幸的是，我被困在 Camel 2.17.6 上，所以 SQL 组件的 StreamList 选项不可用.(虽然根据 Camel-SQL 为什么使用 StreamList 似乎加载所有 ResultSet? 由于 Spring JDBC 限制，这不能用作流列表.)

Unfortunately, I'm stuck on Camel 2.17.6, so the StreamList option for the SQL component isn't available. (Although according to Camel-SQL Why using StreamList seems to load all ResultSet? this doesn't work as a stream list due to Spring JDBC limitations.)

因此，我使用支持流列表的 JDBC 组件重新编写了我的路由，并且一旦我增加要提取的记录数，我仍然会出现内存不足异常.似乎出于某种原因，JDBC 组件试图在传递给拆分器之前提取所有记录.

So I've re-written my route using the JDBC component, which supports a stream list, and I'm still getting out of memory exceptions as soon as I increase the number of records to extract. It would appear that for some reason, the JDBC component is trying to extract all the records before passing to the splitter.

我现在的形式是:

from("timer:timer...")
  .to( "language:constant:resource:classpath:pathToSqlStatement/sqlStatement.sql" )
  .to( "jdbc:msSqlServerDataSource?outputType=StreamList" )
  .split( body() ).streaming()
  .setBody().simple("$body[XMLDOC]")
  .setHeader("HeaderName").xpath("xpath/to/data")
  .to("jms:topic:name");

我最初确实有一个聚合策略 UseLatestAggregationStrategy 和 split() 之后的一个额外步骤，但我已经把它去掉了，试图删除所有可能的导致整个查询被保存在内存中，但我看不到我现在还能做什么.

I did originally have an aggregation strategy UseLatestAggregationStrategy and an extra step after the split() but I've stripped that out in an attempt to remove everything that could possibly result in the whole query being held in memory, but I can't see what else I can do now.

我注意到问题 camel jdbc 内存不足异常 引发了类似的问题，而且似乎没有解决办法.

I note the question camel jdbc out of memory exception raises a similar problem, and didn't appear to have a resolution.

(我应该注意到我遇到的内存不足错误确实出现在不同的地方，并且在 WinNTFileSystem 中包含了 GC 开销限制超过，我没有理解，还有其他与 ZippedInputStream 有关的事情，我又不明白.)

(I should note that the out of memory errors I've had do appear in different places, and included GC overhead limit exceeded at WinNTFileSystem which I don't understand, and something else to do with a ZippedInputStream, which again I don't understand.)

这是否意味着 StreamList 也不适用于 JDBC 组件，或者我必须做一些特定的事情来确保 JDBC 组件不会尝试缓存整个结果?

Does that mean that StreamList doesn't work on the JDBC component either, or do I have to do something specific to ensure that the JDBC component doesn't try to cache the whole results?

推荐答案

StreamList 从 v.18.x 版本开始，camel-sql 支持输出类型.在早期版本中，camel-sql 组件加载 内存中的结果集作为列表.我认为在camel-sql 2.17.x版本中无法避免.

StreamList output type is supported in camel-sql since version v.18.x. In earlier version, camel-sql component load the result set in memory as list. I don't think it can be avoided in camel-sql version 2.17.x.

camel-sql 组件将结果加载到内存中后，聚合/拆分确实适用.

Aggregation /splitting does apply after loading the result in memory by camel-sql component.

另一个相关答案.

相关文章