最近公司集成clickhouse的过程中,进行了一系列的性能及压力测试。其中应用使用了官方的clickhouse-jdbc驱动,数据源连接池使用的hikariCP,测试结果显示如果查询时间超长就会报read timeout错误,经过排查官方issues给出了解决办法:
ClickHouseProperties properties = new ClickhouseProperties(); properties.setSocketTimeout(timeout); new ClickHouseDataSource(url , properties);or
Properties properties = new Properties(); properties.put("socket_timeout", timeout); new ClickHouseDataSource(url, properties);
但是实际情况我们使用的默认的hikariCP连接池,进行的默认配置,无法设置该参数。
样例代码如下:
配置文件大致如下:
spring.datasource.type=com.zaxxer.hikari.HikariDataSource
spring.datasource.driver-class-name: ru.yandex.clickhouse.ClickHouseDriver
spring.datasource.hikari.connectionTimeout=30000
spring.datasource.hikari.idleTimeout=600000
spring.datasource.hikari.maxLifetime=1800000
connectionTimeout 参数根本无效,经过代码排查发现,其问题在于cliclhouse-JDBC 是通过http方式进行连接,并且没有完全实现jdbc接口。设置超时代码如下:
hikariCP部分:
com.zaxxer.hikari.pool.PoolBase
/** * Set the network timeout, if <code>isUseNetworkTimeout</code> is <code>true</code> and the * driver supports it. * * @param connection the connection to set the network timeout on * @param timeoutMs the number of milliseconds before timeout * @throws SQLException throw if the connection.setNetworkTimeout() call throws */ private void setNetworkTimeout(final Connection connection, final long timeoutMs) throws SQLException { if (isNetworkTimeoutSupported == TRUE) { connection.setNetworkTimeout(netTimeoutExecutor, (int) timeoutMs); } }clickHouse-JDBC部分:
ru.yandex.clickhouse.ClickHouseConnectionImpl
public void setNetworkTimeout(Executor executor, int milliseconds) throws SQLException { }
经过仔细研究代码发现,clickhosue超时的设置是在http请求建立时设置的,使用的Httpclient。
ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder
public CloseableHttpClient buildClient() throws Exception { return HttpClientBuilder.create() .setConnectionManager(getConnectionManager()) .setKeepAliveStrategy(createKeepAliveStrategy()) .setDefaultConnectionConfig(getConnectionConfig()) .setDefaultRequestConfig(getRequestConfig()) .setDefaultHeaders(getDefaultHeaders()) .disableContentCompression() // gzip здесь ни к чему. Используется lz4 при compress=1 .build(); }设置socketTimeout部分:
private RequestConfig getRequestConfig() { return RequestConfig.custom() .setSocketTimeout(properties.getSocketTimeout()) .setConnectTimeout(properties.getConnectionTimeout()) .build(); }
而hikariCP找到可以直接设置数据源的方法:
com.zaxxer.hikari.HikariConfig
/** * Set a {@link DataSource} for the pool to explicitly wrap. This setter is not * available through property file based initialization. * * @param dataSource a specific {@link DataSource} to be wrapped by the pool */ public void setDataSource(DataSource dataSource) { checkIfSealed(); this.dataSource = dataSource; }
最终解决办法,修改定义数据源中的语句:
HikariConfig config = new HikariConfig(properties); ClickHouseProperties ckProperties = new ClickhouseProperties(); ckProperties.setSocketTimeout(timeout); ClickHouseDataSource chDs=new ClickHouseDataSource(url , ckProperties); config.setDataSource(chDs); return new HikariDataSource(config);