Adjust formula for Netty worker threads. - #18493
Conversation
Generally, the new formula leads to much fewer threads at small processor counts, and similar numbers of threads at large processor counts.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @JsonProperty | ||
| @Min(1) | ||
| private int numMaxThreads = Math.max(10, (JvmUtils.getRuntimeInfo().getAvailableProcessors() * 17) / 16 + 2) + 30; | ||
| private int numMaxThreads = JvmUtils.getRuntimeInfo().getAvailableProcessors() * 3 / 2 + 1; |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation Note
|
@gianm , it makes sense to reduce the number of netty worker threads since these But I wonder if the change is not too drastic, especially since this affects all services. Do you think it would make sense to maybe roll this out gradually, like have a formula which reduces |
I believe the Netty worker threads are only supposed to be running callbacks, so they should need to scale with number of CPUs rather than number of outgoing connections. For that reason I am not expecting the smaller pools to be a bottleneck & I would rather jump straight to a new formula. Maybe we can validate this with some scale testing prior to release? |
Thanks for the clarification, @gianm .
Sure, I think this should be feasible. |
documents should be updated to align this change. |
|
Hey @gianm , we are testing this change in a cluster and will bombard the same with high load. We shall post the results once done. |
Good point. Updated. |
| |`druid.global.http.compressionCodec`|Compression codec to communicate with others. May be "gzip" or "identity".|`gzip`| | ||
| |`druid.global.http.readTimeout`|The timeout for data reads.|`PT15M`| | ||
| |`druid.global.http.unusedConnectionTimeout`|The timeout for idle connections in connection pool. The connection in the pool will be closed after this timeout and a new one will be established. This timeout should be less than `druid.global.http.readTimeout`. Set this timeout = ~90% of `druid.global.http.readTimeout`|`PT4M`| | ||
| |`druid.global.http.numMaxThreads`|Maximum number of I/O worker threads|`max(10, ((number of cores * 17) / 16 + 2) + 30)`| |
There was a problem hiding this comment.
Curious, how did we come up with these old formulas?
There was a problem hiding this comment.
@gianm , could you please add a short comment in the code regarding the heuristics behind the formula?
|
Hi @gianm , the local tests look good with these I/O worker threads setup (the supervisors for ingestion looks stable and are being queried just fine) |
|
Thanks for performing the verifications, @uds5501 ! |
* Adjust formula for Netty worker threads. Generally, the new formula leads to much fewer threads at small processor counts, and similar numbers of threads at large processor counts. * Update documentation.
8 participants
Generally, the new formula leads to much fewer threads at small processor counts, and similar numbers of threads at large processor counts. The purpose of this change is to avoid issues due to having too-large worker thread counts on small machines, such as OutOfMemoryError for too many threads or not enough direct memory.
Comparison table