Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
4.2
-
None
-
None
-
In our setup we have multiple nodes with different tasks. We have two nodes which are reserved for endusers and 4-5 additional nodes for reconcile/recompute and maintanance tasks. The maintanace tasks will send cluster requests for cache invalidation or node health checks. This will use the whole memory on enduser nodes and blocks them for every user or administrator even though there are not part of the actual working nodes.The only option left is a hard server reboot (no graceful shutdown).
In our setup we have multiple nodes with different tasks. We have two nodes which are reserved for endusers and 4-5 additional nodes for reconcile/recompute and maintanance tasks. The maintanace tasks will send cluster requests for cache invalidation or node health checks. This will use the whole memory on enduser nodes and blocks them for every user or administrator even though there are not part of the actual working nodes.The only option left is a hard server reboot (no graceful shutdown).
-
Active subscription
Description
Midpoint will need more and more memory over time and this can lead to an OutOfMemoryException.
We could isolate the problem to a few undelying problems
- For each session midpoint will create seperate filter chains. This is not optimal because the same filter chain will be created many times. This behavior is espacially strange during cluster requests because the authentication is hardcorded for this part.
- The Filter will not be garbage collected after the session timeout. After a session timeout the session will be removed correctly but the filters will be referenced as disposableBeans in AutowireBeanFactoryObjectPostProcessor.
Possible root of the problem
Our first idea is that spring is not made for the creation of many filters. So one filter should be created once and used whenever needed. Cluster requests are session based. Every internal cluster request e.g. for cache invalidation will create a session and a new filter chain. Theses sessions are not reused from the midpoint client side (ClusterExecutionHelperImpl).
Requested solution
The filter creation must be fixed and it should be evaluated wether sessions are a good idea for cluster requests. It can be reproduced if some kind of session (user-login, cluster login) will be created. During a heap dump or during debugging you can see the addition filters.