Uploaded image for project: 'MidPoint'
  1. MidPoint
  2. MID-4122

DB connection leak

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.4.2
    • Fix Version/s: backlog
    • Component/s: Repository
    • Labels:
      None
    • Environment:

      Database: MySQL 5.7.18, for Linux (x86_64)
      DB server: SUSE Linux Enterprise Server 11.3
      Application server: WildFly 10.1.0
      App server os: SUSE Linux Enterprise Server 11.3

    • Subscription:
      No subscription (community)
    • Git Revision:
      2c863fa350c834f3fefffdd477ae7e777212c5bb

      Description

      I have observed couple of issues when I have attempted to deploy Midpoint on WildFly application server (possible due to MID-863).

      1. Transaction characteristics can't be changed while a transaction is in progress

      This issue is related to MySQL database. I have not observed this problem before on other deployments (using Tomcat + MySQL), only when I started to use Wildfly. Nevertheless this error occurs because in com.evolveum.midpoint.repo.sql.helpers.BaseHelper.beginTransaction(boolean readOnly) transaction access mode and isolation level is set after transaction is started.
      Stacktrace:

      2017-08-17 22:38:24,358 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (midPointScheduler_Worker-1) SQL Error: 1568, SQLState: 25001
      2017-08-17 22:38:24,358 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (midPointScheduler_Worker-1) Transaction characteristics can't be changed while a transaction is in p
      rogress
      2017-08-17 22:38:24,359 ERROR [com.evolveum.midpoint.model.impl.sync.CorrelationConfirmationEvaluator] (midPointScheduler_Worker-1) Couldn't search users in repository, based on filt
      er (simplified)
      Filter:
        EQUAL:
          PATH: name
          DEF: PPD:{.../common/common-3}name {http://prism.evolveum.com/xml/ns/public/types-3}PolyStringType[0,1],RAM
          VALUE:
              PolyString(3570e710-fb9d-409d-a36c-803510f9f6d9,3570e710fb9d409da36c803510f9f6d9)
          MATCHING: {http://prism.evolveum.com/xml/ns/public/matching-rule-3}polyStringNorm
      Paging: null., reason: error executing work (class com.evolveum.midpoint.util.exception.SystemException)
      2017-08-17 22:38:24,359 ERROR [com.evolveum.midpoint.model.impl.sync.SynchronizationServiceImpl] (midPointScheduler_Worker-1) Error occurred during resource object shadow owner looku
      p.
      2017-08-17 22:38:24,359 ERROR [com.evolveum.midpoint.provisioning.impl.ChangeNotificationDispatcherImpl] (midPointScheduler_Worker-1) Exception class com.evolveum.midpoint.util.excep
      tion.SystemException thrown by object change listener model synchronization service: Error occurred during resource object shadow owner lookup, reason: Couldn't search users in repos
      itory, based on filter (See logs).: com.evolveum.midpoint.util.exception.SystemException: Error occurred during resource object shadow owner lookup, reason: Couldn't search users in
      repository, based on filter (See logs).
              at com.evolveum.midpoint.model.impl.sync.SynchronizationServiceImpl.determineSituation(SynchronizationServiceImpl.java:548)
              at com.evolveum.midpoint.model.impl.sync.SynchronizationServiceImpl.notifyChange(SynchronizationServiceImpl.java:245)
              at com.evolveum.midpoint.provisioning.impl.ChangeNotificationDispatcherImpl.notifyChange(ChangeNotificationDispatcherImpl.java:148)
              at com.evolveum.midpoint.provisioning.impl.ShadowCache.notifyResourceObjectChangeListeners(ShadowCache.java:1433)
              at com.evolveum.midpoint.provisioning.impl.ShadowCache.createShadowInRepository(ShadowCache.java:1117)
              at com.evolveum.midpoint.provisioning.impl.ShadowCache.lookupOrCreateShadowInRepository(ShadowCache.java:1056)
              at com.evolveum.midpoint.provisioning.impl.ShadowCache.access$200(ShadowCache.java:150)
              at com.evolveum.midpoint.provisioning.impl.ShadowCache$2.handle(ShadowCache.java:858)
              at com.evolveum.midpoint.provisioning.impl.ResourceObjectConverter$3.handle(ResourceObjectConverter.java:1104)
              at com.evolveum.midpoint.provisioning.ucf.impl.ConnectorInstanceIcfImpl$2.handle(ConnectorInstanceIcfImpl.java:2334)
              at org.identityconnectors.framework.impl.api.SearchResultsHandlerLoggingProxy.handle(SearchResultsHandlerLoggingProxy.java:64)
              at org.identityconnectors.framework.impl.api.StreamHandlerUtil$ObjectStreamHandlerAdapter.handle(StreamHandlerUtil.java:101)
              at org.identityconnectors.framework.impl.api.BufferedResultsProxy.invoke(BufferedResultsProxy.java:262)
              at org.identityconnectors.framework.impl.api.DelegatingTimeoutProxy.invoke(DelegatingTimeoutProxy.java:94)
              at com.sun.proxy.$Proxy152.search(Unknown Source)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.identityconnectors.framework.impl.api.LoggingProxy.invoke(LoggingProxy.java:83)
              at com.sun.proxy.$Proxy152.search(Unknown Source)
              at org.identityconnectors.framework.impl.api.AbstractConnectorFacade.search(AbstractConnectorFacade.java:177)
              at com.evolveum.midpoint.provisioning.ucf.impl.ConnectorInstanceIcfImpl.search(ConnectorInstanceIcfImpl.java:2430)
              at com.evolveum.midpoint.provisioning.impl.ResourceObjectConverter.searchResourceObjects(ResourceObjectConverter.java:1114)
              at com.evolveum.midpoint.provisioning.impl.ShadowCache.searchObjectsIterative(ShadowCache.java:897)
              at com.evolveum.midpoint.provisioning.impl.ShadowCache.searchObjectsIterative(ShadowCache.java:822)
              at com.evolveum.midpoint.provisioning.impl.ProvisioningServiceImpl.searchObjectsIterative(ProvisioningServiceImpl.java:1239)
              at com.evolveum.midpoint.model.impl.ModelObjectResolver.searchIterative(ModelObjectResolver.java:224)
              at com.evolveum.midpoint.model.impl.util.AbstractSearchIterativeTaskHandler.runInternal(AbstractSearchIterativeTaskHandler.java:258)
              at com.evolveum.midpoint.model.impl.util.AbstractSearchIterativeTaskHandler.run(AbstractSearchIterativeTaskHandler.java:165)
              at com.evolveum.midpoint.task.quartzimpl.execution.JobExecutor.executeHandler(JobExecutor.java:473)
              at com.evolveum.midpoint.task.quartzimpl.execution.JobExecutor.executeSingleTask(JobExecutor.java:294)
              at com.evolveum.midpoint.task.quartzimpl.execution.JobExecutor.execute(JobExecutor.java:160)
              at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
              at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
      Caused by: com.evolveum.midpoint.util.exception.SystemException: Couldn't search users in repository, based on filter (See logs).
              at com.evolveum.midpoint.model.impl.sync.CorrelationConfirmationEvaluator.findUsersByCorrelationRule(CorrelationConfirmationEvaluator.java:239)
              at com.evolveum.midpoint.model.impl.sync.CorrelationConfirmationEvaluator.findFocusesByCorrelationRule(CorrelationConfirmationEvaluator.java:101)
              at com.evolveum.midpoint.model.impl.sync.SynchronizationServiceImpl.determineSituationWithCorrelation(SynchronizationServiceImpl.java:637)
              at com.evolveum.midpoint.model.impl.sync.SynchronizationServiceImpl.determineSituation(SynchronizationServiceImpl.java:542)
              ... 34 more
      Caused by: com.evolveum.midpoint.util.exception.SystemException: error executing work
              at com.evolveum.midpoint.repo.sql.helpers.BaseHelper.handleGeneralRuntimeException(BaseHelper.java:178)
              at com.evolveum.midpoint.repo.sql.helpers.BaseHelper.handleGeneralException(BaseHelper.java:158)
              at com.evolveum.midpoint.repo.sql.helpers.ObjectRetriever.searchObjectsAttempt(ObjectRetriever.java:386)
              at com.evolveum.midpoint.repo.sql.SqlRepositoryServiceImpl.searchObjects(SqlRepositoryServiceImpl.java:236)
              at com.evolveum.midpoint.repo.cache.RepositoryCache.searchObjects(RepositoryCache.java:184)
              at com.evolveum.midpoint.model.impl.sync.CorrelationConfirmationEvaluator.findUsersByCorrelationRule(CorrelationConfirmationEvaluator.java:231)
              ... 37 more
      Caused by: org.hibernate.exception.GenericJDBCException: error executing work
              at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
              at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
              at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)
              at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:318)
              at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2095)
              at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2080)
              at com.evolveum.midpoint.repo.sql.helpers.BaseHelper.beginTransaction(BaseHelper.java:105)
              at com.evolveum.midpoint.repo.sql.helpers.BaseHelper.beginReadOnlyTransaction(BaseHelper.java:78)
      
      2. Modification can't be performed because operation runs in read-only transaction

      Transaction access mode in MySQL relates to next transaction. It seams that some transactions are not closed and then MySQL next transaction also inherits this access mode.
      Stack trace:

      2017-08-22 10:14:10,436 INFO  [org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (pool-5-thread-1) HHH000010: On release of batch it still contained JDBC statements
      2017-08-22 10:14:10,436 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (pool-5-thread-1) SQL Error: 1792, SQLState: 25006
      2017-08-22 10:14:10,436 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (pool-5-thread-1) Cannot execute statement in a READ ONLY transaction.
      2017-08-22 10:14:10,436 ERROR [org.hibernate.engine.jdbc.batch.internal.BatchingBatch] (pool-5-thread-1) HHH000315: Exception executing batch [could not execute batch]
      2017-08-22 10:14:10,438 ERROR [com.evolveum.midpoint.task.quartzimpl.TaskQuartzImpl] (pool-5-thread-1) Couldn't store statistical information into task Task(id:1464602532788-0-1, name:Import APOS Groups, oid:2d06a57f-73da-4448-9456-d1010bcb43d4).: com.evolveum.midpoint.util.exception.SystemException: could not execute batch
      	at com.evolveum.midpoint.repo.sql.helpers.BaseHelper.handleGeneralRuntimeException(BaseHelper.java:184)
      	at com.evolveum.midpoint.repo.sql.helpers.BaseHelper.handleGeneralException(BaseHelper.java:164)
      	at com.evolveum.midpoint.repo.sql.helpers.ObjectUpdater.modifyObjectAttempt(ObjectUpdater.java:459)
      	at com.evolveum.midpoint.repo.sql.SqlRepositoryServiceImpl.modifyObject(SqlRepositoryServiceImpl.java:509)
      	at com.evolveum.midpoint.repo.sql.SqlRepositoryServiceImpl.modifyObject(SqlRepositoryServiceImpl.java:449)
      	at com.evolveum.midpoint.task.quartzimpl.TaskQuartzImpl.savePendingModifications(TaskQuartzImpl.java:336)
      	at com.evolveum.midpoint.task.quartzimpl.TaskQuartzImpl.storeOperationStats(TaskQuartzImpl.java:2986)
      	at com.evolveum.midpoint.model.impl.util.AbstractSearchIterativeResultHandler.processRequest(AbstractSearchIterativeResultHandler.java:379)
      	at com.evolveum.midpoint.model.impl.util.AbstractSearchIterativeResultHandler.access$200(AbstractSearchIterativeResultHandler.java:52)
      	at com.evolveum.midpoint.model.impl.util.AbstractSearchIterativeResultHandler$WorkerHandler.run(AbstractSearchIterativeResultHandler.java:287)
      	at com.evolveum.midpoint.task.quartzimpl.TaskManagerQuartzImpl$1.run(TaskManagerQuartzImpl.java:848)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.hibernate.exception.GenericJDBCException: could not execute batch
      	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
      	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
      	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:136)
      	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:114)
      	at org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:163)
      	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:226)
      	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:484)
      	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:351)
      	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:350)
      	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56)
      	at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1258)
      	at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:425)
      	at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.beforeTransactionCommit(JdbcTransaction.java:101)
      	at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:177)
      	at com.evolveum.midpoint.repo.sql.helpers.ObjectUpdater.modifyObjectAttempt(ObjectUpdater.java:439)
      	... 13 more
      Caused by: java.sql.BatchUpdateException: Cannot execute statement in a READ ONLY transaction.
      	at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1815)
      	at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1277)
      	at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1190)
      	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:127)
      	... 25 more
      Caused by: java.sql.SQLException: Cannot execute statement in a READ ONLY transaction.
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
      	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
      	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
      	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
      	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
      	at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
      	at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
      	... 28 more
      
      3. WildFly connection pool gets exhausted

      When Midpoint attempts to close connections they are not in fact closed, but they get back to pool. If such connection contains statements that are still open then such connection will not be used again and eventually the pool gets exhausted

      2017-08-18 21:32:37,307 WARN  [org.jboss.jca.adapters.jdbc.WrappedConnection] (ClusterManagerThread) IJ030040: Closing a statement you left open, please do your own housekeeping for: java:/
      jdbc/mysql/midpointDS: java.lang.Throwable: STACKTRACE
              at org.jboss.jca.adapters.jdbc.WrappedConnection.registerStatement(WrappedConnection.java:2001)
              at org.jboss.jca.adapters.jdbc.WrappedStatement.<init>(WrappedStatement.java:129)
              at org.jboss.jca.adapters.jdbc.WrappedStatement.<init>(WrappedStatement.java:94)
              at org.jboss.jca.adapters.jdbc.jdk7.WrappedStatementJDK7.<init>(WrappedStatementJDK7.java:51)
              at org.jboss.jca.adapters.jdbc.jdk7.WrappedConnectionJDK7.wrapStatement(WrappedConnectionJDK7.java:68)
              at org.jboss.jca.adapters.jdbc.WrappedConnection.createStatement(WrappedConnection.java:346)
              at com.evolveum.midpoint.repo.sql.helpers.BaseHelper$2.execute(BaseHelper.java:108)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lazyman Viliam Repan
                Reporter:
                dzasiewski Dominik Zasiewski
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: