Uploaded image for project: 'MidPoint'
  1. MidPoint
  2. MID-5873

Cleanup Task deleted Multi-Node tasks incorrectly

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.8, 3.9
    • 3.9.1, 3.8.1
    • None
    • None
    • Production

    • Active subscription
    • M1

    Description

      In UNSW, due to the single-thread live sync can't handle the load, we set the a multi-node import account task (Sync SIMS User - Multi-Nodes) to replace it.
      The task (see attached) segments data by user ID suffix (0-9) and runs in 4 per node config in a three-server cluster and run every 180 seconds.

      It had been running for about 1 weeks with some minor errors, but otherwise, it did its job. Those errors are like:
      Last work bucket finished with status other than SUCCESS in Task(id:1571621186066-0-1, name:Sync SIMS User - Multi-Nodes (10.116.182.177:3)
      Modification precondition does not hold for task:a81ab1d6-beae-43eb-8a4f-db9a1f712f9e(Sync SIMS User - Multi-Nodes)
      c.e.m.t.q.TaskManagerQuartzImpl: A task cannot be made runnable; task = Task(id:1569019304350-0-1, name:Sync SIMS User - Multi-Nodes,

      Then this morning (2019-10-22 9:15am), we found the 'Sync SIMS User - Multi-Nodes' task disappeared from our the task list. We ran audit report and there was no deletion event associated with it. The last trace of the task as yesterday 6:41pm in the m_audit_event table.
      select * from m_audit_event where channel = 'http://midpoint.evolveum.com/xml/ns/public/provisioning/channels-3#import' order by timestampValue desc;
      – 2019-10-21 18:41:29.889

      After we gone through log files and other tasks, we found that it was very likely that the cleanup task purge the 'Sync SIMS User - Multi-Nodes' task.
      The cleanup task (see attached) last ran was 6:42pm yesterday and deleted 15 tasks. The task log did not cleanly recorded deleted the Sync SIMS User - Multi-Nodes task. But it recorded deletion of most of its 12 sub-tasks (4x3).
      In addition, it had this result:
      <partialResults>
      <operation>com.evolveum.midpoint.task.api.TaskManager.deleteTask</operation>
      <status>success</status>
      <hiddenRecordsCount>5</hiddenRecordsCount>
      <token>1000000000004174801</token>
      <message>5 record(s) were hidden to save space. Total number of records: 15</message>
      </partialResults>

      So I believed the Cleanup Task incorrectly interpreted the 'Sync SIMS User - Multi-Nodes' condition last night and purged it and its sub tasks.

      Our "Clean Up Policy" for 'Closed Task' is using default setting: Clean up interval = P1M

      Attachments

        Issue Links

          Activity

            People

              robert.li Robert Li
              robert.li Robert Li
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: