In UNSW, due to the single-thread live sync can't handle the load, we set the a multi-node import account task (Sync SIMS User - Multi-Nodes) to replace it.
The task (see attached) segments data by user ID suffix (0-9) and runs in 4 per node config in a three-server cluster and run every 180 seconds.
It had been running for about 1 weeks with some minor errors, but otherwise, it did its job. Those errors are like:
Last work bucket finished with status other than SUCCESS in Task(id:1571621186066-0-1, name:Sync SIMS User - Multi-Nodes (10.116.182.177:3)
Modification precondition does not hold for task:a81ab1d6-beae-43eb-8a4f-db9a1f712f9e(Sync SIMS User - Multi-Nodes)
c.e.m.t.q.TaskManagerQuartzImpl: A task cannot be made runnable; task = Task(id:1569019304350-0-1, name:Sync SIMS User - Multi-Nodes,
Then this morning (2019-10-22 9:15am), we found the 'Sync SIMS User - Multi-Nodes' task disappeared from our the task list. We ran audit report and there was no deletion event associated with it. The last trace of the task as yesterday 6:41pm in the m_audit_event table.
select * from m_audit_event where channel = 'http://midpoint.evolveum.com/xml/ns/public/provisioning/channels-3#import' order by timestampValue desc;
– 2019-10-21 18:41:29.889
After we gone through log files and other tasks, we found that it was very likely that the cleanup task purge the 'Sync SIMS User - Multi-Nodes' task.
The cleanup task (see attached) last ran was 6:42pm yesterday and deleted 15 tasks. The task log did not cleanly recorded deleted the Sync SIMS User - Multi-Nodes task. But it recorded deletion of most of its 12 sub-tasks (4x3).
In addition, it had this result:
<message>5 record(s) were hidden to save space. Total number of records: 15</message>
So I believed the Cleanup Task incorrectly interpreted the 'Sync SIMS User - Multi-Nodes' condition last night and purged it and its sub tasks.
Our "Clean Up Policy" for 'Closed Task' is using default setting: Clean up interval = P1M