Please Share your Product Ideas with us!

All ideas are welcome. Just because the Idea doesn't make it into the product immediately does not make it a bad idea.

Collect background process improvement that reduces collect from hours to minutes

Within collect we are using Load Balencer. There is a setting for max jobs per LB queue but due to the way Collect service pages allocates the jobs, we are not seeing multiplue jobs in a LB queue. We are using manual SDS jobs, but i suspect the same is true for all collect job and also applies to no-load balancer solutions.

At the moment, the Service page goes down the list of LB queues adding a single job to each then sleeps. If the jobs run fast, then the queue empties quickly and the LB does nothing until the Service page runs again and adds another job. This means that our collect process is comprised of about 95% wait time. As a reuslt we have had to change the service page intervals from several minutes to 10 seconds and even now, about 30% of the time appears to be wait time.

Ideally, this process would run in a loop until all jobs are in the LB queues, or the LB is fully subscribed. This way the Service page run frequency can be longer.

  • Phil Towe
  • Mar 4 2020
  • Unlikely to Implement
  • Feb 17, 2021

    Admin response

    This idea relates to a custom web application, not core product. The details have been passed onto the relevant team in Syniti.

  • Attach files
  • Phil Towe commented
    17 Feb, 2021 09:35am

    Andy, this refers to the way all collect jobs are run, i think. In our case we are using Load balancer, but even when we are not, there is a massive amount of wait time.

    I think its the way the core product works.

  • Phil Towe commented
    8 Sep, 2020 06:52am

    Andy, I do not think it is limited to the Load Balancer, but I put that in as it maybe. I think it applies to all collect jobs, but you will know better.

    Basically, the way jobs are processed at the moment, it appears ADM spend a lot of time in a wait state. The jobs get called one at a time with a sleep in the middle instead of all "available slots" being used, and then the launcher sleeping until the next jobs are launched.

  • Admin
    Andy Lund commented
    8 Sep, 2020 05:00am

    Hello Phil - Are you referring to psaLoadBalance? If so, this not part of our core product, so this feedback with need to be referred to the owners of that application.