Queuing: Notes on Multi-Node Configurations

Overview¶

When you use queuing in a multi-node configuration with the "Multi-Tenant Management Function", set an appropriate value for the WorkerCount parameter of BackgroundJobs.json, referring to the following.

For notes in a single-tenant environment, see the following manual.

• "Queuing Function: Notes on Operation in a Multi-Node Configuration"

Workers¶

For details of workers and of the "WorkerCount" parameter that specifies the number of workers, see "Multi-Tenant Management Function: Queuing".

Notes on Setting WorkerCount¶

Set the Same Value for WorkerCount on All Nodes (Required)¶

When clustering is enabled, the registration of workers is shared by all nodes. At startup, each node deletes the registrations other than those calculated from its own WorkerCount. For this reason, if the value of WorkerCount differs between nodes, each node deletes the registrations of the others every time it starts, and the registrations do not become stable. When a deletion occurs, a warning is recorded in the system log.

Leave Room in the Thread Slots¶

Workers share thread slots with services such as reminders and LDAP synchronization, which are enabled or disabled in BackgroundService.json. To start each service without delay, set the Clustering.MaxConcurrency parameter of Quartz.json to the number of services enabled in BackgroundService.json plus WorkerCount. Clustering.MaxConcurrency is the parameter that determines the size of the thread slots. If the value of Clustering.MaxConcurrency is too small, a warning is recorded in the system log at startup.

Parallel Execution and Distribution Across Nodes¶

In one run (the periodic run at each BackgroundJobDispatcherInterval parameter of BackgroundJobs.json), the job dispatcher gets one waiting job for each tenant it handles and runs them in parallel. Jobs of the same tenant are always run one at a time in order (serial execution per tenant).

Therefore, the number of jobs run at the same time is determined by the number of tenants that have waiting jobs among the tenants handled. It is not determined by the number of nodes.

Specify the number of workers with WorkerCount of BackgroundJobs.json. The default value (not set) is 1.

When WorkerCount is 1, one job dispatcher handles all tenants, so the load of running jobs is concentrated on the one node that runs it. Adding nodes only changes the node on which the load is concentrated, and does not distribute the load.

When WorkerCount is 2 or more, each tenant is assigned to one of the workers according to the remainder of the tenant ID divided by WorkerCount. If the workers run on different nodes, the load of running jobs is divided among the nodes. However, which node runs which worker is decided each time the job is triggered, so it is not fixed to a specific node, and an even distribution is not guaranteed.

Adding nodes has the following two effects.

1. Availability (redundancy): When a running node stops, another node takes over the processing of the job dispatcher.
2. Load distribution: Only when WorkerCount is set to 2 or more, the load of running jobs is divided among multiple nodes. In an environment where the CPU and memory of a single node were the limit of processing, a shorter processing time can be expected.

Adding nodes does not increase the number of jobs that can be run at the same time. Adding only nodes without increasing WorkerCount hardly changes the processing time.

Supported Versions¶