Notes on Operation in a Multi-Node Configuration

(This feature can be tried with the Pleasanter Extensions Trial.)

Overview¶

When you run queue-based background jobs in a multi-node (multiple servers) configuration, configure the following two settings. If you do not, jobs may run twice and downloads may fail.

1. Enable Quartz clustering
2. Specify shared storage as the save location of output files

The purpose of a multi-node configuration is availability (redundancy). Even if one node stops, the other nodes can continue processing, but the throughput of jobs registered in the queue does not improve (for the reason, see "1-4. Effect on Throughput").

Using This Function in a Multi-Tenant Environment¶

When you use "Queuing" on multiple nodes with the "Multi-Tenant Management Function", see "Multi-Tenant Management Function: Queuing: Notes on Multi-Node Configurations".

1. Enable Quartz Clustering¶

In a multi-node configuration, be sure to enable Quartz clustering.

1-1. Settings¶

Set Clustering.Enabled of Quartz.json to true.

1-2. Run CodeDefiner¶

After setting the Clustering.Enabled parameter to true, be sure to run CodeDefiner. The tables for using Quartz (the QRTZ_ tables) are created in the database.

1-3. Effects of Not Enabling It¶

If you start multiple nodes with clustering disabled (the default), the job dispatcher of each node (the process that periodically checks and runs the jobs registered in the queue) does not recognize the others, polls the job table independently, and gets and runs jobs at the same time. As a result, the following occurs.

1. The same job is run twice on multiple nodes
2. The control of serial execution per tenant breaks down
3. The process that recovers stuck jobs rolls back jobs running on other nodes (treats them as failed)

When clustering is enabled, Quartz uses exclusive control to always limit the node running the job dispatcher to one. This solves the problems above.

1-4. Effect on Throughput¶

As described in 1-3, when clustering is enabled, the node running the job dispatcher at any given moment is always limited to one. The job dispatchers of multiple nodes do not run at the same time to share the processing of the queue, so adding nodes does not improve the throughput of jobs registered in the queue.

What adding nodes gives you is availability (redundancy): when a running node stops, the job dispatcher of another node can take over the processing.

2. Specify Shared Storage as the Save Location of Output Files¶

Set the save location of output files to shared storage that all nodes can access.

2-1. Settings¶

For OutputFilePath of the BackgroundJobs.json parameter, specify a path that all nodes can access in common, such as a shared folder or NAS. If it is not specified or is null, an error occurs right after a job is registered.

2-2. Effects of Not Specifying Shared Storage¶

When clustering is enabled, the node that runs a job is not fixed (it changes depending on the load and failover). For this reason, if the output location is a path that exists only on a specific node, the output file cannot be found when it is downloaded, and the download fails.

For this reason, be sure to specify shared storage that all nodes can refer to with the same path.

2-3. Note on Deleting a Job During Download¶

If you delete a job from another node while its result file is being downloaded on one node, the deletion is not blocked and is run, and the download may fail. The exclusive control during the download of a result file is effective only within each node (it does not span nodes).

3. List of Related Parameters¶

Supported Versions¶