Creating a Separate Custom Queue and Assign CPU Cores

(Example: item_reposting queue)

1. Concepts (quick but important)

Before jumping into config, let’s align terminology so things don’t get misconfigured later.

1.1 Queue

A queue is a logical lane for background jobs.

Examples:

• default

• short

• long

• item_reposting (custom)

Jobs are enqueued like:

frappe.enqueue(
    method="path.to.method",
    queue="item_reposting",
    timeout=3600
)

1.2 Worker

A worker is a Python process that:

• Listens to one or more queues

• Executes jobs serially (1 job at a time per worker)

So:

• 1 worker = 1 CPU core max

• 8 workers ≈ 8 cores (best case)

1.3 “Assigning 8 cores” (Reality Check)

Linux does not hard-assign CPU cores unless you use taskset / cgroups.

In Frappe:

• You “assign cores” by running enough worker processes

• OS scheduler distributes them across CPU cores

So when we say “assign 8 cores”, it means:

Run 8 worker processes for that queue

2. Step 1 – Define the Queue Timeout

Edit:

sites/common_site_config.json

Add or confirm your custom queue:

{
  "workers": {
    "default": {
      "timeout": 300
    },
    "short": {
      "timeout": 300
    },
    "long": {
      "timeout": 600
    },
    "item_reposting": {
      "timeout": 3600
    }
  }
}

📌 Why this matters

• Prevents long jobs from getting killed

• Must match (or be slightly lower than) Supervisor stopwaitsecs

3. Step 2 – Enqueue Jobs to the Custom Queue

Example usage in code:

frappe.enqueue(
    "erpnext.stock.doctype.repost_item_valuation.repost_item_valuation.repost",
    queue="item_reposting",
    timeout=3600,
    job_name="Item Reposting"
)

Always explicitly set:

• queue

• timeout

4. Step 3 – Create Dedicated Workers in Supervisor

Edit:

/etc/supervisor/conf.d/bench12-testing.conf

4.1 Example: item_reposting with 8 cores

[program:bench12-testing-frappe-item_reposting-worker]
command=/usr/local/bin/bench worker --queue item_reposting
priority=4
autostart=true
autorestart=true
stdout_logfile=/home/erpadmin/bench12-testing/logs/item_reposting_worker.log
stderr_logfile=/home/erpadmin/bench12-testing/logs/item_reposting_worker.error.log
user=erpadmin
directory=/home/erpadmin/bench12-testing
killasgroup=true
stopwaitsecs=3600
numprocs=8
process_name=%(program_name)s-%(process_num)d
startretries=10

What this achieves

5. Step 4 – Background Workers vs Cores (Important Clarification)

In common_site_config.json you have:

"background_workers": 4

What this controls

• Used by bench start

• Ignored when Supervisor is used

👉 Since you’re using Supervisor:

• Supervisor numprocs is the real authority

• background_workers can stay as-is

✅ Your setup is already correct

6. Step 5 – Grouping Workers (Optional but Recommended)

Edit:

/etc/supervisor/conf.d/bench12-testing.conf

Add your custom queue to worker groups:

[group:bench12-testing-workers]
programs=
    bench12-testing-frappe-schedule,
    bench12-testing-frappe-default-worker,
    bench12-testing-frappe-short-worker,
    bench12-testing-frappe-long-worker,
    bench12-testing-frappe-item_reposting-worker

This allows:

sudo supervisorctl restart bench12-testing-workers:

7. Step 6 – Reload Supervisor

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl restart all
sudo supervisorctl status | grep item_reposting

Expected output:

bench12-testing-frappe-item_reposting-worker-0 RUNNING
bench12-testing-frappe-item_reposting-worker-1 RUNNING
...
bench12-testing-frappe-item_reposting-worker-7 RUNNING

8. Step 7 – Verify Core Usage

8.1 CPU check

htop

Filter:

• Python

• bench worker

You should see up to 8 cores utilized when jobs are running.

9. Best Practices (Hard-earned advice)

✔ One heavy queue = one dedicated worker pool

Never mix heavy queues like item_reposting with:

• default

• short

✔ Do not oversubscribe CPU

If server has:

• 16 cores total

Good split:

• Web: 6–8 cores

• item_reposting: 8 cores

• Other queues: remaining

✔ Always use locking for heavy jobs

You already do this correctly using Redis locks.
Without locks, 8 workers = 8 disasters.

10. Template for Any New Custom Queue

Checklist

1. Add queue timeout in common_site_config.json

2. Enqueue jobs with queue="custom_queue"

3. Create Supervisor worker with numprocs = desired cores

4. Reload Supervisor

5. Monitor CPU + Redis

Example Formula

Desired CPU cores = numprocs
Jobs per core      = 1
Total concurrency  = numprocs