Background Jobs and Queue Optimization

Introduction#

A healthy UI with a forty-thousand-job backlog is not healthy operations. Teams discover queue debt only when orders stop importing or inventory stops updating on the channel.

This article covers cron overlap, queue channel design, worker allocation, and integration backpressure. Pair with API performance when external callers enqueue floods and with monitoring for lag alerts.

Cron overlap and scheduling hygiene#

Map every scheduled action: owner, average duration, and overlap risk. Stagger heavy jobs — do not run full catalog sync, accounting lock date checks, and MRP explosions in the same minute.

Disable or defer non-critical crons during peak warehouse or retail hours. Failed crons that retry immediately can amplify load — fix root cause and add exponential backoff in custom jobs.

• ▸Document cron duration trends weekly
• ▸Use dedicated cron workers on large estates when supported
• ▸Avoid unbounded search in cron without limit and chunking
• ▸Alert when cron duration exceeds interval (implicit overlap)

Queue channels and priorities#

Separate channels for interactive-critical maintenance, integration import, integration export, and bulk reporting. Assign concurrency per channel so a product full sync cannot consume every worker.

Priority flags should reflect business impact: order import before descriptive attribute sync. Document channel map for on-call engineers.

Chunking, idempotency, and retries#

Jobs should process bounded batches — five hundred order lines, one thousand products — with cursor or offset state stored on the job. Retries must be idempotent using external IDs to avoid duplicate documents.

Failed jobs need visible error text and correlation IDs linking Shopify or API payloads to Odoo records. Silent retry loops burn CPU without operations awareness.

Worker allocation and backpressure#

Queue workers share PostgreSQL with RPC workers. Increasing queue concurrency without database headroom increases lock contention. Apply backpressure at the integration edge — rate-limit webhook handlers and defer enqueue when depth exceeds thresholds.

Development teams should expose job metrics hooks in custom connectors — enqueue rate, success rate, oldest pending age — for monitoring dashboards.

Practical recommendations#

• ✓Maintain a cron and queue registry with owners and SLOs for max lag.
• ✓Use separate channels for import vs export integration traffic.
• ✓Chunk all bulk jobs with resumable state and idempotent writes.
• ✓Alert on queue age and cron overlap before operators notice stale data.
• ✓Rate-limit external triggers that enqueue unbounded work.
• ✓Review Shopify queue patterns when ecommerce drives backlog spikes.

Common mistakes#

• !Single default channel for all integration and report jobs.
• !Cron jobs that search entire tables every five minutes.
• !Retrying failed imports without deduplication keys.
• !Scaling queue workers without monitoring database locks.
• !Running manual queue_job requeue all during incidents blindly.
• !Hiding job failures from operations dashboards.

Conclusion#

Background job performance is queue design: prioritized channels, chunked idempotent work, and backpressure when integrations spike.

Next, tune API and webhook entry points so enqueue rate stays within worker and database capacity.