Скачать или смотреть Understanding Cadence Configuration for Worker Threads and Workflows

Explore the optimal management of worker threads and workflows in Cadence to prevent outages and ensure smooth execution.
---
This video is based on the question https://stackoverflow.com/q/63941122/ asked by the user 'Aditya_G' ( https://stackoverflow.com/u/2275840/ ) and on the answer https://stackoverflow.com/a/63948758/ provided by the user 'Maxim Fateev' ( https://stackoverflow.com/u/1664318/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Cadence configuration for worker threads and workflow

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Understanding Cadence Configuration for Worker Threads and Workflows

In modern software development, particularly in systems that manage complex workflows, configuring threads and resource allocation is critical for maintaining high availability and performance. A common challenge developers face is managing the ratio between worker threads and workflows in systems like Cadence. The right configuration can prevent issues such as workflow execution failures and system outages. In this post, we'll delve into the fundamental aspects of Cadence configuration, addressing a common question: "What should be the ratio between worker threads and workflows, and how can we manage threads to prevent outages?"

The Issue: Worker and Workflow Outages

When dealing with a high volume of workflows, developers may encounter the error: "Not enough threads to execute workflows." This often arises when there are more workflows requiring execution than there are available threads. Let's break down the reasons behind this issue and explore potential solutions.

Understanding the Error

Not Enough Threads: This error indicates that the system is under-resourced to execute the currently active workflow tasks. To address this, developers need to adjust certain configuration settings:

WorkerOptions.maxConcurrentWorkflowExecutionSize: This determines how many workflow tasks can run concurrently.

WorkerOptions.maxWorkflowThreads: This defines the total number of threads available for executing both cached and running workflows.

Configuration Options

Setting the Right Ratio of Worker Threads

Interestingly, there is no strict ratio of workflows to workers. Blocked workflows do not consume significant memory after being pushed out of the cache. Therefore, it is entirely feasible to manage billions of blocked workflows with a minimal number of active workers, provided those workflows are not progressing. However, for efficient processing:

Ensure that maxWorkflowThreads is significantly higher than maxConcurrentWorkflowExecutionSize.

For instance, if maxConcurrentWorkflowExecutionSize is set to 100 and each workflow requires two threads, then maxWorkflowThreads should be at least 200 to avoid overwhelming the system.

Managing Blocked Workflows

Worker Memory Management: Blocked workflows remain cached in memory until they need to make progress. They only reload when an event triggers them, such as the completion of an activity, a signal being sent, or a timer expiring.

Await States and Their Impact

Workflows in an await state do not continuously occupy worker resources. Instead, they check conditions and execute only when a new event occurs. This minimizes the resource usage of these workflows, keeping worker threads free for other tasks.

Practical Considerations

Let's consider a scenario where a company scales its workflow management to handle one million workflows per day, with a closure time of two days, and an average signal trigger time of one day after a workflow starts. Such an architecture can work well if the workers can keep pace with the workload. However, developers must recognize the importance of proper configuration to ensure smooth operations:

Ensure adequate thread availability to handle peaks in workflow executions.

Monitor workflow states to anticipate and respond to blocking conditions effectively.

Conclusion

Balancing the configuration of worker threads and workflows in Cadence is essential for efficient system performance. By understanding key parameters like maxConcurrentWorkflowExecutionSize and maxWorkflowThreads, and properly managing the state of workflows, developers can avoid common pitfalls and ensure a seamless workflow execution. Remember, fine-tuning these settings should be an ongoi