Скачать или смотреть How to Wait on Multiple Process Executions in Python’s Multiprocessing Module

Learn how to effectively manage and wait for multiple `multiprocess.Process` instances to finish execution in Python using simple techniques and concepts.
---
This video is based on the question https://stackoverflow.com/q/75559776/ asked by the user 'KamilCuk' ( https://stackoverflow.com/u/9072753/ ) and on the answer https://stackoverflow.com/a/75560438/ provided by the user 'KamilCuk' ( https://stackoverflow.com/u/9072753/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: How to wait on multiple multiprocess.Process to finish execution when they exec()?

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
How to Wait on Multiple Process Executions in Python’s Multiprocessing Module

In the world of concurrent programming, particularly in Python's multiprocessing module, managing process execution can be a bit tricky, especially when you need to wait for multiple processes to complete their tasks. If you've ever found yourself unsure about how to handle multiple subprocess execution in Python, you’re not alone! This guide aims to clarify how to wait for multiple multiprocess.Process instances to finish execution when they exec() a command.

The Problem at Hand

You may want to execute different processes concurrently and require a way to pause your main program until all of them have completed. In some instances, such as adjusting the Out-Of-Memory (OOM) score of subprocesses, you might rely on the exec() method. The typical approach to wait for a subprocess might not yield the expected result using multiprocessing.connection.wait() as it doesn't seem to effectively wait for the process to complete. This is a common pain point for developers using Python's multiprocess library.

Key Code Example

Consider the following example code where a process is intended to adjust the OOM score and then run a command to sleep for a second:

[[See Video to Reveal this Text or Code Snippet]]

In this code, you're likely to notice that while timestamps are being printed, they appear close together, indicating that the wait() function is not pausing for the execution duration of the sleep command.

The Solution

While the initial strategy utilized the sentinel attributes in multiprocessing, it's important to understand their nature. The sentinel isn't a direct representation of the process's execution state. Here’s a valid approach to ensure that the main thread waits for subprocess completion.

Step-by-Step Guide

Understand the Mechanics: Realize that the sentinel property opens a pipe rather than directly linking to the execution status of the subprocess. When the execv function is called, the original process (that started the subprocess) exits, and hence the select() function on the pipe returns immediately.

Use subprocess.Popen: Instead of relying solely on multiprocessing.Process to manage execution wait time, you can utilize the subprocess.Popen method with the preexec_fn parameter to adjust OOM scores before executing the command.

Implement Event Handling: In your main thread, implement threading.Event.wait() for effective synchronization. Set up a SIGCHLD signal handler that updates the event state when a child process finishes, allowing the main thread to manage completion appropriately.

Example Refined Code

Here is a refined example reflecting the above strategy using subprocess.Popen:

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

By employing subprocess.Popen alongside signal handling and threading events, you can effectively manage the execution of multiple subprocesses in Python. This method ensures that your main program reliably waits for the complete execution of all child processes where needed, even when using exec() to replace their process space.

This pattern not only solves the waiting issue but also adheres to the best practices for resource management and synchronization in concurrent programming.

Feel free to experiment with this setup in your projects, and you'll find it a robust way to handle multiprocessing task completion!