Скачать или смотреть How to Efficiently Run Multiple Python Scripts with subprocess.Popen and Flask

Discover solutions to manage background processes like running scripts using Python's `subprocess.Popen` with Flask to ensure your web application runs smoothly.
---
This video is based on the question https://stackoverflow.com/q/65012778/ asked by the user 'Rex-Raiden-X' ( https://stackoverflow.com/u/14709087/ ) and on the answer https://stackoverflow.com/a/65014199/ provided by the user 'André Ginklings' ( https://stackoverflow.com/u/6327784/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Can't get python subprocess.Popen() to start another python script in the background

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Can’t Get Python subprocess.Popen() to Start Another Python Script in the Background? Here’s How to Fix It!

If you've ever run into the problem of wanting to run a Python script in the background while still having your Flask application serve requests, you're not alone! A user has encountered a situation where they want their gather.py script (which collects data in an infinite loop) to run simultaneously with their main.py Flask app. Directly invoking the script using subprocess.Popen() seemed like a solution, but it left the database empty and didn’t work as expected.

So, what's going on? Let’s break down the solution into more manageable parts.

Understanding the Problem

Background Context

The gather.py script collects data from an XML file and uploads it to a database in an infinite loop every 60 seconds.

The Flask application (main.py) is designed to interact with this database, but when gather.py is imported directly, it halts the process, preventing Flask from starting.

Why subprocess.Popen() Isn't Working

When using subprocess.Popen([sys.executable, 'path\to\gather.py'],...), it may not generate errors but still doesn’t result in expected database updates. This happens because:

Infinite Loops: If your script runs indefinitely, it may block resources needed by other scripts.

Output Handling: Redirecting output without proper handling can lead to stalls (especially with stdout=subprocess.PIPE).

Background Process Management: Ensuring that subprocesses are running independently requires a solid management strategy.

Solutions to Run Scripts Simultaneously

Solution 1: Using Threads and Queue in Python

You can refactor your code to utilize Python's threading capabilities, which allows scripts to run in the background while maintaining Flask responsiveness. Here's how:

[[See Video to Reveal this Text or Code Snippet]]

Explanation of the Code

Threading: The background_task function runs in a separate thread, ensuring that main.py remains responsive.

Queue: It uses a queue to send messages from the background task to the Flask route handler.

On each request to the root endpoint (/), it retrieves a message from the queue, showing how long the background task has been running.

Solution 2: Running Scripts with Batch Files (Windows)

If threading feels complex, consider using a simple batch file approach to manage script execution:

Create a .bat file that starts both scripts.

Here’s an example of what your batch file could look like:

[[See Video to Reveal this Text or Code Snippet]]

Explanation of the Batch File

start Command: This command allows gather.py to start independently of the Flask server.

Environment Variable: The FLASK_APP variable specifies which Flask application to run.

Minimized Mode: To run the gather script in a minimized window, change the command to start /min python 'path\to\gather.py'.

Conclusion

Whether you choose threading or batch files, both solutions have their merits based on your application's needs. By employing these techniques, you can ensure that your background processes run smoothly without blocking your Flask app. Don’t forget to test each solution and refine it to best suit your application's specific requirements!

Remember, managing background tasks efficiently is essential for maintaining responsive applications, especially when they deal with real-time data. Happy coding!