🔍 Deep Dive: gh-108388: Split test_multiprocessing_spawn by vstinner · Pull Request #108396 · python/cpython · GitHub

vstinner · 2023-08-24T02:26:53Z

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.

Issue: Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.

vstinner · 2023-08-24T02:37:13Z

Example running 4 slow tests:

./python -m test test_asyncio test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn -j0 --slowest

Timing (reference => this PR): 3 min 22 sec => 2 min 22 sec (-1 min)

On the main branch:

10 slowest tests:
- test_multiprocessing_spawn: 3 min 22 sec
- test_asyncio: 2 min 10 sec
- test_multiprocessing_forkserver: 1 min 43 sec
- test_multiprocessing_fork: 1 min 17 sec

Total duration: 3 min 22 sec

With this PR + PR #108393 (needed for this test):

40 tests OK.

10 slowest tests:
- test_multiprocessing_spawn.test_processes: 2 min 3 sec
- test_multiprocessing_forkserver.test_processes: 1 min 41 sec
- test_multiprocessing_spawn.test_misc: 1 min 22 sec
- test_multiprocessing_spawn.test_manager: 1 min 2 sec
- test_multiprocessing_forkserver.test_misc: 54.3 sec
- test_asyncio.test_tasks: 52.9 sec
- test_multiprocessing_forkserver.test_manager: 45.5 sec
- test_multiprocessing_fork.test_processes: 43.4 sec
- test_asyncio.test_events: 39.3 sec
- test_multiprocessing_fork.test_misc: 26.3 sec

2 tests skipped:
    test_asyncio.test_windows_events test_asyncio.test_windows_utils

Total duration: 2 min 22 sec

All CPUs are well used, the system load peak is 12.33 which is a good thing (the machine has 12 threads/6 cores).

Full logs:

Details

vstinner@mona$ ./python -m test test_asyncio test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn -j0 --slowest
0:00:00 load avg: 6.17 Run tests in parallel using 14 child processes
0:00:01 load avg: 6.17 [ 1/42] test_asyncio.test_protocols passed
0:00:01 load avg: 6.17 [ 2/42] test_asyncio.test_futures2 passed
0:00:01 load avg: 6.17 [ 3/42] test_asyncio.test_buffered_proto passed
0:00:01 load avg: 6.17 [ 4/42] test_asyncio.test_context passed
0:00:01 load avg: 6.17 [ 5/42] test_asyncio.test_pep492 passed
0:00:02 load avg: 6.17 [ 6/42] test_asyncio.test_runners passed
0:00:02 load avg: 6.17 [ 7/42] test_asyncio.test_proactor_events passed
0:00:02 load avg: 6.17 [ 8/42] test_asyncio.test_queues passed
0:00:03 load avg: 6.17 [ 9/42] test_asyncio.test_server passed
0:00:03 load avg: 6.17 [10/42] test_asyncio.test_selector_events passed
0:00:03 load avg: 6.17 [11/42] test_asyncio.test_eager_task_factory passed
0:00:04 load avg: 6.80 [12/42] test_asyncio.test_locks passed
0:00:04 load avg: 6.80 [13/42] test_asyncio.test_threads passed
0:00:06 load avg: 6.80 [14/42] test_asyncio.test_timeouts passed
0:00:06 load avg: 6.80 [15/42] test_asyncio.test_transports passed
0:00:07 load avg: 6.80 [16/42] test_asyncio.test_streams passed
0:00:07 load avg: 6.80 [17/42] test_asyncio.test_windows_events skipped
test_asyncio.test_windows_events skipped -- Windows only
0:00:07 load avg: 6.80 [18/42] test_asyncio.test_base_events passed
0:00:07 load avg: 6.80 [19/42] test_asyncio.test_sslproto passed
0:00:07 load avg: 6.80 [20/42] test_asyncio.test_windows_utils skipped
test_asyncio.test_windows_utils skipped -- Windows only
0:00:08 load avg: 6.80 [21/42] test_asyncio.test_unix_events passed
0:00:09 load avg: 7.70 [22/42] test_asyncio.test_waitfor passed
0:00:10 load avg: 7.70 [23/42] test_asyncio.test_sock_lowlevel passed
0:00:10 load avg: 7.70 [24/42] test_asyncio.test_sendfile passed
0:00:11 load avg: 7.70 [25/42] test_asyncio.test_taskgroups passed
0:00:12 load avg: 7.70 [26/42] test_asyncio.test_futures passed
0:00:18 load avg: 8.68 [27/42] test_asyncio.test_ssl passed
0:00:28 load avg: 10.42 [28/42] test_multiprocessing_fork.test_threads passed
0:00:29 load avg: 11.02 [29/42] test_asyncio.test_subprocess passed
Unknown child process pid 313199, will report returncode 255
Loop <_UnixSelectorEventLoop running=False closed=True debug=False> that handles pid 313199 is closed
0:00:32 load avg: 11.02 [30/42] test_multiprocessing_fork.test_manager passed -- running: test_asyncio.test_events (32.0 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310261) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=313364) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
0:00:32 load avg: 11.02 [31/42] test_multiprocessing_forkserver.test_threads passed -- running: test_asyncio.test_events (32.9 sec)
0:00:34 load avg: 11.50 [32/42] test_multiprocessing_fork.test_misc passed -- running: test_asyncio.test_events (34.6 sec), test_asyncio.test_tasks (31.7 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=313266) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
0:00:40 load avg: 12.02 [33/42] test_asyncio.test_events passed (39.3 sec) -- running: test_multiprocessing_forkserver.test_misc (30.9 sec), test_asyncio.test_tasks (37.1 sec), test_multiprocessing_fork.test_processes (32.4 sec), test_multiprocessing_forkserver.test_manager (31.1 sec)
0:00:51 load avg: 12.33 [34/42] test_multiprocessing_spawn.test_threads passed -- running: test_multiprocessing_forkserver.test_processes (40.9 sec), test_multiprocessing_spawn.test_processes (32.8 sec), test_multiprocessing_forkserver.test_misc (42.1 sec), test_multiprocessing_spawn.test_misc (38.9 sec), test_asyncio.test_tasks (48.2 sec), test_multiprocessing_fork.test_processes (43.5 sec), test_multiprocessing_spawn.test_manager (39.7 sec), test_multiprocessing_forkserver.test_manager (42.2 sec)
0:00:51 load avg: 12.33 [35/42] test_multiprocessing_fork.test_processes passed (43.4 sec) -- running: test_multiprocessing_forkserver.test_processes (41.5 sec), test_multiprocessing_spawn.test_processes (33.5 sec), test_multiprocessing_forkserver.test_misc (42.7 sec), test_multiprocessing_spawn.test_misc (39.5 sec), test_asyncio.test_tasks (48.8 sec), test_multiprocessing_spawn.test_manager (40.4 sec), test_multiprocessing_forkserver.test_manager (42.9 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310329) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310329) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
0:00:55 load avg: 12.06 [36/42] test_multiprocessing_forkserver.test_manager passed (45.5 sec) -- running: test_multiprocessing_forkserver.test_processes (45.0 sec), test_multiprocessing_spawn.test_processes (37.0 sec), test_multiprocessing_forkserver.test_misc (46.2 sec), test_multiprocessing_spawn.test_misc (43.0 sec), test_asyncio.test_tasks (52.3 sec), test_multiprocessing_spawn.test_manager (43.9 sec)
0:00:56 load avg: 12.06 [37/42] test_asyncio.test_tasks passed (52.9 sec) -- running: test_multiprocessing_forkserver.test_processes (46.3 sec), test_multiprocessing_spawn.test_processes (38.2 sec), test_multiprocessing_forkserver.test_misc (47.5 sec), test_multiprocessing_spawn.test_misc (44.3 sec), test_multiprocessing_spawn.test_manager (45.1 sec)
0:01:04 load avg: 10.75 [38/42] test_multiprocessing_forkserver.test_misc passed (54.3 sec) -- running: test_multiprocessing_forkserver.test_processes (53.8 sec), test_multiprocessing_spawn.test_processes (45.7 sec), test_multiprocessing_spawn.test_misc (51.8 sec), test_multiprocessing_spawn.test_manager (52.6 sec)
0:01:14 load avg: 9.63 [39/42] test_multiprocessing_spawn.test_manager passed (1 min 2 sec) -- running: test_multiprocessing_forkserver.test_processes (1 min 4 sec), test_multiprocessing_spawn.test_processes (56.1 sec), test_multiprocessing_spawn.test_misc (1 min 2 sec)
0:01:35 load avg: 12.00 [40/42] test_multiprocessing_spawn.test_misc passed (1 min 22 sec) -- running: test_multiprocessing_forkserver.test_processes (1 min 24 sec), test_multiprocessing_spawn.test_processes (1 min 16 sec)
0:01:52 load avg: 13.34 [41/42] test_multiprocessing_forkserver.test_processes passed (1 min 41 sec) -- running: test_multiprocessing_spawn.test_processes (1 min 34 sec)
0:02:22 load avg: 8.84 running: test_multiprocessing_spawn.test_processes (2 min 4 sec)
0:02:22 load avg: 8.84 [42/42] test_multiprocessing_spawn.test_processes passed (2 min 3 sec)

== Tests result: SUCCESS ==

40 tests OK.

10 slowest tests:
- test_multiprocessing_spawn.test_processes: 2 min 3 sec
- test_multiprocessing_forkserver.test_processes: 1 min 41 sec
- test_multiprocessing_spawn.test_misc: 1 min 22 sec
- test_multiprocessing_spawn.test_manager: 1 min 2 sec
- test_multiprocessing_forkserver.test_misc: 54.3 sec
- test_asyncio.test_tasks: 52.9 sec
- test_multiprocessing_forkserver.test_manager: 45.5 sec
- test_multiprocessing_fork.test_processes: 43.4 sec
- test_asyncio.test_events: 39.3 sec
- test_multiprocessing_fork.test_misc: 26.3 sec

2 tests skipped:
    test_asyncio.test_windows_events test_asyncio.test_windows_utils

Total duration: 2 min 22 sec

gpshead · 2023-08-24T15:43:44Z


-def install_tests_in_module_dict(remote_globs, start_method):
+def install_tests_in_module_dict(remote_globs, start_method,
+                                 only_type=None, exclude_types=False):


This function needs proper documentation as to what it does and how it behaves. The new parameters behaviors are unobvious.

miss-islington · 2023-08-24T17:35:28Z

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

bedevere-bot · 2023-08-24T17:35:38Z

GH-108442 is a backport of this pull request to the 3.12 branch.

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>

gh-108388: Split test_multiprocessing_spawn (GH-108396) Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>

miss-islington · 2023-09-21T19:37:07Z

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>

bedevere-app · 2023-09-21T19:37:18Z

GH-109688 is a backport of this pull request to the 3.11 branch.

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>

gh-108388: Split test_multiprocessing_spawn (GH-108396) Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>

bedevere-bot mentioned this pull request Aug 24, 2023

Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Closed

bedevere-bot added the awaiting core review label Aug 24, 2023

vstinner merged commit aa9a359 into python:main Aug 24, 2023

bedevere-bot removed the awaiting core review label Aug 24, 2023

vstinner deleted the test_mp_packages branch August 24, 2023 03:50

vstinner mentioned this pull request Aug 24, 2023

gh-108388: test_concurrent_futures requires cpu #108389

Closed

gpshead reviewed Aug 24, 2023

View reviewed changes

gpshead added the needs backport to 3.12 only security fixes label Aug 24, 2023

bedevere-bot removed the needs backport to 3.12 only security fixes label Aug 24, 2023

vstinner added the needs backport to 3.11 only security fixes label Sep 21, 2023

bedevere-app Bot removed the needs backport to 3.11 only security fixes label Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gh-108388: Split test_multiprocessing_spawn#108396

gh-108388: Split test_multiprocessing_spawn#108396
vstinner merged 1 commit into
python:mainfrom
vstinner:test_mp_packages

vstinner commented Aug 24, 2023 •

edited by bedevere-bot

Loading

Uh oh!

vstinner commented Aug 24, 2023

Uh oh!

gpshead Aug 24, 2023

Uh oh!

miss-islington commented Aug 24, 2023

Uh oh!

bedevere-bot commented Aug 24, 2023

Uh oh!

miss-islington commented Sep 21, 2023

Uh oh!

bedevere-app Bot commented Sep 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

vstinner commented Aug 24, 2023 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Aug 24, 2023

Uh oh!

gpshead Aug 24, 2023

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Aug 24, 2023

Uh oh!

bedevere-bot commented Aug 24, 2023

Uh oh!

miss-islington commented Sep 21, 2023

Uh oh!

bedevere-app Bot commented Sep 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vstinner commented Aug 24, 2023 •

edited by bedevere-bot

Loading