Python Multiprocessing Module With Example

Today, in this Python tutorial, we will see Python Multiprocessing. Moreover, we will look at the package and structure of Multiprocessing in Python.

Also, we will discuss process class in Python Multiprocessing and also get information about the process.

To make this happen, we will borrow several methods from the multithreading module.

Along with this, we will learn lock and pool class Python Multiprocessing. 

So, let’s begin the Python Multiprocessing tutorial.

Python Multiprocessing Module With Example

What is Python Multiprocessing?

First, let’s talk about parallel processing. This is a way to simultaneously break up and run program tasks on multiple microprocessors.

In effect, this is an effort to reduce processing time and is something we can achieve with a computer with two or more processors or using a computer network. We also call this parallel computing.

Okay, now coming to Python Multiprocessing, this is a way to improve performance by creating parallel code.

CPU manufacturers make this possible by adding more cores to their processors. In a multiprocessing system, applications break into smaller routines to run independently.

Take a look at a single processor system. Given several processes at once, it struggles to interrupt and switch between tasks.

How would you do being the only chef in a kitchen with hundreds of customers to manage? You would have to be the one to execute every single routine task from baking to kneading the dough. 

Python Multiprocessing Package

Multiprocessing in Python is a package we can use with Python to spawn processes using an API that is much like the threading module.

With support for both local and remote concurrency, it lets the programmer make efficient use of multiple processors on a given machine.

Before we can begin explaining it to you, let’s take an example of Pool- an object, a way to parallelize executing a function across input values and distributing input data across processes.

This is data parallelism (Make a module out of this and run it)-

from multiprocessing import Pool
def f(x):
    return x*x
with Pool(5) as p:
    print(p.map(f,[1,2,3]))

Output

Want to find out how many cores your machine has? Try the cpu_count() method.

>>> import multiprocessing
>>> multiprocessing.cpu_count()

Output

Structure of a Python Multiprocessing System

So what is such a system made of? We have the following possibilities:

• A multiprocessor- a computer with more than one central processor.
• A multi-core processor- a single computing component with more than one independent actual processing units/ cores.

In either case, the CPU is able to execute multiple tasks at once assigning a processor to each task.

Python Multiprocessing Process Class

Let’s talk about the Process class in Python Multiprocessing first. This is an abstraction to set up another process and lets the parent application control execution.

Here, we observe the start() and join() methods. Let’s first take an example.

import multiprocessing
from multiprocessing import Process
def testing():
      print("Works")
def square(n):
       print("The number squares to ",n**2)
def cube(n):
       print("The number cubes to ",n**3)
if __name__=="__main__":
   p1=Process(target=square,args=(7,))
   p2=Process(target=cube,args=(7,))
   p3=Process(target=testing)
   p1.start()
   p2.start()
   p3.start()
   p1.join()
   p2.join()
   p3.join()
   print("We're done")

We saved this as pro.py on our desktop and then ran it twice from the command line. This is the output we got:

Process class in Python Multiprocessing

Python Multiprocessing – Process class

Let’s understand this piece of code. Process() lets us instantiate the Process class. start() tells Python to begin processing.

But then if we let it be, it consumes resources and we may run out of those at a later point in time. This is because it lets the process stay idle and not terminate.

To avoid this, we make a call to join(). With this, we don’t have to kill them manually. Join stops execution of the current program until a process completes.

This makes sure the program waits for p1 to complete and then p2 to complete. Then, it executes the next statements of the program.

One last thing, the args keyword argument lets us specify the values of the argument to pass. Also, target lets us select the function for the process to execute.

Getting Information about Processes in Python

1. Getting Process ID and checking if alive

We may want to get the ID of a process or that of one of its child. We may also want to find out if it is still alive. The following program demonstrates this functionality:

import multiprocessing
from multiprocessing import Process
import os
def child1():
     print("Child 1",os.getpid())
def child2():
        print("Child 2",os.getpid())
if __name__=="__main__":
   print("Parent ID",os.getpid())
   p1=Process(target=child1)
   p2=Process(target=child2)
   p1.start()
   p2.start()
   p1.join()
   alive='Yes' if p1.is_alive() else 'No'
   print("Is p1 alive?",alive)
   alive='Yes' if p2.is_alive() else 'No'
   print("Is p2 alive?",alive)
   p2.join()
   print("We're done")

Getting Information

Getting Information

In Python multiprocessing, each process occupies its own memory space to run independently. It terminates when the target function is done executing.

2. Getting Process Name

We can also set names for processes so we can retrieve them when we want. This is to make it more human-readable.

import multiprocessing
from multiprocessing import Process, current_process
import os
def child1():
     print(current_process().name)
def child2():
         print(current_process().name)
if __name__=="__main__":
   print("Parent ID",os.getpid())
   p1=Process(target=child1,name='Child 1')
   p2=Process(target=child2,name='Child 2')
   p1.start()
   p2.start()
   p1.join()
   p2.join()
   print("We're done")

Python Multiprocessing – Process name

As you can see, the current_process() method gives us the name of the process that calls our function. See what happens when we don’t assign a name to one of the processes:

import multiprocessing
from multiprocessing import Process, current_process
import os
def child1():
        print(current_process().name)
def child2():
        print(current_process().name)
if __name__=="__main__":
   print("Parent ID",os.getpid())
   p1=Process(target=child1)
   p2=Process(target=child2,name='Child 2')
   p1.start()
   p2.start()
   p1.join()
   p2.join()
   print("We're done")

Getting process name

Well, the Python Multiprocessing Module assigns a number to each process as a part of its name when we don’t.

Python Multiprocessing Lock

Just like the threading module, multiprocessing in Python supports locks. The process involves importing Lock, acquiring it, doing something, and then releasing it. Let’s take a look.

In the following piece of code, we make a process acquire a lock while it does its job.

from multiprocessing import Process, Lock
lock=Lock()
def printer(item):
  lock.acquire()
  try:
      print(item)
  finally:
      lock.release()
if __name__=="__main__":
  items=['nacho','salsa',7]
  for item in items:
     p=Process(target=printer,args=(item,))
     p.start()

Let’s run this code thrice to see what different outputs we get.

The lock doesn’t let the threads interfere with each other. The next process waits for the lock to release before it continues.

Output

Python Multiprocessing Pool Class

This class represents a pool of worker processes; its methods let us offload tasks to such processes. Let’s take an example (Make a module out of this and run it).

from multiprocessing import Pool
def double(n):
   return n*2
if __name__=='__main__':
   nums=[2,3,6]
   pool=Pool(processes=3)
   print(pool.map(double,nums))

Output

We create an instance of Pool and have it create a 3-worker process. map() maps the function double and an iterable to each process. The result gives us [4,6,12].

Another method that gets us the result of our processes in a pool is the apply_async() method.

from multiprocessing import Pool
def double(n):
   return n*2
if __name__=='__main__':
   pool=Pool(processes=3)
   result=pool.apply_async(double,(7,))
   print(result.get(timeout=1))

Output

So, this was all in Python Multiprocessing. Hope you like our explanation.

Python Interview Questions on Multiprocessing Module

1. What is Python multiprocessing module?
2. How does multiprocessing work in Python?
3. How to stop multiprocessing in Python?
4. Explain the purpose for using multiprocessing module in Python.
5. Is multiprocessing faster than multithreading in Python?

Conclusion

Hence, in this Python Multiprocessing Tutorial, we discussed the complete concept of Multiprocessing in Python.

Moreover, we looked at Python Multiprocessing pool, lock, and processes. Now, you have an idea of how to utilize your processors to their full potential. 

Feel free to explore other blogs on Python attempting to unleash its power. See you again.