Python Multiprocessing Module With Example
Free Python course with 25 projects (coupon code: DATAFLAIR_PYTHON) Start Now
1. Python Multiprocessing – Objective
Today, in this Python tutorial, we will see Python Multiprocessing. Moreover, we will look at the package and structure of Multiprocessing in Python. Also, we will discuss process class in Python Multiprocessing and also get information about the process. To make this happen, we will borrow several methods from the multithreading module. Along with this, we will learn lock and pool class Python Multiprocessing.
So, let’s begin the Python Multiprocessing tutorial.
2. What is Python Multiprocessing?
First, let’s talk about parallel processing. This is a way to simultaneously break up and run program tasks on multiple microprocessors. In effect, this is an effort to reduce processing time and is something we can achieve with a computer with two or more processors or using a computer network. We also call this parallel computing.
You must learn about Python Modules
Okay, now coming to Python Multiprocessing, this is a way to improve performance by creating parallel code. CPU manufacturers make this possible by adding more cores to their processors. In a multiprocessing system, applications break into smaller routines to run independently. Take a look at a single processor system. Given several processes at once, it struggles to interrupt and switch between tasks. How would you do being the only chef in a kitchen with hundreds of customers to manage? You would have to be the one to execute every single routine task from baking to kneading the dough.
a. Python Multiprocessing Package
Multiprocessing in Python is a package we can use with Python to spawn processes using an API that is much like the threading module. With support for both local and remote concurrency, it lets the programmer make efficient use of multiple processors on a given machine. Before we can begin explaining it to you, let’s take an example of Pool- an object, a way to parallelize executing a function across input values and distributing input data across processes. This is data parallelism (Make a module out of this and run it)-
from multiprocessing import Pool def f(x): return x*x with Pool(5) as p: print(p.map(f,[1,2,3]))
[1, 4, 9]
Want to find out how many cores your machine has? Try the cpu_count() method.
>>> import multiprocessing >>> multiprocessing.cpu_count()
b. Structure of a Python Multiprocessing System
So what is such a system made of? We have the following possibilities:
- A multiprocessor- a computer with more than one central processor.
- A multi-core processor- a single computing component with more than one independent actual processing units/ cores.
Do you know about Python Library
In either case, the CPU is able to execute multiple tasks at once assigning a processor to each task.
3. Python Multiprocessing Process Class
Let’s talk about the Process class in Python Multiprocessing first. This is an abstraction to set up another process and lets the parent application control execution. Here, we observe the start() and join() methods. Let’s first take an example.
import multiprocessing from multiprocessing import Process def testing(): print("Works") def square(n): print("The number squares to ",n**2) def cube(n): print("The number cubes to ",n**3) if __name__=="__main__": p1=Process(target=square,args=(7,)) p2=Process(target=cube,args=(7,)) p3=Process(target=testing) p1.start() p2.start() p3.start() p1.join() p2.join() p3.join() print("We're done")
We saved this as pro.py on our desktop and then ran it twice from the command line. This is the output we got:
Let’s revise Python Class and object
Let’s understand this piece of code. Process() lets us instantiate the Process class. start() tells Python to begin processing. But then if we let it be, it consumes resources and we may run out of those at a later point in time. This is because it lets the process stay idle and not terminate. To avoid this, we make a call to join(). With this, we don’t have to kill them manually. Join stops execution of the current program until a process completes. This makes sure the program waits for p1 to complete and then p2 to complete. Then, it executes the next statements of the program. One last thing, the args keyword argument lets us specify the values of the argument to pass. Also, target lets us select the function for the process to execute.
4. Getting Information about Processes
a. Getting Process ID and checking if alive
We may want to get the ID of a process or that of one of its child. We may also want to find out if it is still alive. The following program demonstrates this functionality:
import multiprocessing from multiprocessing import Process import os def child1(): print("Child 1",os.getpid()) def child2(): print("Child 2",os.getpid()) if __name__=="__main__": print("Parent ID",os.getpid()) p1=Process(target=child1) p2=Process(target=child2) p1.start() p2.start() p1.join() alive='Yes' if p1.is_alive() else 'No' print("Is p1 alive?",alive) alive='Yes' if p2.is_alive() else 'No' print("Is p2 alive?",alive) p2.join() print("We're done")
In Python multiprocessing, each process occupies its own memory space to run independently. It terminates when the target function is done executing.
Have a look at Python Data Structures
b. Getting Process Name
We can also set names for processes so we can retrieve them when we want. This is to make it more human-readable.
import multiprocessing from multiprocessing import Process, current_process import os def child1(): print(current_process().name) def child2(): print(current_process().name) if __name__=="__main__": print("Parent ID",os.getpid()) p1=Process(target=child1,name='Child 1') p2=Process(target=child2,name='Child 2') p1.start() p2.start() p1.join() p2.join() print("We're done")
As you can see, the current_process() method gives us the name of the process that calls our function. See what happens when we don’t assign a name to one of the processes:
import multiprocessing from multiprocessing import Process, current_process import os def child1(): print(current_process().name) def child2(): print(current_process().name) if __name__=="__main__": print("Parent ID",os.getpid()) p1=Process(target=child1) p2=Process(target=child2,name='Child 2') p1.start() p2.start() p1.join() p2.join() print("We're done")
Well, the Python Multiprocessing Module assigns a number to each process as a part of its name when we don’t.
5. Python Multiprocessing Lock
Just like the threading module, multiprocessing in Python supports locks. The process involves importing Lock, acquiring it, doing something, and then releasing it. Let’s take a look.
Do you know about Python Dictionaries
In the following piece of code, we make a process acquire a lock while it does its job.
from multiprocessing import Process, Lock lock=Lock() def printer(item): lock.acquire() try: print(item) finally: lock.release() if __name__=="__main__": items=['nacho','salsa',7] for item in items: p=Process(target=printer,args=(item,)) p.start()
Let’s run this code thrice to see what different outputs we get.
The lock doesn’t let the threads interfere with each other. The next process waits for the lock to release before it continues.
6. Python Multiprocessing Pool Class
This class represents a pool of worker processes; its methods let us offload tasks to such processes. Let’s take an example (Make a module out of this and run it).
from multiprocessing import Pool def double(n): return n*2 if __name__=='__main__': nums=[2,3,6] pool=Pool(processes=3) print(pool.map(double,nums))
[4, 6, 12]
We create an instance of Pool and have it create a 3-worker process. map() maps the function double and an iterable to each process. The result gives us [4,6,12].
Another method that gets us the result of our processes in a pool is the apply_async() method.
Let’s take a tour to Python Strings
from multiprocessing import Pool def double(n): return n*2 if __name__=='__main__': pool=Pool(processes=3) result=pool.apply_async(double,(7,)) print(result.get(timeout=1))
So, this was all in Python Multiprocessing. Hope you like our explanation.
7. Conclusion – Python Multiprocessing
Hence, in this Python Multiprocessing Tutorial, we discussed the complete concept of Multiprocessing in Python. Moreover, we looked at Python Multiprocessing pool, lock, and processes. Now, you have an idea of how to utilize your processors to their full potential. Feel free to explore other blogs on Python attempting to unleash its power. See you again.
See also –
Python Modules vs Packages