🚀 Multiprocessing in Python
Multiprocessing allows you to create programs that can run multiple processes simultaneously. Unlike threading, multiprocessing bypasses the Global Interpreter Lock (GIL) by using separate memory spaces, making it ideal for CPU-bound tasks.
🧠 Why Multiprocessing?
- True Parallelism: Utilizes multiple CPU cores.
- Bypasses GIL: Each process has its own Python interpreter and memory space.
- Stability: A crash in one process doesn't kill the others.
🛠 Basic Usage
The multiprocessing module API is very similar to threading.
import multiprocessing
import time
def worker(name):
print(f"Worker {name} starting")
time.sleep(2)
print(f"Worker {name} exiting")
if __name__ == '__main__':
p1 = multiprocessing.Process(target=worker, args=('A',))
p2 = multiprocessing.Process(target=worker, args=('B',))
p1.start()
p2.start()
p1.join()
p2.join()
📨 Communication Between Processes
Since processes have separate memory, they cannot share global variables easily. You must use IPC (Inter-Process Communication).
📬 Queues
Thread and process safe queues.
def producer(q):
q.put('Hello')
def consumer(q):
print(q.get())
if __name__ == '__main__':
q = multiprocessing.Queue()
p1 = multiprocessing.Process(target=producer, args=(q,))
p2 = multiprocessing.Process(target=consumer, args=(q,))
p1.start(); p2.start()
p1.join(); p2.join()
🚰 Pipes
A two-way connection between two processes.
def sender(conn):
conn.send("Message from sender")
conn.close()
if __name__ == '__main__':
parent_conn, child_conn = multiprocessing.Pipe()
p = multiprocessing.Process(target=sender, args=(child_conn,))
p.start()
print(parent_conn.recv())
p.join()
💾 Shared Memory
For performance, you can share memory using Value or Array.
def increment(n):
n.value += 1
if __name__ == '__main__':
num = multiprocessing.Value('i', 0)
p = multiprocessing.Process(target=increment, args=(num,))
p.start()
p.join()
print(num.value)
🏊 Process Pools
The Pool class offers a convenient means of parallelizing the execution of a function across multiple input values.
def square(x):
return x * x
if __name__ == '__main__':
with multiprocessing.Pool(processes=4) as pool:
results = pool.map(square, range(10))
print(results)
📝 Summary
- ✅ CPU-Bound: Best for heavy computations.
- ✅ Isolation: Processes are independent.
- ✅ IPC: Use Queues or Pipes to talk between processes.
- ✅ Overhead: Creating processes is heavier than creating threads.
Created with ❤️ by Pynfinity
