Sławomir Sawicki
19 December 2023, 15 min read
What's inside
- Harnessing Python's Hidden Treasures: Modules
- Here are 30 built-in Python modules you can use in your Python project:
- Final Remarks
- Contact Us
Harnessing Python's Hidden Treasures: Modules
Python's ascension in the programming realm isn't just about simplicity — it's about power. At its heart lies an array of modules and libraries, each tailored to make a developer's life effortless.
These tools aren't mere conveniences but the backbone of efficient, reliable, and scalable code. Whether you're piecing together a vast script or seeking reusable solutions, Python's modules have you covered. Dive in as I unravel the magic behind them and discover why they're the unsung heroes of Python development.
Here are 30 built-in Python modules you can use in your Python project:
abc
This tiny module delivers the environment you need for creating abstract base classes.
Example:
>>> from abc import ABCMeta, abstractmethod
>>> class MyAbstractClass(metaclass=ABCMeta):
... @abstractmethod
... def x(self):
... pass
... @abstractmethod
... def y(self):
... pass
...
>>> obj = MyAbstractClass()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyAbstractClass with abstract methods x, y
>>> class MyCustomClass(MyAbstractClass):
... def x(self):
... return 'x'
...
>>> obj = MyCustomClass()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyCustomClass with abstract methods y
>>> class MyCustomClass(MyAbstractClass):
... def x(self):
... return 'x'
... def y(self):
... return 'y'
...
>>> obj = MyCustomClass()
>>> obj.x(), obj.y()
('x', 'y')
argparse
This module contains tools that make it easier to create user-friendly interfaces from the level of the command line.
Example:
>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.exit = lambda x='', y='': print(x, y) # We don't want to exit python shell
>>> _ = parser.add_argument('foo', help='help text for `foo`')
>>> _ = parser.add_argument('-b', '--bar', help='help text for `--bar`')
>>> parser.parse_args(['-h'])
usage: [-h] [-b BAR] foo foo
positional arguments:
foo help text for `foo`
foo help text for `foo`
optional arguments:
-h, --help show this help message and exit
-b BAR, --bar BAR help text for `--bar`
usage: [-h] [-b BAR] foo foo
2 : error: the following arguments are required: foo, foo
Namespace(bar=None, foo=None)
>>> parser.parse_args()
usage: [-h] [-b BAR] foo foo
2 : error: the following arguments are required: foo, foo
Namespace(bar=None, foo=None)
>>> parser.parse_args('example.py foo_data'.split(' '))
Namespace(bar=None, foo='foo_data')
>>> parser.parse_args('example.py foo_data --bar bar_data'.split(' '))
Namespace(bar='bar_data', foo='foo_data')
>>> result = parser.parse_args('example.py foo_data --bar bar_data'.split(' '))
>>> result.foo
'foo_data'
>>> result.bar
'bar_data'
asyncio
This is a huge module that delivers the framework and environment for asynchronous programming. It was added to Python 3.4 as a temporary module and worked as an alternative to the popular framework called Twisted. In short, asyncio is handy if you want to create asynchronous, concurrent, and single-threaded code. The module launches a loop where the asynchronous code is executed as a task. The module executes another task when one task is inactive (for example, waiting for server response).
Example:
>>> import asyncio
>>> async def printer(msg, sleep):
... await asyncio.sleep(sleep)
... print (msg)
...
>>> task0 = asyncio.ensure_future(printer('task0 -> sleep 5', 5))
>>> task1 = asyncio.ensure_future(printer('task1 -> sleep 1', 1))
>>> task2 = asyncio.ensure_future(printer('task2 -> sleep 3', 3))
>>> future = asyncio.gather(task0, task1, task2)
>>> future.add_done_callback(lambda x: asyncio.get_event_loop().stop())
>>> loop = asyncio.get_event_loop()
>>> loop.run_forever()
task1 -> sleep 1
task2 -> sleep 3
task0 -> sleep 5
>>> future.done()
True
base64
This well-known module delivers a tool for coding and decoding binary code into a format that can be displayed - and the other way round.
Example:
>>> import base64
>>> data = base64.b64encode(b'ASCII charakters to be encoded')
>>> data
b'QVNDSUkgY2hhcmFrdGVycyB0byBiZSBlbmNvZGVk'
>>> data = base64.b64decode(data)
>>> data
b'ASCII charakters to be encoded'
collections
This module offers specialized container data types that work as an alternative to basic contained types for general purposes such as dict, list, set and tuple.
Example:
>>> from collections import defaultdict
>>> data, ddata = dict(), defaultdict(int)
>>> data, ddata
({}, defaultdict(<class 'int'>, {}))
>>> ddata['key'] += 5
>>> data['key'] += 5
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'key'
>>> data, ddata
({}, defaultdict(<class 'int'>, {'key': 5}))
copy
Everyone uses this tiny module that contains tools for deep copying of container-type data. Its most famous function is deepcopy - if not for this function, copying lists and dictionaries would become torture for developers.
Example:
>>> from copy import deepcopy
>>> data = {'key0': {'key1': False}}
>>> cdata = data.copy()
>>> data, cdata
({'key0': {'key1': False}}, {'key0': {'key1': False}})
>>> cdata['key0']['key1'] = True
>>> data, cdata
({'key0': {'key1': True}}, {'key0': {'key1': True}})
>>> data = {'key0': {'key1': False}}
>>> dcdata = deepcopy(data)
>>> data, dcdata
({'key0': {'key1': False}}, {'key0': {'key1': False}})
>>> dcdata['key0']['key1'] = True
>>> data, dcdata
({'key0': {'key1': False}}, {'key0': {'key1': True}})
csv
Delivers functionalities for exporting and importing tabular data in CSV format. The module allows developers to say “load data” or “save data” from a CSV file.
Example:
>>> import csv
>>> with open('example.csv', 'w', newline='') as csvfile:
... writer = csv.writer(csvfile)
... writer.writerow(['Column A', 'Column B', 'Column C'])
... writer.writerow(['Cell 0 A', 'Cell 0 B', 'Cell 0 C'])
... writer.writerow(['Cell 1 A', 'Cell 1 B', 'Cell 1 C'])
...
28
28
28
>>> with open('example.csv', newline='') as csvfile:
... reader = csv.reader(csvfile)
... print('\n'.join([' | '.join(row) for row in reader]))
...
Column A | Column B | Column C
Cell 0 A | Cell 0 B | Cell 0 C
Cell 1 A | Cell 1 B | Cell 1 C
datetime
A simple and one of the most popular modules in Python. It delivers tools that make it easier to work with dates and times. The most popular classes are datetime, timezone, and timedelta, but date,time, and tzinfo can also be helpful.
Example:
>>> from datetime import datetime, timedelta, timezone
>>> obj = datetime.now()
>>> obj
datetime.datetime(2018, 10, 7, 12, 27, 25, 527961)
>>> obj += timedelta(days=7)
>>> obj
datetime.datetime(2018, 10, 14, 12, 27, 25, 527961)
>>> obj.astimezone(timezone.utc)
datetime.datetime(2018, 10, 14, 10, 27, 25, 527961, tzinfo=datetime.timezone.utc)
decimal
The module delivers a data type called Decimal. Its main advantage is the correct rounding of decimal numbers which is extremely important in billing systems - even a slight distortion with several hundred thousand operations can change the final result significantly.
Example:
>>> import decimal
>>> float(0.1 + 0.2)
0.30000000000000004
>>> float(decimal.Decimal('0.1') + decimal.Decimal('0.2'))
0.3
functools
The functools module comes in handy for higher-order functions, i.e., the functions that act on or return other functions. You can treat any callable object as a function for this module.
Example:
>>> from functools import wraps
>>> def example_decorator(func):
... @wraps(func)
... def wrapper(*args, **kwargs):
... print("Print from decorator")
... return func(*args, **kwargs)
... return wrapper
...
>>> @example_decorator
... def example_func():
... """Func docstring"""
... print("Print from func")
...
>>> example_func()
Print from decorator
Print from func
>>> example_func.__name__
'example_func'
>>> example_func.__doc__
'Func docstring'
>>> from functools import lru_cache
>>> @lru_cache()
... def example_func(n):
... return n * n
...
>>> [example_func(n) for n in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> example_func.cache_info()
CacheInfo(hits=0, misses=10, maxsize=128, currsize=10)
>>> [example_func(n) for n in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> example_func.cache_info()
CacheInfo(hits=10, misses=10, maxsize=128, currsize=10)
hashlib
This handy module implements a standard interface to numerous secure hash and message digest algorithms like the FIPS secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2), as well as RSA’s MD5 algorithm (defined in Internet RFC 1321). Note the terminology: “secure hash” and “message digest” are interchangeable.
Example:
>>> import hashlib
>>> obj = hashlib.sha256()
>>> obj.update(b"Example text for hash")
>>> obj.digest()
b'\xc0\xf4\xad\x1a\xf5IK\x9a\x17x\xf6"\xfagG\x92\xe1\x0c\x8f\xef\xe2\x99\xfa\x97\x12Q#\xe7Za\xb3\xe2'
>>> obj.hexdigest()
'c0f4ad1af5494b9a1778f622fa674792e10c8fefe299fa97125123e75a61b3e2'
http
This package collects several modules for working with the HyperText Transfer Protocol, such as http.client (low-level HTTP protocol client), http.server (includes basic HTTP server classes based on socketserver), http.cookies (with utilities for implementing state management with cookies) and http.cookiejar (that provides persistence of cookies).
Example:
>>> import http.client, http.server, http.cookies, http.cookiejar
>>> conn = http.client.HTTPSConnection("sunscrapers.com")
>>> conn.request("GET", "/")
>>> resp = conn.getresponse()
>>> resp.status
200
>>> resp.readlines()[2:5]
[b'<!DOCTYPE HTML>\n', b'<html>\n', b'<head>\n']
>>> conn.close()
>>> http.HTTPStatus
<enum 'HTTPStatus'>
>>> http.client
<module 'http.client' from '/usr/lib/python3.11/http/client.py'>
>>> http.server
<module 'http.server' from '/usr/lib/python3.11/http/server.py'>
>>> http.cookies
<module 'http.cookies' from '/usr/lib/python3.11/http/cookies.py'>
>>> http.cookiejar
<module 'http.cookiejar' from '/usr/lib/python3.11/http/cookiejar.py'>
importlib
The package provides the implementation of the import statement (and the import() function) in Python source code. The components to implement import are exposed, making it easier for developers to create custom objects to participate in the import process.
Example:
>>> import importlib
>>> importlib.sys
<module 'sys' (built-in)>
>>> importlib.import_module("this")
The Zen of Python, by Tim Peters
...
<module 'this' from '/usr/lib/python3.11/this.py'>
itertools
This useful module implements iterator building blocks inspired by constructs from APL, Haskell, and SML (all in a form that matches Python programs). It standardizes a core collection of quick, memory-efficient tools you can use on their own or combine to construct specialized tools efficiently in pure Python.
Example:
>>> import itertools
>>> "-".join(list(itertools.chain("ABC", "DEF")))
'A-B-C-D-E-F'
>>> list(itertools.combinations("ABC", 2))
[('A', 'B'), ('A', 'C'), ('B', 'C')]
inspect
The module offers several useful functions that help developers get information about live objects such as modules, classes, methods, functions, tracebacks, frame objects, and code objects in Python code. It provides four main functions: type checking, getting source code, inspecting classes and functions, and examining the interpreter stack. You can use it to retrieve the source code of a method, examine the contents of a class, or just get all the relevant information to display a detailed traceback.
Example:
>>> import inspect
>>> inspect.getmodulename(inspect.__file__)
'Inspect'
>>> inspect.isfunction(lambda _: _)
True
>>> inspect.signature(lambda _: _)
<Signature (_)>
json
JSON (JavaScript Object Notation) is a lightweight data-interchange format that was inspired by JavaScript object literal syntax. The module json exposes an API that looks similar to the standard library marshal and pickle modules.
Example:
>>> import json
>>> data = json.dumps({"A": 1, "B": True, "C": [1, 2, 3]})
>>> data
'{"A": 1, "B": true, "C": [1, 2, 3]}'
>>> json.loads(data)
{'A': 1, 'B': True, 'C': [1, 2, 3]}
logging
This module defines functions and classes that provide a flexible event-logging system for applications and libraries. It’s a good idea to use it because it ensures that all Python modules can log in - your application log may include your messages integrated with messages from third-party modules.
Example:
>>> import logging
>>> FORMAT = '[%(asctime)-15s] [%(name)s] [%(levelname)s] - %(message)s'
>>> logging.basicConfig(format=FORMAT)
>>> logger = logging.getLogger("MY_LOGGER")
>>> logger.error("My error message")
[2023-10-29 09:22:10,880] [MY_LOGGER] [ERROR] - My error message
math
This module gives developers access to the mathematical functions implemented directly in the C language. You can’t use them with complex numbers, which is good if you’re not looking to learn a lot of mathematics required to understand complex problems.
Example:
>>> import math
>>> math.pow(2, 4)
16.0
>>> math.sqrt(16)
4.0
>>> math.floor(5/2)
2
>>> math.ceil(5/2)
3
multiprocessing
This handy package supports spawning processes with the help of an API similar to the threading module. It provides local and remote concurrency using sub processes instead of threads to side-step the Global Interpreter Lock. Developers use it to take full advantage of multiple processors on a given machine. It runs on both Unix and Windows.
Example:
>>> from multiprocessing import Process
>>> def my_function():
... count = 0
... while True:
... print ("My function: {0}".format(count))
... count += 1
... time.sleep(1)
...
>>> proc = Process(target=my_function, args=())
>>> proc.start()
>>> My function: 0
My function: 1
My function: 2
...
os
This module offers a portable method of using operating system-dependent functionality.
Example:
>>> import os
>>> os.getpid()
7171
>>> os.name
'posix'
>>> os.environ['_']
'/usr/bin/python3.11'
>>> os.path.join('dir0', 'dir1', 'file.py')
'dir0/dir1/file.py'
pdb
This module defines an interactive source code debugger. It supports single stepping at the source line level, setting (conditional) breakpoints, an inspection of stack frames, source code listing, and more.
Example:
slawek@swiegy:/tmp$ echo "print('Hello World')" > test.py
slawek@swiegy:/tmp$ python test.py
Hello World
slawek@swiegy:/tmp$ echo "import pdb;pdb.set_trace()" > test.py
slawek@swiegy:/tmp$ python test.py
--Return--
> /tmp/test.py(1)<module>()->None
-> import pdb;pdb.set_trace()
(Pdb)
random
This useful module implements pseudo-random number generators for various distributions. For example, you get a uniform selection from a range of integers. For sequences, there is a uniform selection of a random element (as well as a function to generate a random permutation of a list in place and for random sampling without replacement).
Example:
>>> import random
>>> random.randint(0,100)
20
>>> random.randint(0,100)
42
>>> random.random()
0.6609538830840114
>>> random.random()
0.486448371691216
re
This module provides regular expression matching operations similar to those you get in Perl. You can search Unicode and 8-bit strings, but they can’t be mixed.
Example:
>>> import re
>>> res = re.search('(?<=abc)def', 'abcdef')
>>> res.group(0)
'def'
>>> bool(res)
True
>>> res = re.search('(?<=abc)def', 'xyzdef')
>>> bool(res)
False
Read more: What every Pythonista should know about Unicode
shutil
The module provides several high-level operations on files and collections of files, mainly functions that support file copying and removal.
Example:
>>> import shutil
>>> import os
>>> os.listdir()
[]
>>> open('test.py', 'a+').close()
>>> os.listdir()
['test.py']
>>> shutil.copyfile('test.py', 'test2.py')
'test2.py'
>>> os.listdir()
['test2.py', 'test.py']
sys
The modul offers access to variables used or maintained by the interpreter and functions that interact strongly with the interpreter.
Example:
>>> import sys
>>> sys.argv
['']
>>> sys.modules.keys()
dict_keys(['sys', 'builtins', '_frozen_importlib', '_imp', '_thread', '_warnings', '_weakref', 'zipimport', '_frozen_importlib_external', '_io', 'marshal', 'posix', 'encodings', 'codecs', '_codecs', 'encodings.aliases', 'encodings.utf_8', '_signal', '__main__', 'encodings.latin_1', 'io', 'abc', '_abc', 'site', 'os', 'stat', '_stat', 'posixpath', 'genericpath', 'os.path', '_collections_abc', '_sitebuiltins', '_bootlocale', '_locale', 'types', 'importlib', 'importlib._bootstrap', 'importlib._bootstrap_external', 'warnings', 'importlib.util', 'importlib.abc', 'importlib.machinery', 'contextlib', 'collections', 'operator', '_operator', 'keyword', 'heapq', '_heapq', 'itertools', 'reprlib', '_collections', 'functools', '_functools', 'zope', 'sitecustomize', 'apport_python_hook', 'readline', 'atexit', 'rlcompleter', 'shutil', 'fnmatch', 're', 'enum', 'sre_compile', '_sre', 'sre_parse', 'sre_constants', 'copyreg', 'errno', 'zlib', 'bz2', '_compression', 'threading', 'time', 'traceback', 'linecache', 'tokenize', 'token', '_weakrefset', '_bz2', 'lzma', '_lzma', 'pwd', 'grp'])
>>> sys.path
['', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages']
threading
This useful module builds higher-level threading interfaces on top of the lower-level _thread module.
Example:
>>> import threading
>>> def worker(num):
... print ("Worker num: {0}".format(num))
...
>>> for i in range(4):
... threading.Thread(target=worker, args=(i,)).start()
...
Worker num: 0
Worker num: 1
Worker num: 2
Worker num: 3
types
The module defines utility functions that support the dynamic creation of new types. It defines names for object types used by the standard Python interpreter but not exposed as builtins. It also offers additional type-related utility classes and functions that aren’t builtins.
Example:
>>> import types
>>> def gen():
... yield True
...
>>> isinstance(gen(), types.GeneratorType)
True
>>> import logging
>>> logging
<module 'logging' from '/usr/lib/python3.11/logging/__init__.py'>
>>> types.ModuleType(“logging”)
<module 'logging'>
unittest
Originally inspired by JUnit, the module works similarly to major unit testing frameworks in other programming languages. It supports a broad range of functions: test automation, sharing of setup, shutdown code for tests, test aggregation into collections, and more.
Example:
>>> import unittest
>>> class TestStringMethods(unittest.TestCase):
... def test_example(self):
... self.assertTrue(True, True)
...
>>> unittest.main()
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
urllib
This handy package collects several modules for working with URLs: urllib.request (opening and reading URLs), urllib.error (includes exceptions raised by urllib.request), urllib.parse (for parsing URLs), and urllib.robotparser (for parsing robots.txt files).
Example:
>>> from urllib.parse import urlparse
>>> urlparse('https://sunscrapers.com/aboutus/')
ParseResult(scheme='https', netloc='sunscrapers.com', path='/aboutus/', params='', query='', fragment='')
uuid
The module provides immutable UUID objects, as well as the following functions for generating version 1, 3, 4, and 5 UUIDs: uuid1(), uuid3(), uuid4(), uuid5().
Example:
>>> import uuid
>>> uuid.UUID('{ab4c0103-e305-0668-060a-0a3af12d3a0d}')
UUID('ab4c0103-e305-0668-060a-0a3af12d3a0d')
>>> uuid.uuid4()
UUID('76e1708f-1f12-4034-87cc-7c82f0591c61')
Final Remarks
Python's extensive range of built-in modules offers immense power to developers, enabling them to craft robust, versatile, and efficient applications. These modules, like the ones I've discussed, can greatly enhance the speed and functionality of your coding endeavors, whether you're a beginner or a seasoned professional.
Remember, while I've touched on just a few, a vast ocean of modules is awaiting your exploration. Every module has its unique capabilities, designed to simplify specific tasks. It's paramount for developers to familiarize themselves with these resources. Not only does this knowledge provide a competitive edge, but it also fosters a more intuitive and efficient approach to problem-solving.
I strongly encourage you to incorporate these modules into your projects, experiment with them, and experience the wonders they bring firsthand. And if you're eager for deeper dives, the Python documentation is an invaluable treasure trove of information.
Contact Us
Got more queries about Python modules? Or perhaps you're keen to expand your knowledge on other Python-related topics? Don't hesitate to reach out. At Sunscrapers, our passion is to disseminate knowledge and provide unparalleled technical expertise.
Whether you're in the market for Python or Django developers or simply want guidance on your tech journey, we're here for you.
Sunscrapers is committed to harnessing the power of Python, Django, and many other technologies, ensuring your projects meet and exceed expectations.
Ready to elevate your project? Get in touch with us today.