Sławomir Sawicki
29 January 2019, 15 min read
What's inside
One feature of Python that has made it so popular is the sheer number and variety of tools that make a developer’s life easier. Modules and libraries help to speed up the development process, saving your team plenty of time to focus on the business logic or other key aspects of your project.
If you’re writing something long in Python, be sure to use a text editor to prepare your input for the interpreter - quitting the interpreter means that all your definitions will be lost.
Smart developers create their script in a text editor. As your code becomes longer, you’ll need to split it for easier maintenance. That way get to use a specific function in other programs without having to copy its definition into each of them.
Python allows developers placing definitions in a file and using them in a script or an interactive instance of the interpreter. That kind of file is called a module; it’s a file that contains Python definitions and statements.
Here are 30 built-in Python modules you better be using in your Python project.
abc
Doc: https://docs.python.org/3.7/library/abc.html
This tiny module delivers the environment you need for creating abstract base classes.
Example:
>>> from abc import ABCMeta, abstractmethod
>>> class MyAbstractClass(metaclass=ABCMeta):
... @abstractmethod
... def x(self):
... pass
... @abstractmethod
... def y(self):
... pass
...
>>> obj = MyAbstractClass()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyAbstractClass with abstract methods x, y
>>> class MyCustomClass(MyAbstractClass):
... def x(self):
... return 'x'
...
>>> obj = MyCustomClass()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyCustomClass with abstract methods y
>>> class MyCustomClass(MyAbstractClass):
... def x(self):
... return 'x'
... def y(self):
... return 'y'
...
>>> obj = MyCustomClass()
>>> obj.x(), obj.y()
('x', 'y')
argparse
Doc: https://docs.python.org/3.7/howto/argparse.html
This module contains tools that make it easier to create user-friendly interfaces from the level of the command line.
Example:
>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.exit = lambda x='', y='': print(x, y) # We don't want to exit python shell
>>> _ = parser.add_argument('foo', help='help text for \`foo\`')
>>> _ = parser.add_argument('-b', '--bar', help='help text for \`--bar\`')
>>> parser.parse_args(['-h'])
usage: [-h] [-b BAR] foo foo
positional arguments:
foo help text for \`foo\`
foo help text for \`foo\`
optional arguments:
\-h, --help show this help message and exit
\-b BAR, --bar BAR help text for \`--bar\`
usage: [-h] [-b BAR] foo foo
2 : error: the following arguments are required: foo, foo
Namespace(bar=None, foo=None)
>>> parser.parse_args()
usage: [-h] [-b BAR] foo foo
2 : error: the following arguments are required: foo, foo
Namespace(bar=None, foo=None)
>>> parser.parse_args('example.py foo_data'.split(' '))
Namespace(bar=None, foo='foo_data')
>>> parser.parse_args('example.py foo_data --bar bar_data'.split(' '))
Namespace(bar='bar_data', foo='foo_data')
>>> result = parser.parse_args('example.py foo_data --bar bar_data'.split(' '))
>>> result.foo
'foo_data'
>>> result.bar
'bar_data'
asyncio
Doc: https://docs.python.org/3.7/library/asyncio.html
This is a very large module that delivers the framework and environment for asynchronous programming. It was added to Python 3.4 as a temporary module and works as an alternative to the popular framework called Twisted. In short, asyncio comes in handy if you want to create asynchronous, concurrent, and single-threaded code. The module launches a loop where the asynchronous code is executed in the form of tass. When one task is inactive (for example, waiting for server response), the module executes another task.
Example:
>>> import asyncio
>>> async def printer(msg, sleep):
... await asyncio.sleep(sleep)
... print (msg)
...
>>> task0 = asyncio.ensure_future(printer('task0 -> sleep 5', 5))
>>> task1 = asyncio.ensure_future(printer('task1 -> sleep 1', 1))
>>> task2 = asyncio.ensure_future(printer('task2 -> sleep 3', 3))
>>> future = asyncio.gather(task0, task1, task2)
>>> future.add_done_callback(lambda x: asyncio.get_event_loop().stop())
>>> loop = asyncio.get_event_loop()
>>> loop.run_forever()
task1 -> sleep 1
task2 -> sleep 3
task0 -> sleep 5
>>> future.done()
True
base64
Doc: https://docs.python.org/3.7/library/base64.html
This well-known module delivers a tool for coding and decoding binary code into a format that can be displayed - and the other way round.
Example:
>>> import base64
>>> data = base64.b64encode(b'ASCII charakters to be encoded')
>>> data
b'QVNDSUkgY2hhcmFrdGVycyB0byBiZSBlbmNvZGVk'
>>> data = base64.b64decode(data)
>>> data
b'ASCII charakters to be encoded'
collections
Doc: https://docs.python.org/3.7/library/collections.html
This module offers specialized container data types that work as an alternative to basic contained types for general purposes such as dict, list, set, and tuple.
Example:
>>> from collections import defaultdict
>>> data, ddata = dict(), defaultdict(int)
>>> data, ddata
({}, defaultdict(<class 'int'>, {}))
>>> ddata['key'] += 5
>>> data['key'] += 5
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'key'
>>> data, ddata
({}, defaultdict(<class 'int'>, {'key': 5}))
copy
Doc: https://docs.python.org/3.7/library/copy.html
Everyone uses this tiny module that contains tools for deep copying of container type data. Its most famous function is `deepcopy` - if not for this function, copying lists and dictionaries would become a torture for developers.
Example:
>>> from copy import deepcopy
>>> data = {'key0': {'key1': False}}
>>> cdata = data.copy()
>>> data, cdata
({'key0': {'key1': False}}, {'key0': {'key1': False}})
>>> cdata['key0']['key1'] = True
>>> data, cdata
({'key0': {'key1': True}}, {'key0': {'key1': True}})
>>> data = {'key0': {'key1': False}}
>>> dcdata = deepcopy(data)
>>> data, dcdata
({'key0': {'key1': False}}, {'key0': {'key1': False}})
>>> dcdata['key0']['key1'] = True
>>> data, dcdata
({'key0': {'key1': False}}, {'key0': {'key1': True}})
csv
Doc: https://docs.python.org/3.7/library/csv.html
Delivers functionalities for exporting and importing tabular data in CSV format. The module allows developers to say “load data” or “save data” from to a CSV file.
Example:
>>> import csv
>>> with open('example.csv', 'w', newline='') as csvfile:
... writer = csv.writer(csvfile)
... writer.writerow(['Column A', 'Column B', 'Column C'])
... writer.writerow(['Cell 0 A', 'Cell 0 B', 'Cell 0 C'])
... writer.writerow(['Cell 1 A', 'Cell 1 B', 'Cell 1 C'])
...
28
28
28
>>> with open('example.csv', newline='') as csvfile:
... reader = csv.reader(csvfile)
... print('\\n'.join([' | '.join(row) for row in reader]))
...
Column A | Column B | Column C
Cell 0 A | Cell 0 B | Cell 0 C
Cell 1 A | Cell 1 B | Cell 1 C
datetime
Doc: https://docs.python.org/3.7/library/datetime.html
A simple and one of the most popular modules in Python. It delivers tools that make it easier to work with dates and times. The most popular classes are `datetime`, `timezone` and `timedelta`, but `date`,`time`, and `tzinfo` can be useful as well.
Example:
>>> from datetime import datetime, timedelta, timezone
>>> obj = datetime.now()
>>> obj
datetime.datetime(2018, 10, 7, 12, 27, 25, 527961)
>>> obj += timedelta(days=7)
>>> obj
datetime.datetime(2018, 10, 14, 12, 27, 25, 527961)
>>> obj.astimezone(timezone.utc)
datetime.datetime(2018, 10, 14, 10, 27, 25, 527961, tzinfo=datetime.timezone.utc)
decimal
Doc: https://docs.python.org/3.7/library/decimal.html
The module delivers a data type called Decimal. Its main advantage is correct rounding of decimal numbers which is extremely important in billing systems - even a slight distortion with several hundred thousand operations can change the final result significantly.
Example:
>>> import decimal
>>> float(0.1 + 0.2)
0.30000000000000004
>>> float(decimal.Decimal('0.1') + decimal.Decimal('0.2'))
0.3
functools
Doc: https://docs.python.org/3.7/library/functools.html
The functools module comes in handy for higher-order functions, ie. the functions that act on or return other functions. You can treat any callable object as a function for the purposes of this module.
Example:
>>> from functools import wraps
>>> def example_decorator(func):
... @wraps(func)
... def wrapper(\*args, \*\*kwargs):
... print("Print from decorator")
... return func(\*args, \*\*kwargs)
... return wrapper
...
>>> @example_decorator
... def example_func():
... """Func docstring"""
... print("Print from func")
...
>>> example_func()
Print from decorator
Print from func
>>> example_func.__name__
'example_func'
>>> example_func.__doc__
'Func docstring'
>>> from functools import lru_cache
>>> @lru_cache()
... def example_func(n):
... return n \* n
...
>>> [example_func(n) for n in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> example_func.cache_info()
CacheInfo(hits=0, misses=10, maxsize=128, currsize=10)
>>> [example_func(n) for n in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> example_func.cache_info()
CacheInfo(hits=10, misses=10, maxsize=128, currsize=10)
hashlib
Doc: https://docs.python.org/3.7/library/hashlib.html
This handy module implements a common interface to numerous secure hash and message digest algorithms like the FIPS secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2), as well as RSA’s MD5 algorithm (defined in Internet RFC 1321). Note the terminology: “secure hash” and “message digest” are interchangeable.
Example:
>>> import hashlib
>>> obj.update(b"Example text for hash")
>>> obj.digest()
b'\\xc0\\xf4\\xad\\x1a\\xf5IK\\x9a\\x17x\\xf6"\\xfagG\\x92\\xe1\\x0c\\x8f\\xef\\xe2\\x99\\xfa\\x97\\x12Q#\\xe7Za\\xb3\\xe2'
>>> obj.hexdigest()
'c0f4ad1af5494b9a1778f622fa674792e10c8fefe299fa97125123e75a61b3e2'
http
Doc: https://docs.python.org/3.7/library/http.html
This package collects several modules for working with the HyperText Transfer Protocol such as http.client (low-level HTTP protocol client), http.server (includes basic HTTP server classes based on socketserver), http.cookies (with utilities for implementing state management with cookies) and http.cookiejar (that provides persistence of cookies).
Example:
>>> import http.client, http.server, http.cookies, http.cookiejar
>>> conn = http.client.HTTPSConnection("sunscrapers.com")
>>> conn.request("GET", "/")
>>> resp = conn.getresponse()
>>> resp.status
200
>>> resp.readlines()[2:5]
[b'<!DOCTYPE HTML>\\n', b'<html>\\n', b'<head>\\n']
>>> conn.close()
>>> http.HTTPStatus
<enum 'HTTPStatus'>
>>> http.client
<module 'http.client' from '/usr/lib/python3.7/http/client.py'>
>>> http.server
<module 'http.server' from '/usr/lib/python3.7/http/server.py'>
>>> http.cookies
<module 'http.cookies' from '/usr/lib/python3.7/http/cookies.py'>
>>> http.cookiejar
<module 'http.cookiejar' from '/usr/lib/python3.7/http/cookiejar.py'>
importlib
Doc: https://docs.python.org/3.7/library/importlib.html
The package provides the implementation of the import statement (and the import() function) in Python source code. The components to implement import are exposed, making it easier for developers to create their own custom objects to participate in the import process.
Example:
>>> import importlib
>>> importlib.sys
<module 'sys' (built-in)>
>>> importlib.import_module("this")
The Zen of Python, by Tim Peters
...
<module 'this' from '/usr/lib/python3.7/this.py'>
itertools
Doc: https://docs.python.org/3.7/library/itertools.html
This useful module implements iterator building blocks inspired by constructs from APL, Haskell, and SML (all in a form that matches Python programs). It standardizes a core collection of quick, memory-efficient tools you can use on their own or in combination to construct specialized tools efficiently in pure Python.
Example:
>>> import itertools
>>> "-".join(list(itertools.chain("ABC", "DEF")))
'A-B-C-D-E-F'
>>> list(itertools.combinations("ABC", 2))
[('A', 'B'), ('A', 'C'), ('B', 'C')]
inspect
Doc: https://docs.python.org/3.7/library/inspect.html
The module offers several useful functions that help developers get information about live objects such as modules, classes, methods, functions, tracebacks, frame objects, and code objects in Python code. It provides 4 main functions: type checking, getting source code, inspecting classes and functions, and examining the interpreter stack. You can use it to retrieve the source code of a method, examine the contents of a class, or just get all the relevant information to display a detailed traceback.
Example:
>>> import inspect
>>> inspect.getmodulename(inspect.__file__)
'Inspect'
>>> inspect.isfunction(lambda _: _)
True
>>> inspect.signature(lambda _: _)
<Signature (_)>
json
Doc: https://docs.python.org/3.7/library/json.html
JSON (JavaScript Object Notation) is a lightweight data interchange format that was inspired by JavaScript object literal syntax. The module json exposes an API that looks similar to the standard library marshal and pickle modules.
Example:
>>> import json
>>> data = json.dumps({"A": 1, "B": True, "C": [1, 2, 3]})
>>> data
'{"A": 1, "B": true, "C": [1, 2, 3]}'
>>> json.loads(data)
{'A': 1, 'B': True, 'C': [1, 2, 3]}
logging
Doc: https://docs.python.org/3.7/library/logging.html
This module defines functions and classes that provide a flexible event logging system for applications and libraries. It’s a good idea to use it because it ensures that all Python modules can participate in logging - your application log may include your messages integrated together with messages coming from third-party modules.
Example:
>>> import logging
>>> FORMAT = '[%(asctime)-15s] [%(name)s] [%(levelname)s] - %(message)s'
>>> logging.basicConfig(format=FORMAT)
>>> logger = logging.getLogger("MY_LOGGER")
>>> logger.error("My error message")
[2018-10-29 09:22:10,880] [MY_LOGGER] [ERROR] - My error message
math
Doc: https://docs.python.org/3.7/library/math.html
This module gives developers access to the mathematical functions defined by the C standard. You can’t use them with complex numbers, which is a good thing if you’re not looking to learning a lot of mathematics required to understanding complex problems.
Example:
>>> import math
>>> math.pow(2, 4)
16.0
>>> math.sqrt(16)
4.0
>>> math.floor(5/2)
2
>>> math.ceil(5/2)
3
multiprocessing
Doc: https://docs.python.org/3.7/library/multiprocessing.html
This handy package supports spawning processes with the help of an API similar to the threading module. It provides both local and remote concurrency using subprocesses instead of threads to side-step the Global Interpreter Lock. Developers use it to take full advantage of multiple processors on a given machine. Runs on both Unix and Windows.
Example:
>>> from multiprocessing import Process
>>> def my_function():
... count = 0
... while True:
... print ("My function: {0}".format(count))
... count += 1
... time.sleep(1)
...
>>> proc = Process(target=my_function, args=())
>>> proc.start()
>>> My function: 0
My function: 1
My function: 2
...
os
Doc: https://docs.python.org/3.7/library/os.html
This module offers a portable method of using operating system dependent functionality.
Example:
>>> import os
>>> os.getpid()
7171
>>> os.name
'posix'
>>> os.environ['_']
'/usr/bin/python3.7'
>>> os.path.join('dir0', 'dir1', 'file.py')
'dir0/dir1/file.py'
pdb
Doc: https://docs.python.org/3.7/library/pdb.html
This module defines an interactive source code debugger. It supports single stepping at the source line level, setting (conditional) breakpoints, inspection of stack frames, source code listing, and more.
Example:
slawek@swiegy:/tmp$ echo "print('Hello World')" > test.py
slawek@swiegy:/tmp$ python test.py
Hello World
slawek@swiegy:/tmp$ echo "import pdb;pdb.set_trace()" > test.py
slawek@swiegy:/tmp$ python test.py
\--Return--
\> /tmp/test.py(1)<module>()->None
\-> import pdb;pdb.set_trace()
(Pdb)
random
Doc: https://docs.python.org/3.7/library/random.html
This useful module implements pseudo-random number generators for various distributions. For example, you get uniform selection from a range for integers and for sequences, there is uniform selection of a random element (as well as a function to generate a random permutation of a list in-place and for random sampling without replacement).
Example:
>>> import random
>>> random.randint(0,100)
20
>>> random.randint(0,100)
42
>>> random.random()
0.6609538830840114
>>> random.random()
0.486448371691216
re
Doc: https://docs.python.org/3.7/library/re.html
This module provides regular expression matching operations similar to those you get in Perl. You can search Unicode strings and 8-bit strings, but they can’t be mixed.
Example:
>>> import re
>>> res = re.search('(?<=abc)def', 'abcdef')
>>> res.group(0)
'def'
>>> bool(res)
True
>>> res = re.search('(?<=abc)def', 'xyzdef')
>>> bool(res)
False
Read more: What every Pythonista should know about Unicode
shutil
Doc: https://docs.python.org/3.7/library/shutil.html
The module provides a number of high-level operations on files and collections of files, especially functions that support file copying and removal.
Example:
>>> import shutil
>>> import os
>>> os.listdir()
[]
>>> open('test.py', 'a+').close()
>>> os.listdir()
['test.py']
>>> shutil.copyfile('test.py', 'test2.py')
'test2.py'
>>> os.listdir()
['test2.py', 'test.py']
sys
Doc: https://docs.python.org/3.7/library/sys.html
The module offers access to variables used or maintained by the interpreter and functions that interact strongly with the interpreter.
Example:
>>> import sys
>>> sys.argv
['']
>>> sys.modules.keys()
dict_keys(['sys', 'builtins', '_frozen_importlib', '_imp', '_thread', '_warnings', '_weakref', 'zipimport', '_frozen_importlib_external', '_io', 'marshal', 'posix', 'encodings', 'codecs', '_codecs', 'encodings.aliases', 'encodings.utf_8', '_signal', '__main__', 'encodings.latin_1', 'io', 'abc', '_abc', 'site', 'os', 'stat', '_stat', 'posixpath', 'genericpath', 'os.path', '_collections_abc', '_sitebuiltins', '_bootlocale', '_locale', 'types', 'importlib', 'importlib._bootstrap', 'importlib._bootstrap_external', 'warnings', 'importlib.util', 'importlib.abc', 'importlib.machinery', 'contextlib', 'collections', 'operator', '_operator', 'keyword', 'heapq', '_heapq', 'itertools', 'reprlib', '_collections', 'functools', '_functools', 'zope', 'sitecustomize', 'apport_python_hook', 'readline', 'atexit', 'rlcompleter', 'shutil', 'fnmatch', 're', 'enum', 'sre_compile', '_sre', 'sre_parse', 'sre_constants', 'copyreg', 'errno', 'zlib', 'bz2', '_compression', 'threading', 'time', 'traceback', 'linecache', 'tokenize', 'token', '_weakrefset', '_bz2', 'lzma', '_lzma', 'pwd', 'grp'])
>>> sys.path
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages']
threading
Doc: https://docs.python.org/3.7/library/threading.html
This useful module builds higher-level threading interfaces on top of the lower level _thread module.
Example:
>>> import threading
>>> def worker(num):
... print ("Worker num: {0}".format(num))
...
>>> for i in range(4):
... threading.Thread(target=worker, args=(i,)).start()
...
Worker num: 0
Worker num: 1
Worker num: 2
Worker num: 3
types
Doc: https://docs.python.org/3.7/library/types.html
The module defines utility functions that support the dynamic creation of new types.
It defines names for object types used by the standard Python interpreter but not exposed as builtins. It also offers additional type-related utility classes and functions that aren’t builtins.
Example:
>>> import types
>>> def gen():
... yield True
...
>>> isinstance(gen(), types.GeneratorType)
True
>>> import logging
>>> logging
<module 'logging' from '/usr/lib/python3.7/logging/__init__.py'>
>>> types.ModuleType(“logging”)
<module 'logging'>
unittest
Doc: https://docs.python.org/3.7/library/unittest.html
Originally inspired by JUnit, the module works similarly to major unit testing frameworks in other programming languages. It supports a broad range of functions: test automation, sharing of setup, shutdown code for tests, test aggregation into collections, and more.
Example:
>>> import unittest
>>> class TestStringMethods(unittest.TestCase):
... def test_example(self):
... self.assertTrue(True, True)
...
>>> unittest.main()
.
\----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
urllib
Doc: https://docs.python.org/3.7/library/urllib.html
This handy package collects several modules for working with URLs: urllib.request (opening and reading URLs), urllib.error (includes exceptions raised by urllib.request), urllib.parse (for parsing URLs), and urllib.robotparser (for parsing robots.txt files).
Example:
>>> from urllib.parse import urlparse
>>> urlparse('https://sunscrapers.com/aboutus/')
ParseResult(scheme='https', netloc='sunscrapers.com', path='/aboutus/', params='', query='', fragment='')
uuid
Doc: https://docs.python.org/3.7/library/uuid.html
The module provides immutable UUID objects, as well as the following functions for generating version 1, 3, 4, and 5 UUIDs: uuid1(), uuid3(), uuid4(), uuid5().
Example:
>>> import uuid
>>> uuid.UUID('{ab4c0103-e305-0668-060a-0a3af12d3a0d}')
UUID('ab4c0103-e305-0668-060a-0a3af12d3a0d')
>>> uuid.uuid4()
UUID('76e1708f-1f12-4034-87cc-7c82f0591c61')
I hope this Python modules list helps you make the most of Python in your future projects.
Naturally, this list is just the tip of the iceberg. For more information about Python built-in modules, head over the the extensive Python documentation.
Do you have any questions about Python modules? Send us a message ; we’re happy to share our knowledge of this incredibly versatile programming language.
