Memory efficient query execution in batches with PostgreSQL and Python

Sunscrapers

Maciej Wysocki

1 August 2024, 8 min read

thumbnail post

In this article, we will explore strategies for effective batched query execution in Python when working with PostgreSQL databases. We will delve into an approach that can be used for handling large tables, highlighting techniques to ensure your queries remain efficient and your database interactions smooth, regardless of the dataset size.

What's inside

  1. Introduction
  2. Challenges in Data Extraction
  3. Reproducible Setup for Experiments
  4. Simple Query Execution
  5. Batch Results with Pandas and Polars
  6. Setup of an Effective Server-Side Execution
  7. Comparison of Memory Usage and Runtimes
  8. Conclusions

Let's talk

Discover how software, data, and AI can accelerate your growth. Let's discuss your goals and find the best solutions to help you achieve them.

Hi there, we use cookies to provide you with an amazing experience on our site. If you continue without changing the settings, we’ll assume that you’re happy to receive all cookies on Sunscrapers website. You can change your cookie settings at any time.