Recently, I made a post on Reddit asking: Why don’t data engineers test like software engineers do? This was meant to kick off a discussion for our blog series on testing (Part 1, Part 2, Part 3). The thread quickly became one of the most popular on r/dataengineering, staying on the top for two days.

Most of the comments centered around three main reasons:

• Testing is costly and time-consuming.
• Many analytical engineers lack a formal computer science background.
• Testing is often not implemented because projects are volatile and engineers have little control over source systems.

And here is our take on these:

Testing requires time and is costly

Reddit: The decision to invest in testing often depends on the company and the role data plays within its structure. If data pipelines are not central to the company’s main product, many engineers do not see the value in spending additional resources to ensure these pipelines work as expected.

Sunscrapers: Tests are a tool. If you consider your project simple enough and do not plan to scale it, then perhaps you do not need them.

Reddit: It can be more advantageous for engineers to deliver incomplete solutions, as they are often the only ones who can fix the resulting technical debt and are paid more for doing so.

Sunscrapers: Tight deadlines and fixed requirements mean that testing is usually the first thing to be cut. This allows engineers to deliver a solution and close a ticket, and if a bug is found later, extra time and effort are allocated from a different budget. While this approach is accepted by many managers, it is not ideal, as the overall time wasted on fixing issues often exceeds the time it would have taken to test the solution upfront.

Reddit: Stakeholders are rarely willing to pay for testing.

Sunscrapers: Testing is a tool for engineers, not stakeholders. Stakeholders pay for a working product, and it should be the producer's responsibility to ensure that the product meets the requirements. If I personally were about to buy a product from a store and someone told me to pay extra for testing, I would also refuse. If you are certain about your product do not test it, but do not ask non-technical people how to do your job.

Many analytical engineers lack a formal computer science background.

Reddit: Especially in analytical and scientific engineering, many people are not formally trained as software engineers. They are often self-taught programmers who write scripts to solve their immediate problems but may be unaware of software engineering practices that could make their projects more maintainable.

Sunscrapers: This is a common and ongoing challenge. Computers are tools used by almost everyone, but not everyone who uses a computer is a programmer. Many successful projects begin with someone trying to solve a problem in their own field, and in analytics, domain knowledge is often more important than programming expertise when building initial pipelines. In companies just starting their data initiatives, pipelines are typically built by analysts. As long as these pipelines meet expectations, this approach is acceptable. However, as complexity grows, changes become more costly, and tracking down the source of problems can become a nightmare.

No control of source data

Reddit: Data engineers often have no control over the source data, which can lead to issues when the schema changes or when unexpected data is encountered. This makes it difficult to implement testing.

Sunscrapers: This one of the assumptions of data engineering systems. Depending on the type of the data engineering system, data engineers very rarely will have a say in there. Only where we are creating the analytical system for the operational data, we might have a conversation with the operational system maintainers.

In other cases when we are scraping the data from the web or calling external APIs, it is not possible. So what are the ways that we could do to help in such situations?

When the problem is related to the evolution of schema (case when data fields are added or removed, data type changes): First we might use schema-on-read strategy, where we store the raw data as they are ingested, for example in JSON format in the staging models, we extract only the fields that are relevant to us. In this case, we do not care if new fields are added. When columns that were using are removed or changed the the pipeline will break, but if we have tests they will tell us what is the exact reason why. We have a place to start investigation and decide how to fix it

If the problem is unexpected data the issues are similar. It’s impossible to anticipate every possible variation in source data, and equally impossible to write pipelines that handle every scenario. The logic in our pipelines is typically designed for the data identified during initial analysis. If the data changes, we cannot guarantee that the analytics code will handle it correctly. Even simple data tests can alert us to these situations, indicating, for example: “We were not expecting data like this—please check if we can handle it.” This once again saves time on root cause analysis by pinpointing exactly where the problem is and where to start investigating a solution.

Conclusion

Testing in data engineering isn’t always easy or prioritized, but it’s often the difference between reliable insights and constant firefighting.

How does your team approach testing?