News for hackers. Stuff that matters.
Re:Ask HN: How do you test SQL? by MrPowers on Wednesday February 01, 2023 @10:50AM (#34612182)
Sure, this blog post explains the dependency injection design pattern with Spark: https://mrpowers.medium.com/dependency-injection-with-spark-...

You can structure your code to read from paths when run in the production environment, but inject DataFrames you build in memory for your test suite. Spark is designed to read multiple files in parallel, so it's not optimized to read a single tiny file. That's why it's best to avoid I/O in Spark test suites whenever possible.

There are no comments on this comment yet.