How I Reduced ETL Costs by 60% Using Databricks
Published on: December 25, 2025 08:39 AM
How I Reduced ETL Costs by 60% Using Databricks
I’ve spent the better part of two decades designing, building, and fixing data pipelines. My career started in the era of monolithic ETL platforms, where I became an expert in tools like IBM DataStage. They were powerful, reliable workhorses. They were also incredibly expensive, rigid, and slow to change.
About five years ago, I was tasked with leading a major ETL modernization initiative. The mandate from my CIO and CFO was clear: "Reduce our costs and make us more agile." The journey that followed was challenging, full of missteps, but ultimately successful. We migrated our core ETL workloads from DataStage to Databricks and, over 18 months, achieved a sustained total cost reduction of approximately 60%.
This isn't a story about a magic tool. It’s a story about a fundamental shift in how we approached data engineering, cost management, and team skills. Here’s exactly how we did it.
1. The Cost Problem: A Platform Buckling Under Its Own Weight
Before the migration, our data landscape was a classic, centralized ETL hub built around DataStage. We had hundreds of critical jobs processing terabytes of data daily, feeding everything from regulatory reporting to our primary data warehouse.
The problem was that the platform was becoming unsustainable on three fronts:
- Financial: Our multi-year, multi-million dollar DataStage license was a fixed, seven-figure line item on the budget, regardless of whether we used 10% or 100% of its capacity. On top of that, we had dedicated, oversized server infrastructure that was depreciating in a data center.
- Operational: Scaling was a nightmare. A sudden spike in data volume meant a months-long procurement and provisioning project for new hardware. Outages were catastrophic, requiring a small, specialized team of high-cost administrators to work around the clock.
- Strategic: Leadership wanted to explore machine learning and real-time analytics. Our legacy platform simply couldn't support these use cases. The business saw us as a bottleneck; a simple change to a pipeline could take weeks to deploy through rigid development and release cycles.
Leadership expected that moving to the cloud would magically solve these issues. The reality, as we soon found out, was far more complex.
2. Baseline Cost Breakdown (Before Databricks)
To make a credible business case, we had to be brutally honest about our "all-in" costs. It wasn’t just the license.
| Cost Category | Annual Estimated Cost | Notes |
|---|---|---|
| DataStage Licensing | ~$1,200,000 | Fixed, non-negotiable multi-year contract for production & non-prod environments. |
| Infrastructure & Operations | ~$750,000 | Amortized cost of dedicated on-prem servers, power, cooling, and data center space. |
| Support & Maintenance | ~$450,000 | Salaries for a specialized team of 3-4 DataStage administrators and platform support engineers. |
| Hidden "Opportunity" Costs | Significant, but hard to quantify | Cost of delayed projects, business frustration, developer rework on failed jobs, and P1 incident fire-drills. |
| Total Annual Baseline | ~$2,400,000 | This was the number we took to the CFO. |
This ~$2.4M per year was our benchmark. It was a fixed, predictable cost, but it was also a massive, inflexible anchor.
3. Why We Chose Databricks
We evaluated several alternatives, including building our own Spark-on-Kubernetes platform, using cloud-native services like AWS Glue or Azure Data Factory, and looking at other data warehousing platforms.
We chose Databricks for a few key reasons, and "cost" was only part of the equation:
- Pay-for-Use Compute: The theoretical ability to pay only for the compute seconds we used was the core of the financial argument. We could, in theory, turn everything off when it wasn't running.
- Flexibility & Unification: It wasn't just another ETL tool. It offered SQL for our analysts, Python and Scala for our engineers, and a clear path to ML and AI with the same platform. This de-risked future technology choices.
- Separation of Storage and Compute: This was a fundamental architectural advantage. We could scale our compute resources up or down independently of our data, which lives in our own cloud storage account (S3/ADLS).
We also acknowledged the risks upfront: the potential for a runaway cloud bill if we weren't disciplined, the significant re-skilling required for our team, and the cultural shock of moving from a GUI-based tool to a code-first environment.
4. The Reality Check: Our First Cloud Bill Was a Shock
We migrated our first dozen pipelines, "lifting and shifting" the logic as directly as possible. We set up a few large, all-purpose clusters to mimic our old on-prem servers and let the developers run their jobs.
Our first full month's bill was 20% higher than our projected monthly cost for Databricks.
The finance team was immediately on the phone. Here’s what went wrong:
- "Always-On" Cluster Mindset: We provisioned large interactive clusters and left them running 24/7, just like our old servers. Developers would attach, run a job, and walk away. Those idle VMs were burning money every second.
- Lift-and-Shift Logic: We replicated DataStage's stage-by-stage processing in Spark. This created incredibly inefficient execution plans with massive data shuffles, driving up compute time and cost.
- Ignoring Data Layout: We dumped our data into the data lake as-is. Our queries were performing full table scans on petabytes of data because we hadn't implemented partitioning or any form of data compaction.
The lesson was immediate and painful: The cloud rewards efficiency and ruthlessly punishes waste. You can’t bring an on-prem mindset to a pay-as-you-go world.
5. The Changes That Actually Reduced Our Costs
After that initial shock, we formed a small "FinOps-for-ETL" tiger team. We stopped all new migrations and focused entirely on optimization. These five changes were responsible for nearly all of our eventual savings.
- Radical Shift to Job Clusters: This was the single biggest money-saver. We banned the use of all-purpose interactive clusters for production-scheduled workloads. Every single automated pipeline was reconfigured to run on a Job Cluster—an ephemeral cluster that spins up for the job, executes the code, and terminates immediately. Cluster uptime dropped from 24/7 to just the few hours our jobs were actually running.
- Refactoring for Spark, Not DataStage: We stopped thinking in "stages" and started thinking in "DataFrames" and "transformations." We taught developers to use predicate pushdown (filtering data at the source), avoid
select *, and structure joins to minimize data shuffling. A job that took 60 minutes of inefficient shuffling on a large cluster was refactored to run in 10 minutes on a smaller one. - Intelligent Workflow Orchestration: We used Databricks Workflows to break monolithic pipelines into smaller, dependent tasks. This allowed us to use different, right-sized clusters for each step. A light ingestion step used a tiny cluster, while a heavy transformation step used a larger, memory-optimized one. This workload isolation prevented one inefficient query from slowing down and increasing the cost of an entire run.
- Aggressive Delta Lake Optimization: All our tables were converted to Delta Lake. We implemented a mandatory, automated process that ran
OPTIMIZEandZ-ORDERon our most frequently queried tables. This compacted small files and physically reordered the data, drastically reducing the amount of data scanned per query. Less data scanned means faster queries and lower compute costs. We also set up strictVACUUMpolicies to purge old data versions and control storage bloat. - Automating Everything: We used Terraform to define all our jobs, clusters, and permissions as code. This eliminated manual configuration errors and ensured every new job adhered to our cost-saving patterns (e.g., mandatory auto-termination, standardized instance types).
6. What Did NOT Move the Needle
Just as important is what didn't work. We wasted time and money on these:
- Throwing Bigger Clusters at Problems: Our first instinct for a slow job was to double the cluster size. This just masked the underlying inefficiency of the code and doubled the cost.
- Blindly Trusting Auto-scaling: Auto-scaling is powerful, but it's not a silver bullet. If your code has a bottleneck that can’t be parallelized (e.g., all data being sent to a single partition), auto-scaling will just add idle workers you still have to pay for.
- "One-Click Optimization" Tools: Several tools promise to analyze your Spark code and magically fix it. They provided some minor suggestions but couldn't fix fundamental architectural flaws in our pipelines. There is no substitute for a developer who understands Spark fundamentals.
- Copying DataStage Assumptions: We assumed, for example, that staging data in intermediate tables was always necessary. In Spark, it's often more efficient to chain transformations in memory. Unlearning old habits was critical.
7. Measuring the 60% Reduction
After 18 months, once all major workloads were migrated and optimized, we did a final cost comparison with our FinOps team.
| Cost Category | Before (Annual) | After (Annual, Year 2) | Reduction |
|---|---|---|---|
| Licensing + Compute | $1,200,000 (License) | ~$650,000 (Databricks + Cloud) | ~46% |
| Infrastructure & Ops | $750,000 (On-Prem) | ~$100,000 (Cloud Storage/Network) | ~87% |
| Support & Maintenance | $450,000 (Specialized Admins) | ~$210,000 (Re-skilled Cloud Engineers) | ~53% |
| Total Annual Cost | ~$2,400,000 | ~$960,000 | ~60% |
How we validated this:
* The "Before" cost was our established budget line item, signed off by the CFO.
* The "After" cost was pulled directly from our cloud bill, using resource tags that isolated the Databricks platform, associated storage, and networking.
* The support costs were based on the salaries of the new, smaller, and more versatile platform team compared to the old, highly specialized one.
The savings weren't immediate. For the first 3-6 months, costs were volatile. The 60% reduction was the stable, predictable state we reached after 18 months of focused effort.
8. Trade-Offs and Hard Decisions
This wasn't a frictionless process. We had to make tough calls:
- Performance vs. Cost: We had to accept that some jobs might run slightly slower on a smaller, cheaper cluster. The discussion changed from "make it run as fast as possible" to "make it run within the SLA at the lowest possible cost."
- Investment in People: We had to spend significant time and money retraining our best DataStage developers in Spark, Python, and cloud architecture. Some embraced it and became our biggest champions. Others couldn't make the leap and eventually left the team.
- Letting Go of Control: We moved from a centralized governance model to a "Center of Excellence" model. We provided teams with best practices, templates, and guardrails, but gave them autonomy to manage their own workloads. This required trust and a new level of ownership at the team level.
9. Lessons Learned for Next Time
If I were to do this again, I would change three things:
- Start with a Cost-Aware Culture Day One. I would embed a FinOps expert in the data platform team from the very beginning. Every design decision would be viewed through a cost lens.
- Build a "Golden Path" First. Before migrating a single legacy job, I would build a perfect, fully automated, cost-optimized "Hello World" pipeline. This would become the mandatory template for all future development.
- Don't Promise Instant Savings. I would be more explicit with leadership that a "J-curve" is inevitable: costs will likely go up before they come down significantly. Managing expectations is half the battle.
A leading indicator that you're on the right path is when your DBU-per-terabyte-processed starts trending down. A lagging indicator is your monthly cloud bill. Watch the leading one.
10. Executive Summary: For the CXO, CIO, and Finance Leader
If you are sponsoring a similar initiative, please take these three points away:
- Cost reduction is a dedicated program, not a feature of a new tool. Simply buying Databricks (or any other tool) will not save you money. It will likely cost you more unless you actively change how your teams work.
- You must fund the "unsexy" work. The real savings come from refactoring old code, retraining your people, and building automation. This requires upfront investment and dedicated time that competes with new feature development. You must protect this work.
- Enforce new standards. The shift to job clusters, code reviews for efficiency, and mandatory cost tagging won't happen by consensus alone. Leadership must communicate the new standards and hold teams accountable for adhering to them.
Modernizing our ETL platform was one of the most challenging projects of my career. But by moving beyond the tool and focusing on our engineering discipline, financial accountability, and team culture, we turned a costly, rigid system into a strategic asset that is now faster, more capable, and 60% cheaper to run. The savings were real, but they had to be earned.