Databricks for All
Databricks pulls the rug out from Snowflake by throwing open the doors and inviting everyone in...for free.
Databricks is planting the seeds to win the data platform rivalry. Let me tell you why.
While AWS, GCP, and Azure offer broad cloud platform services and solutions, for integrated solutions there are currently only two major players in the space: Databricks and Snowflake. They started with very different ideas - Snowflake aimed at business intelligence and data warehousing, while Databricks was looking at distributed data processing and data science practices. This was obvious in their initial user experience: Snowflake was queries, Databricks was notebooks. Snowflake was aimed at SQL, Databricks was aimed at Scala or Python (or R). Snowflake was business intelligence, Databricks was data science1.
In time, their offerings converged, even as their user experiences remained unique. Snowflake released Snowpark to cover developer languages like Python and Scala. Databricks acquired Redash and eventually built out the “Databricks SQL” section of the interface. While their products each had a unique feel, the Venn diagram of their capabilities overlapped more and more. The rivalry between the two is well-known to data practitioners2.
So I’m surprised that Databricks major shot across Snowflake's bow hasn’t gotten more attention. Back in June at their annual Data + AI Summit, Databricks announced a ‘free’ version of their platform, coupled with full access to their entire self-paced training repository3.
Databricks Academy and Databricks Free Edition
Ho hum. Big deal. Didn’t Databricks already have a free offering?
They did, kinda. The Databricks Community Edition offered an outdated and limited Databricks experience, mainly centered on notebook usage. Aside from missing many of the smaller improvements and refinements across the platform, the Community Edition completely lacked the now familiar Databricks SQL tools and Unity Catalog implementations. It was a stale, stagnant offering that sorely lagged behind the true Databricks commercial offerings4.
What about trials? You can sign up for a trial, but it has a limited timeframe, and it required access to your cloud account. This means even as you’re taking Databricks for a ‘free’ spin, AWS or Azure or GCP are charging you for any non-serverless cluster compute you spin up. After about a week Databricks salespeople account reps will start pinging you, and eventually it the trial gets shut off. While I haven’t tried in awhile, it looks like the trial experience has been improved with an “Express” option; however, to trial with your existing datalake5 requires you to link a cloud account where the storage and cluster compute are going to siphon some pennies out of your wallet:
But with their announcement and new offerings, Databricks has throw the doors wide open in a triple threat to any other data platform providers:
Targeting universities: Tomorrow’s tech jobs are going to continue to leverage data, data platforms, and ‘big data’ (when was the last time we heard that term?). Information is still king, and all the supportive mechanisms for data flow, data prep, and data insight are absolutely key, whether it’s in service to the traditional business intelligence needs (like queries and dashboards), last decade’s hot trends of data science and machine learning (like predictive analytics and auto-classification), or the current need to desperately feed the gluttonous AI movement. All flavors of artificial intelligence eat and breathe data - AI implementations scarf data by the petabyte and are ever hungry for more - and humans that know how to ingest, prepare, and work with data are going to be crucial. If those humans happen to be familiar with the Databricks platform and its end-to-end offerings, all the better for Databricks.
Free access to self-paced training - all the self-paced training: This is big. Many companies will offer some tidbits in their general training, some intro videos or a some basic ‘getting started’ and general ‘how tos’. But Databricks has thrown the door open, similar to sites like AWS Skillbuiler, to all of their training videos6.
Fully featured, unrestricted ‘Free Edition’: So what, you might scoff. Lots of companies offer free training? But here’s the upshot: Databricks Free Edition is almost completely, 100% fully featured7. It’s not nerfed like Tableau Public. It’s not time and/or resource limited like Snowflake. It’s a complete compliment to the training, and it doesn’t go away - you can have it forever. With minimal signup, you get an unrestricted Databricks environment to do….whatever you want. Data science. Machine learning. Basic SQL. Try out your ideas, try out your training, build your portfolio. Pay $0.
So what is the key takeaway? There are really only two big players in the integrated cloud-computing-slash-data-processing-slash-data-infrastructure field, Databricks and Snowflake8. This is primarily a two horse race9. Databricks has now rolled out the red carpet to anyone and everyone looking to gain skills, whether a student, a novice, or a pro. This is the kind of investment that pays off in a big way down the road, the seeds planted here are going to influence future data practitioners, technology leaders, and business people.
Similar to 2010’s release of Tableau Public, Databricks is building both a user base and purposefully cultivating a generation. Databricks has made an easy, low-risk bet that a frictionless and fully featured overture to education and accessibility will be the strategic edge that shapes the platform leaders of tomorrow. I fully expect them to be right. With that in mind, get over there and sign up for Databricks Free Edition.
To this day, despite the ascendance of ‘data science’ and how it was hottest job for the last decade, I still kind of chuckle at it. Despite the no-nonsense wikipedia entry. Despite the resilience of the profession in the face if wide tech layoffs. It still seems a bit silly. I’ve worked in research hospitals and athletic companies and beverage companies, places where - not that often, but sometimes - I would actually see people wearing white lab coats. Carrying clipboards. Maybe looking at some bubbling beakers. So, really, when you’re typing pip install scikit-learn or pretending one-hot encoding isn’t just exploding out a crazy pivot table, are you really doing science?
I have a problem with a lot of the terminology we use in software and data. “Engineers” and “architects” and so on. If I’m an expert at tuning queries and indexes and partitioning data files and I call myself a “database doctor” I’d be laughed out of the room.
Look, I’m getting myself wound up already.
Let’s just admit up front that I’m kind of a stinker, so I thought it was amusing when Snowflake announced their 2023 Snowflake Summit would be the same week as Databricks’ 2023 Data + AI Summit. The conferences didn’t overlap in 2024 and 2025, but were scheduled in back-to-back weeks both years. Since customers would likely have been invested in either one or other, I imagine it was probably partners and vendors that complained the most.
In fact, it is so stale and stagnant that the website for Community Edition currently shows a popup nudging you to the Free Edition:
We’ll see how long Community Edition - whoops, I mean Legacy Community Edition - sticks around before it goes to that great system graveyard in the Sky(net). Soon ends your watch, Community Edition, may you 404 in peace.
Face it, you know as well as I do it’s a dataswamp. Maybe even a dataquagmire. That’s why you’re looking to trial Databricks in the first place, right?
Alas, you’re still paying for live-training and certification exams. The lunch might be free, but drinks and desserts are extra. This is par for the course across most platforms, e.g. AWS, Tableau, Alteryx, etc.
Okay, like 90%. For instance, you don’t get completely unfettered resource creation privileges - as Ghodsen says in his announcement, they’d go bankrupt - so you’re limited to their ‘serverless compute’. You also can’t access an ‘Account’ level like you can in the full commercial version, which means you’re limited to one metastore, one workspace, etc. As I continue to explore, I’m sure I’ll bump into other limitations, but what I’ve bumped-up against so far seems minor and Free Edition is more fully featured than, say, Govcloud Databricks. Feature-wise, Free Edition appears to have near complete feature-parity with Commercial Edition - if you’re trying to learn, to accumulate skills, to hone expertise - this is huge.
Darkhouse honorable mention.