Not long ago we wrote Python, Yaml, SQL, Terraform. Depending on your stack of course. And we had to know about cloud networking, APIs, data modelling, security. 

The architecture of a decent data lake is broad, we have to know a lot of things about a lot of things. And I didn’t even want to talk about machine learning, Power BI modelling and Datavault (yikes).

Our jobs were demanding, we couldn’t keep up. Changes in the sources broke a pipeline regularly.

Until suddenly everything changed and we only had to know…. Markdown…. Really ?! Well sort of.

I am definitely talking to Claude a lot and haven’t written much code myself. 

Has it made me more productive, getting more done with higher quality. Absolutely!

Did my job become easier and is thinking optional now, can I chill at the coffee machine while Claude spits out everything I need. Definitely NOT!

My brain is on high alert. The code is not the bottle neck anymore. There’s more of it though and it’s more complicated. I have to be very specific about what I want (and how a lot) 

My job has shifted to quality control, architecture, security which has gotten harder because everything goed faster, there’s more code. And I’m asked to more myself, the silos are slowly disappearing. 

And on top of that I have to learn this new skill (pun intended). How do I make sure it’s safe? How does it remember? How do you make skills, how big should a context window be, which model is good for what, do I need local models and are they good enough?

I’m going to show what I do and how, there will be lessons learned out of my day job. What worked, what didn’t. What got better so quickly you may have discarded it half a year ago, but should try again perhaps.

This talk translates easily to other stacks, but I spend a lot of time using Databricks as the main platform, therefore all demos will focus around using Databricks as the main data lake.