* No persist(). Not being able to cache dataframes is a nightmare when it's a workflow that involves taking a massive source of data, doing some rough filtering on it that gets it down to a tiny subset, and then doing more complex stuff with that.
* No good way to get usage info programmatically that I've found. For things like monitoring for periodic queries that get out of hand.
* Can't set Spark config. There are often ways to get around this, like when I recently had to set S3A credentials and needed a way that wasn't OS environment variables (this doesn't work for worker nodes). Eventually, through much documentation browsing and finally an exasperated hail mary question to ChatGPT that solved it (told me the things to pass into options() ) I got it working. But all the documentation and online QA resources just say to use Spark config.
* This is more of a Unity Catalog problem, but kind of applies because Serverless and UC often go very hand in hand (particularly when dealing with things that used to be stored in a cluster like credentials), but it drives me insane that I can only mount external volumes with the same block storage as my workspace provider. So I can't mount an external volume to an AWS bucket on an Azure UC. That means if I want to write stuff that can run the same regardless of what my customer is running their Databricks workspace with, I need to use less sophisticated approaches.
It's still nowhere near the pain that Databrick's attempt at copying Snowflake's VARIANT data type has caused me, but there are many times when I find myself having to work around serverless limitations. Especially when these limitations aren't really mentioned much upfront when Databricks pushes serverless aggressively.
@OP, I'm an EM at "frugal", ldap is hcliff if you want to chat.
I'm assuming OP is L5, which does require "people influencing", although no formal leadership and or TL role is required. You can stay at this level indefinitely.
OP, is your manager ok with your current performance? Is your EM trying to grow you for promo or into the full level reqs? You should seek clarity here and if the former set boundaries/expectations as per the parent comment.
> But now I'm struggling to find motivation to do my work
Is this recent? I'm seeing this quite a bit with the current job insecurity and turmoil at the company. As other comments have advocated: therapy and consider extended leave or a team switch. FWIW I don't think lower stakes work is going to fix your motivation, something you care about may.
the user you're replying to has delightful previous comments such as "the US media is controlled by people eligible for Israeli citizenship" which is more dog-foghorn than dog-whistle
Not a question, but I just want to thank you for "inventing" review apps back in the day, the idea of having a full version of your app running for a branch was pretty game changing for the companies I worked at.
Awesome! IMHO it was more of a refinement of the concept of deploying each commit; doing it at a PR and push level worked better. Open-sourcing it, and getting Heroku to bake it in was the icing on the cake for us!
You should revisit this, in the last couple of years this has moved quite a bit, I'd say it's closer to 70-75 now.
Obviously you'll get more in the states (or even frankfurt!), but it's changing rapidly.
There's also some offers in berlin for meaningfully higher, I'm just taking a ballpark average for senior.
