What happened

We accidentally deployed code late at night that wasn't fully tested. The code broke our login cookies, which led to users attempting to log in or out to be caught in a bad state where Bubble interpreted all requests from them as being unauthenticated, leading to a variety of data loading and saving errors.

Root causes

1. We built the cookie fix on a main development branch rather than a specialized feature branch, which made it easy to deploy by mistake.
2. Our deployment workflow doesn't show the list of changes that are being deployed -- we have to proactively check this
3. Our automated monitoring isn't looking for people having issues logging in, so we didn't catch the issue until we started seeing user reports on the forum

What we're doing to prevent similar issues in the future

1. We are changing our development workflow so that new development happens on item or project specific branches, to make it much harder to accidentally deploy code
2. We are going to change our deployment system to print out a list of the changes that are being deployed, as further protection against inadvertent mistake
3. We are adding additional monitoring and alerting around our login system to try to proactively catch issues like this immediately.