Your SaaS Will Explode Mid-Flight If You Don’t Do This

How I spent 2.5 days defusing a production bomb — and the checklist so you don’t have to.

11 min read

There’s a rite of passage every indie SaaS founder goes through. It sits somewhere between “I’ll just push to prod, what could go wrong?” and “Why is the database on fire at 2 AM?”

That rite of passage is called setting up a dev environment.

I know. Boring. Unsexy. The kind of task you keep postponing because there are features to ship, users to onboard, and a Product Hunt launch to prepare. You tell yourself: “I’ll do it when the project is more mature.” Then one day you fat-finger a migration, your production database eats itself, and suddenly you’re mature — emotionally, at least.

This is the story of how I set up a proper Dev/Prod split on a live SaaS, the 12 things that went wrong, and the battle-tested checklist so you can skip the trauma.

TL;DR You've been building your SaaS directly in production (like everyone does). One day you finally decide to set up a proper dev environment. Result: 2.5 days, 12 migration files, an infinite loop that fired 3,500 LLM calls, and a greatest hits collection of silent bugs (ghost 403s, RLS with no policies, secrets that don't travel, a CLI that ignores your flags). The article walks through every disaster plus a checklist so you don't have to learn this the hard way.

Your SaaS Will Explode Mid-Flight If You Don't Do This

The Starting Point: YOLO-Driven Development

Picture this: a SaaS product running in production for two months. One database. No dev environment. Schema changes are a fun cocktail of version-controlled migrations and random clicks in the production dashboard. Every deploy is a live surgery with no anesthesia.

If you’re reading this and thinking “that sounds like my setup” — congratulations, you’re the target audience. Keep reading. Your future self will thank you.

The goal was simple: create a separate development environment so we could break things without breaking the business. The execution was… educational.

Step 1: Choose Your Architecture (AKA Pick Your Poison)

For any cloud-hosted database (Supabase, PlanetScale, Neon, etc.), you basically have three options for Dev/Prod separation.

Option A — Two cloud projects. You create a second cloud project for development. No local Docker needed, works from any machine, and your dev environment mirrors prod as closely as possible. The tradeoff: you need to keep both in sync manually, and depending on your provider, it might cost a bit more.

Option B — Local dev with Docker. You run your database stack locally. Total isolation, completely free, and blazing fast. The tradeoff: 2–4 GB of RAM eaten by containers, it’s not portable across machines, and you’ll inevitably hit “works on my machine” moments.

Option C — Built-in branching. Some providers offer branch-like features where you can fork your database. It’s integrated and elegant. The tradeoff: usually requires a paid plan, and these features tend to be young and occasionally surprising.

We went with Option A — two cloud projects. No Docker tax, accessible from anywhere, and the free tier covered it. Your mileage may vary, but the principles below apply regardless of which option you pick.

Step 2: Confront the Schema Drift Monster

Here’s where things got spicy.

When I tried to recreate the development database from scratch using our existing migration files, it blew up immediately. The reason? Schema drift — the silent killer of reproducible deployments.

Over the weeks of building in production, several tables had been created directly through the dashboard UI. Columns had been added with a quick click instead of a migration file. RPC functions existed only in production with no code equivalent anywhere in the repo.

The migration files told one story. The actual database told a very different one.

This is the SaaS equivalent of those horror movies where the call is coming from inside the house. Your “source of truth” is lying to you.

The fix: We had to write backfill migrations — migration files that retroactively capture everything that was done outside of version control. Think of it as a database confession booth. “Forgive me, Git, for I have sinned. I created four tables via the dashboard.”

This took six migration files just to get the dev database to match what production actually looked like. Six files of CREATE TABLE IF NOT EXISTS, ALTER TABLE ADD COLUMN IF NOT EXISTS, and CREATE OR REPLACE FUNCTION statements.

The lesson is painful but simple: never, ever modify your database through a UI in production. Every change goes through a migration file. Every. Single. One. Even that “tiny column I’ll just add real quick.” Especially that one.

Step 3: The Silent 403s (Or: GRANT Me Patience)

After applying all the backfill migrations to the dev database, the app loaded. Pages rendered. Things looked fine.

Except nothing actually worked.

Queries returned empty arrays instead of data. Deletes silently did nothing. Updates vanished into the void. No error messages. No stack traces. Just… emptiness.

The culprit? Missing GRANT statements. The database role used by the application didn’t have SELECT permission on several tables. Without it, queries don’t fail — they just return zero rows. It’s PostgreSQL’s way of saying “I’m not mad, I’m just disappointed.”

This is particularly insidious because DELETE and UPDATE operations also need an internal SELECT to evaluate their WHERE clauses. No SELECT permission means your DELETE FROM orders WHERE user_id = 'abc' affects zero rows, even when there are 50 matching records. It just shrugs and moves on.

The fix: Add explicit GRANT statements in your migration files, right alongside the CREATE TABLE. Don't assume permissions will be inherited or magically configured. Be explicit. Be verbose. Be paranoid.

Step 4: The RLS Trap (AKA the Invisible Wall)

This one deserves its own section because it’s the most devious bug I’ve ever encountered in a database.

Row Level Security (RLS) is a fantastic PostgreSQL feature that lets you control which rows each user can see. When enabled on a table, you define “policies” — rules like “users can only see their own data.” It’s security at the database level and it’s genuinely great.

Here’s the catch: if you enable RLS on a table but define zero policies, every single operation is blocked. Not just writes. Not just deletes. Everything. SELECT included. The table becomes a black hole.

And because this was a dev environment that got its schema from migrations but its policies had been hand-configured through the production dashboard… you can see where this is going. Three tables had RLS enabled via migrations but their policies only existed in production, created through the UI.

The dev database had walls with no doors.

The fix: Always — always — create your RLS policies in the same migration file where you enable RLS. Treat ALTER TABLE ... ENABLE ROW LEVEL SECURITY and the corresponding CREATE POLICY statements as an atomic unit. One without the other is a foot-gun.

We ended up writing 12 policies across three tables just to restore basic CRUD functionality. Four policies per table: SELECT, INSERT, UPDATE, DELETE. Each one joining back to the user’s auth ID.

Step 5: Views, Invokers, and Permission Spaghetti

Just when I thought the permission saga was over, the views entered the chat.

We had database views that were set to security_invoker = on, meaning they execute with the permissions of the calling user, not the view owner. Normally fine. Except a previous migration had also revoked SELECT on the underlying tables from the application role.

So the view says “I’ll use your permissions.” The user says “I don’t have any.” The database says “permission denied.” Everyone is technically correct — the best kind of correct, and the worst kind of bug.

The fix: Set security_invoker = off on views where the underlying table permissions are restricted. This makes the view execute as its owner (typically the superuser), while RLS policies on the underlying tables still enforce row-level access control. You get the security without the permission paradox.

Step 6: Edge Functions and the Secret Problem

If your stack includes serverless functions (Edge Functions, Lambda, Cloud Functions, etc.), here’s a fun fact: each environment has its own secrets store.

Every API key, webhook URL, and authentication token you configured for production doesn’t magically appear in your dev environment. The dev functions will boot up, look for their secrets, find nothing, and return 401 Unauthorized on every single call.

This sounds obvious in retrospect. In practice, it’s always the last thing you check after two hours of debugging your function code.

The fix: Create a deployment checklist (more on that below) that includes copying or configuring all environment secrets for each new project. Don’t rely on memory. You will forget. I forgot. We all forget.

Step 7: The CLI Lie (Or: Why You Need a Sync Script)

Here’s a small but rage-inducing detail about many database CLIs: the push command (which applies pending migrations to a remote database) often ignores project flags.

You might think db push --project-ref my-dev-project would push to your dev project. It doesn't. It pushes to whatever project is currently "linked" in your local config, blissfully ignoring the flag you just passed.

Imagine typing what you think is a dev push, then watching your migrations hit production. Not great.

The fix: We built a small shell script that acts as a wrapper around the CLI. It handles linking to the right project before pushing, does a dry-run first to show you what will change, and — critically — always re-links back to the dev project when it’s done. The default should always be dev. Production should require an explicit, conscious, “I know what I’m doing” action.

Something like:

  • sync dev → Links to dev, dry-run, push, done.
  • sync prod → Links to prod, dry-run, push, then re-links to dev.
  • sync status → Dry-runs on both environments, touches nothing.

Three commands. No surprises. No accidental production pushes at 11 PM.

Step 8: The Automation Apocalypse

If your SaaS includes workflow automation (n8n, Make, Temporal, whatever), the multi-environment problem gets worse.

Most automation tools use “credentials” or “connections” that point to a single database or API endpoint. There’s no built-in concept of dev vs. prod. Your workflows are hardcoded to production by default.

We had a monolithic workflow with 46+ nodes handling five completely different features. Switching it to support multiple environments meant replacing every native database node with generic HTTP nodes that could dynamically resolve their target based on an environment variable.

And during this migration, a test run triggered 3,500+ loop executions in about 30 seconds. The cause? A JSON field that changed structure when we switched from native nodes to HTTP nodes. The loop guard was checking $json[0].total_products, which returned undefined in the new format. Without a valid number to stop the loop... it didn't stop.

Three thousand five hundred API calls. Some of them hitting an LLM endpoint. I’ll let you imagine the billing implications.

The fix (multi-part):

  • Split monolithic workflows into independent, focused units. One workflow per feature. Easier to test, easier to maintain, easier to not accidentally trigger 3,500 LLM calls.
  • Add safeguards on every loop. Before iterating, verify that your loop counter is actually a positive number. If it’s undefined, null, NaN, or negative — stop. Hard stop.
  • Test in isolation. Disconnect downstream nodes before testing upstream changes. A modified node that feeds into an LLM chain is a loaded gun.
  • Always test with minimal data. One or two records. Never “let’s just run it on the full dataset to see what happens.” You know what happens.

The Checklist: Don’t Leave Home Without It

After 2.5 days and 12 migration files (half of which were fixing problems that shouldn’t have existed), here’s the distilled checklist for setting up a dev/prod split on any SaaS.

Database setup:

  • Create the dev project/instance
  • Ensure every table, column, function, and type exists in your migration files — no schema drift
  • Apply all migrations to the dev database
  • Verify GRANT permissions on every table for every role your app uses
  • Verify that every table with RLS enabled has corresponding policies (at minimum: SELECT and INSERT)

Serverless functions:

  • Configure all environment secrets on the dev project (copy from prod, adjust URLs)
  • Verify JWT verification settings match between environments
  • Test every function endpoint individually

Application config:

  • Create environment-specific config files (.env.development, .env.production)
  • Ensure CORS settings allow your local development URL
  • Update any hardcoded URLs (yes, you have some — everyone does)

Automation / workflows:

  • Replace any hardcoded credentials with environment-aware lookups
  • Split monolithic workflows into independent units
  • Add loop safeguards and null checks on every dynamic value
  • Test with minimal data before running against real datasets

Deployment workflow:

  • Create a sync script that handles project linking automatically
  • Default to dev — always. Production requires explicit action.
  • Run dry-runs before every push, on every environment
  • Re-link to dev after any production operation

Validation:

  • Test the full user flow end to end: signup → core feature → data persistence → retrieval
  • Verify that dev and prod behave identically (same schema, same policies, same functions)
  • Document everything as you go — architecture docs, security docs, runbooks

The Rules Carved in Stone

After living through this, a few rules became non-negotiable:

Never modify the database through a GUI. Not in production, not ever. Every change is a migration file. Every migration file is in version control. No exceptions. That “quick fix” in the dashboard is future-you’s nightmare.

RLS and policies are a package deal. Enabling RLS without policies is like installing a lock and throwing away all the keys. Same migration. Same commit. Same PR.

Secrets don’t teleport. Every new environment needs its own secrets configured. Add it to the checklist. Add it to the checklist again.

Test in dev, verify in dev, break in dev. That’s literally why it exists. If you’re still testing in production, you don’t have a dev environment — you have two production environments, and one of them is haunted.

Migrations should be idempotent. Use IF NOT EXISTS, CREATE OR REPLACE, DROP IF EXISTS. A migration that crashes on re-run is a migration that will betray you during a recovery.

Audit the full delta, not just the latest change. When reconciling two environments, don’t fix things piecemeal. List everything that’s different, then fix it all in one coordinated batch. Piecemeal fixes create piecemeal bugs.

The Uncomfortable Truth

Here’s the thing nobody tells you when you start building a SaaS: the unsexy infrastructure work is the work that keeps your product alive.

Features get you users. A dev/prod split keeps you from losing them overnight.

It took 2.5 days. Twelve migration files. One infinite loop scare. Countless silent 403s. And now the codebase has something it never had before: a safety net.

Your SaaS is a plane, and you’ve been building it while flying it. Setting up a dev environment is installing the ejection seat. You hope you never need it. But when you do — and you will — you’ll be very, very glad it’s there.

Now go set up your dev environment. Before the next DROP TABLE does it for you.

This article is based on a real 2.5-day setup sprint on a live SaaS product. No production databases were harmed in the writing of this article. Several were harmed before it.

Every SaaS has a moment where dev environments go from 'nice to have' to 'absolutely critical'. Catch the battle-tested lessons from real production incidents in the weekly newsletter.

→ Get the Production Survival Kit