Supabase pricing starts free. We break down real costs for Pro ($25/mo) and Team ($599/mo) plans, explain overages, and compare each service head-to-head with AWS.

Supabase vs AWS: Feature and Pricing Comparison (2026)


When choosing a PostgreSQL database platform, whether for a side project or a production app, you need to look beyond features and assess the **total cost of ownership** — including compute, storage, backups, and bandwidth.

In this guide, we compare **Supabase vs AWS (RDS and Aurora)** across **free, entry-level, and production tiers**. We focus strictly on database-related costs and explain **on-demand vs reserved pricing** where applicable.

## 0. Free Plans: What Do You Get for \$0?

Both platforms offer free tiers, but they differ significantly in **duration**, **compute power**, and **resource isolation**.

| Feature          | Supabase Free Tier           | AWS Free Tier (12 months)                              |
| ---------------- | ---------------------------- | ------------------------------------------------------ |
| Duration         | Forever                      | 12 months from signup                                  |
| Compute          | Shared CPU / 500MB RAM       | 2 vCPU (burstable) / 1GB RAM (t4g.micro, 750 hrs/mo)\* |
| Compute Type     | Shared container environment | Dedicated EC2 instance (Graviton2, burstable)          |
| CPU Architecture | x86                          | ARM (AWS Graviton2)                                    |
| Storage (DB)     | 500MB                        | 20GB gp3\*                                             |
| Backup           | 7-day snapshot               | 20GB snapshot                                          |
| Bandwidth        | 5GB outbound                 | 15GB outbound                                          |
| OS / Isolation   | Serverless (no OS access)    | Full OS-level isolation                                |

- **Supabase Free** is ideal for hobby projects, quick MVPs, or internal tools with light traffic. It requires no setup and stays free forever.
- **AWS Free Tier** gives you significantly more power and isolation but is limited to the first 12 months after signup.

## 1. Entry-Level (Low Cost)

When you outgrow the free tier but don’t need production-grade resources, these plans offer low-cost paths. Supabase simplifies everything with a flat rate. AWS offers more control and deeper savings through reserved pricing.

| Feature          | Supabase Pro Tier                      | RDS t4g.micro (On-Demand)             | RDS t4g.micro (1yr Reserved)        | RDS t4g.micro (3yr Reserved)        |
| ---------------- | -------------------------------------- | ------------------------------------- | ----------------------------------- | ----------------------------------- |
| Monthly Price    | \$25 (includes \$10 compute)\*         | \$11.68                               | \$6.69                              | \$4.76                              |
| Compute          | 2-core ARM (shared) / 1GB RAM          | 2 vCPU (burstable) / 1GB RAM          | 2 vCPU (burstable) / 1GB RAM        | 2 vCPU (burstable) / 1GB RAM        |
| Compute Type     | Shared container environment           | Dedicated ARM instance (Graviton2)    | Dedicated ARM instance (Graviton2)  | Dedicated ARM instance (Graviton2)  |
| OS Access        | No                                     | Yes                                   | Yes                                 | Yes                                 |
| Storage Included | 8GB                                    | 20GB gp3                              | 20GB gp3                            | 20GB gp3                            |
| Extra Storage    | \$0.125/GB                             | \$0.115/GB                            | \$0.115/GB                          | \$0.115/GB                          |
| Backups          | 7-day snapshot                         | Free up to DB size, then \$0.095/GB\* | Free up to DB size, then \$0.095/GB | Free up to DB size, then \$0.095/GB |
| Bandwidth        | 250GB outbound included, then $0.09/GB | \$0.09/GB outbound                    | \$0.09/GB outbound                  | \$0.09/GB outbound                  |

- **Supabase Pro** is great if you want simple, predictable pricing without dealing with EC2, storage classes, or IOPS tuning.
- **RDS On-Demand** offers low-cost dedicated compute with more configurability.
- **Reserved instances** (1yr and 3yr) reduce costs dramatically, but require long-term commitment.

## 2. Mid-Tier Production (100GB Storage + Moderate Usage)

For established applications with real user traffic and non-trivial data volumes, cost differences and platform flexibility become more significant.

| Feature           | Supabase (Large)                    | RDS m5.large (On-Demand) | RDS m5.large (1yr Reserved) | RDS m5.large (3yr Reserved) | Aurora r5.large (On-Demand)      | Aurora r5.large (1yr Reserved) | Aurora r5.large (3yr Reserved) |
| ----------------- | ----------------------------------- | ------------------------ | --------------------------- | --------------------------- | -------------------------------- | ------------------------------ | ------------------------------ |
| Monthly Price     | \$110 (flat)\*                      | \$130                    | \$81                        | \$56                        | \$211                            | \$138                          | \$96                           |
| Compute           | 2-core ARM (dedicated) / 8GB RAM    | 2 vCPU / 8GB RAM         | 2 vCPU / 8GB RAM            | 2 vCPU / 8GB RAM            | 2 vCPU / 16GB RAM                | 2 vCPU / 16GB RAM              | 2 vCPU / 16GB RAM              |
| Compute Type      | Dedicated ARM instance              | Dedicated EC2 (x86)      | Dedicated EC2 (x86)         | Dedicated EC2 (x86)         | Aurora cluster (I/O-Optimized)\* | Aurora cluster (I/O-Optimized) | Aurora cluster (I/O-Optimized) |
| OS Access         | No                                  | Yes                      | Yes                         | Yes                         | Yes                              | Yes                            | Yes                            |
| Storage (100GB)   | Included                            | \$11.50 (gp3)            | \$11.50 (gp3)               | \$11.50 (gp3)               | \$10 (I/O-Optimized)             | \$10 (I/O-Optimized)           | \$10 (I/O-Optimized)           |
| Backup            | Included                            | Free up to DB size\*     | Free up to DB size          | Free up to DB size          | \$0.021/GB\*                     | \$0.021/GB                     | \$0.021/GB                     |
| Bandwidth (500GB) | \$22.50 (250GB included + $0.09/GB) | \$45 (\$0.09/GB)         | \$45 (\$0.09/GB)            | \$45 (\$0.09/GB)            | \$45 (\$0.09/GB)                 | \$45 (\$0.09/GB)               | \$45 (\$0.09/GB)               |
| **Total/Month**   | **\$145**                           | **\$186**                | **\$138**                   | **\$112.50**                | **\$266**                        | **\$193**                      | **\$161**                      |

- **Supabase (Large)** bundles all costs and removes infrastructure complexity — ideal for fast-moving teams.
- **RDS Reserved** (especially 3-year) cuts monthly bills by over 50%.
- **Aurora Reserved** costs more, but adds built-in high availability, multi-AZ replication, and better scaling.

## 3. Heavy Workload (500GB+ Storage, High Throughput)

For mission-critical workloads with large storage, high concurrency, and peak traffic, this tier shows how pricing stacks up across platforms.

| Feature              | Supabase 2XL                        | RDS r5.xlarge (On-Demand) | RDS r5.xlarge (1yr Reserved) | RDS r5.xlarge (3yr Reserved) | Aurora r5.xlarge (On-Demand)     | Aurora r5.xlarge (1yr Reserved) | Aurora r5.xlarge (3yr Reserved) |
| -------------------- | ----------------------------------- | ------------------------- | ---------------------------- | ---------------------------- | -------------------------------- | ------------------------------- | ------------------------------- |
| Monthly Price        | \$410 (flat)\*                      | \$422                     | \$246                        | \$170                        | \$422                            | \$287                           | \$210                           |
| Compute              | 8-core ARM (dedicated) / 32GB RAM   | 4 vCPU / 32GB RAM         | 4 vCPU / 32GB RAM            | 4 vCPU / 32GB RAM            | 4 vCPU / 32GB RAM                | 4 vCPU / 32GB RAM               | 4 vCPU / 32GB RAM               |
| Compute Type         | Dedicated ARM instance              | Dedicated EC2 (x86)       | Dedicated EC2 (x86)          | Dedicated EC2 (x86)          | Aurora cluster (I/O-Optimized)\* | Aurora cluster (I/O-Optimized)  | Aurora cluster (I/O-Optimized)  |
| OS Access            | No                                  | Yes                       | Yes                          | Yes                          | Yes                              | Yes                             | Yes                             |
| Storage (500GB)      | Included                            | \$57.50 (gp3)             | \$57.50 (gp3)                | \$57.50 (gp3)                | \$50 (I/O-Optimized)             | \$50 (I/O-Optimized)            | \$50 (I/O-Optimized)            |
| IOPS / Throughput    | Included (abstracted)\*             | \$100+ (io1 estimated)\*  | \$100+ (io1 estimated)       | \$100+ (io1 estimated)       | Included                         | Included                        | Included                        |
| Backup (500GB extra) | Included                            | \$47.50\*                 | \$47.50                      | \$47.50                      | \$10.50                          | \$10.50                         | \$10.50                         |
| Bandwidth (1TB)      | \$67.50 (250GB included + $0.09/GB) | \$90 (\$0.09/GB)          | \$90 (\$0.09/GB)             | \$90 (\$0.09/GB)             | \$90 (\$0.09/GB)                 | \$90 (\$0.09/GB)                | \$90 (\$0.09/GB)                |
| **Total/Month**      | **\$477.5**                         | **\$717**                 | **\$541**                    | **\$465**                    | **\$572**                        | **\$437**                       | **\$361**                       |

- **Supabase 2XL** is an all-inclusive bundle that scales without requiring DBAs or infra tuning.
- **RDS Reserved** (3yr) delivers maximum cost-efficiency if your workload is stable.
- **Aurora Reserved** is ideal for high-volume, multi-region, or high-availability requirements.

## Cost Reference for Storage & Compute

| Metric        | Supabase                | AWS RDS (On-Demand) | AWS RDS (Reserved)\* | Aurora (I/O-Optimized)\* |
| ------------- | ----------------------- | ------------------- | -------------------- | ------------------------ |
| **Storage**   | \$0.125/GB              | \$0.115/GB (gp3)\*  | \$0.115/GB (gp3)     | \$0.10–\$0.225/GB\*      |
| **Backup**    | Included                | \$0.095/GB\*        | \$0.095/GB           | \$0.021/GB (snapshot)\*  |
| **Bandwidth** | 250GB outbound included | \$0.09/GB outbound  | \$0.09/GB outbound   | \$0.09/GB outbound       |
| **Compute**   | \$10–\$3,730 (flat)     | \$11–\$1,688        | \$6–\$1,080          | \$67–\$3,376             |

- **Reserved pricing** can reduce compute cost by 30–60%, especially for year-long or 3-year commitments.
- Aurora charges **by I/O operations**, unless you're on their newer I/O-optimized pricing model.

## 📝 Explanatory Notes

- **t4g.micro (burstable)**: AWS uses burstable instances like `t4g.micro` for its Free and entry-level RDS tiers. These provide 2 ARM-based vCPUs with a CPU credit system — ideal for low-to-moderate workloads with occasional spikes.

- **gp3 Storage (AWS)**: gp3 is now the default storage type for new RDS instances. It provides a **baseline of 3,000 IOPS and 125 MB/s throughput** at no extra cost — a significant improvement over the older gp2 type. The per-GB price remains the same at \$0.115/GB-month.

- **Supabase Flat Pricing**: Supabase’s pricing includes compute, storage, backups, and bandwidth in a **single monthly rate** — simplifying cost tracking and reducing surprise bills.

- **Backup Costs (AWS)**: RDS provides **free backup storage up to the size of your DB**. Additional snapshot storage is charged at:

  - \$0.095/GB for RDS
  - \$0.021/GB for Aurora

- **Aurora I/O-Optimized**: This newer Aurora pricing tier eliminates per-I/O charges and instead bills a flat rate per GB stored. Ideal for write-heavy or high-throughput workloads with unpredictable I/O.

- **IOPS Costs (RDS)**: For high-performance needs beyond gp3's 3,000 IOPS baseline, RDS users may provision additional IOPS on gp3 or upgrade to `io1`/`io2` storage — typically adding \$100+ per month for workloads requiring consistent low-latency throughput.

- **Reserved Pricing (AWS)**: RDS and Aurora support **1-year and 3-year reservations**, reducing monthly compute costs by 30–60%. These are billed upfront or monthly and require workload stability.

## Conclusion

| Use Case                           | Best Choice     | Price Range (Monthly) | Why                                                                 |
| ---------------------------------- | --------------- | --------------------- | ------------------------------------------------------------------- |
| Free hobby project                 | Supabase Free   | \$0                   | No time limit, zero config                                          |
| Low-cost dev/test DB               | Supabase Pro    | \$25                  | Simple flat rate, includes compute, storage, and bandwidth          |
| AWS trial or AWS-focused team      | AWS Free Tier   | \$0 (12 mo)           | Best value compute & storage for teams already using AWS            |
| Cost-sensitive production workload | RDS Reserved    | \$4.76–\$96           | Long-term commitment cuts RDS compute cost by up to 60%             |
| Simple mid-sized app               | Supabase or RDS | \$110–\$186           | Supabase for ease, RDS for control & cost tuning                    |
| High write/read throughput         | Aurora Reserved | \$96–\$437            | Built-in performance, replication, and I/O-optimized pricing        |
| Multi-region / enterprise scale    | Aurora or RDS   | \$161–\$717           | Supports replication, multi-AZ, and enterprise-grade configurations |

- **Supabase** offers **simple pricing and easy setup**, ideal for fast-moving projects.
- **AWS RDS Reserved** is best for **long-term, cost-optimized workloads** with more control.
- **Aurora** suits **high-performance, high-availability needs**, but at a higher cost.

Choose based on your need for **simplicity vs control**, and how stable your usage will be. Supabase works well for early-stage apps; AWS shines for scaled, mature systems.

Need more than a database? Our next post will compare [Supabase vs AWS pricing across auth, messaging, and more](/blog/supabase-vs-aws-pricing/).


We compare Supabase vs AWS (RDS and Aurora) across free, entry-level, and production tiers

Supabase vs AWS: Database Pricing Comparison in 2026


Text-to-SQL felt easy at first.

You ask a question in plain English, and you get SQL. The query runs.

For small setups, that's often fine. If something looks wrong, someone notices and fixes it.

Things change once Text-to-SQL is used across shared databases, by many teams, with real business and compliance impact. At that point, the SQL doesn't just need to run. It needs to be right, safe, and something you can explain later.

That's the point where enterprise-grade Text-to-SQL turns into an engineering problem.

## Different teams hit the same problems

Very different teams ran into the same issues once they put Text-to-SQL in front of real enterprise data.

- OpenAI wrote about building an internal data agent so employees could ask questions about company data, and what broke once it was used day to day. [OpenAI Builds an Internal Data Agent](https://openai.com/index/inside-our-in-house-data-agent/)

- Google Cloud shared practical techniques they had to add just to get Text-to-SQL working reliably over large, messy schemas. [Getting AI to write good SQL: Text-to-SQL techniques explained](https://cloud.google.com/blog/products/databases/techniques-for-improving-text-to-sql)

- Hex reflected on what actually matters when you ship AI features to real users and the environment keeps changing underneath you. [Bitter lessons building AI products](https://hex.tech/blog/bitter-lessons-building-ai-in-hex-product-management/)

- Vercel explained why they eventually removed most of their agent tools in production to make the system easier to reason about. [We removed 80% of our agent’s tools](https://vercel.com/blog/we-removed-80-percent-of-our-agents-tools)

Different products. Different users. But once you get into the details, the problems and the fixes start to look very familiar.

## Context: where things first went wrong

Almost everyone started the same way:

"Let's just give the model the schema."

That works until the schema is large, joins aren't obvious, and column names don't explain how the data is actually used.

At **OpenAI**, the problem wasn't just schemas, but choice. Their internal agent had many tools and many paths it could take. It was powerful, but hard to predict. When something went wrong, it was difficult to understand why. The fix wasn't better prompts. It was cutting things down: fewer tools, clearer scope, more fixed steps.

At **Google Cloud**, the failures showed up as incorrect joins and missing filters. The model wasn't broken — it was guessing. Raw DDL didn't give enough signal. Their solution was to stop relying on schema text alone and instead add structure: explicit table relationships, column hints, and examples that reflected how the data is actually queried.

At **Vercel**, this showed up as operational risk. Too many tools meant too many failure modes. They removed most of them. The system became less flexible, but much easier to reason about.

At **Hex**, unclear context turned into confused analysts. If the system didn't reflect how metrics were defined in practice, people drew the wrong conclusions from the results.

Across all of these cases, the same thing happened:

Giving the system everything made it worse.
Giving it less, but making that "less" very clear, made it usable.

In practice, this meant things like:

- Preselecting relevant tables
- Defining allowed join paths
- Limiting what actions are even possible
- Grounding queries in real usage, not just schema text

## Evaluation: when "the query ran" stopped meaning much

The next issue was harder to spot.

Most bad queries didn't fail. They ran fine. They returned data. They were just wrong.

This shows up clearly in **Hex**'s write-up. Analysts would get results that looked reasonable but were slightly off. After that happened a few times, trust disappeared. Once users feel they need to double-check everything, the feature stops being useful.

**Google Cloud** described the same issue from a systems angle. Execution success doesn't tell you whether the query matches the question. A missing filter or a wrong join can quietly change the meaning.

At **OpenAI**, even after tightening context, similar questions could produce different queries unless extra checks were added. That inconsistency was a problem on its own.

**Vercel** focused on something very practical: when something breaks, can you tell why? If you can't explain a result, people won't rely on it.

Over time, teams stopped treating "it runs" as the finish line.

Instead, they started doing things that look very familiar to anyone who's worked on production systems:

- Checking query shape, not just syntax
- Watching for missing filters or suspicious joins
- Preferring repeatable output over "creative" variations
- Making the generated SQL visible so someone can reason about it

That's when Text-to-SQL stopped feeling like a trick and started behaving more like normal software.

## Governance: treating AI like any other database user

This is the part many early systems skipped.

In real systems, SQL is access to sensitive data, business logic, and regulated workflows. Letting AI generate SQL without limits is effectively giving it broad database access.

None of these teams stayed in that mode for long.

**Vercel** reduced what the system could do at all. Fewer capabilities meant fewer risks.

**OpenAI** leaned toward predictability. The agent needed to behave in ways people could anticipate, not surprise them.

**Google Cloud** embedded constraints directly into how queries are formed, instead of trusting the model to always "do the right thing".

**Hex** made transparency non-negotiable. Users could see the SQL and understand how a result was produced.

In practice, this all looks very boring — and that’s a good thing:

- Read-only by default
- Clear limits on tables and columns
- Extra checks for risky operations
- Logs of who asked what, and what actually ran

AI doesn't get special rules. It follows the same ones as everyone else.

## Different implementations, same shape

If you ignore the product names and just look at what changed, the shape is almost identical:

| Area       | What changed                                                |
| ---------- | ----------------------------------------------------------- |
| Context    | From full schemas to small, well-defined inputs             |
| Evaluation | From "it runs" to "does this actually answer the question?" |
| Governance | From implicit trust to explicit limits and logs             |

No one fixed this with clever prompting. No one waited for a better model. They fixed it by putting engineering discipline around the model.

## Closing

Enterprise Text-to-SQL isn't about letting AI write SQL for you. It's about deciding: what SQL is allowed, when it's acceptable, and who owns the result.

Different teams took different routes, but they all ended up in the same place: Text-to-SQL only works at scale when it's constrained, checked, and governed.

That's not what makes it flashy. It's what makes it usable.

Enterprise Text-to-SQL tools are essential for consistent results and compliance in large organizations.

Enterprise Text-to-SQL: Context, Evaluation, and Governance for Consistent Results


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database DevSecOps tool. We update the post every year.

</HintBlock>

| Update History | Comment                                                             |
| -------------- | ------------------------------------------------------------------- |
| 2024/09/11     | Initial version.                                                    |
| 2025/05/27     | 2025 Update.                                                        |
| 2025/12/04     | Add TablePlus and DBHub. Remove Outerbase (acquired by Cloudflare). |
| 2026/02/03     | 2026 Update. Update DataGrip and Vanna.ai with major new features.  |

This is a series articles about SQL Clients / SQL Editors:

1. [Open Source SQL Clients](/blog/top-open-source-sql-clients)
1. Text-to-SQL Query Tools (this one)

---

If you work with databases, SQL AI tools can significantly improve your productivity and efficiency. They help generate SQL commands automatically from natural language input and optimize query performance as functional and starter-friendly tool.

Here we gathered some of the most popular SQL AI tools (Text2SQL) on the market in 2026.

## DataGrip

JetBrains introduces an AI Assistant to its flagship SQL client [DataGrip](https://www.jetbrains.com/datagrip/).

![](/content/blog/top-text-to-sql-query-tools/datagrip.webp)

The DataGrip AI Assistant can:

1. Request queries and information using natural language.
1. Explain complex SQL such as stored procedures.
1. Optimize schema and SQL.
1. Compare DDL of two database objects.
1. Fix SQL mistakes.
1. Format and rewrite SQL.

During 2025, JetBrains significantly enhanced the AI Assistant:

- **Schema and object context**: Attach specific database objects (tables, views) to AI chat for more precise responses, not just the whole schema.
- **Execution plan analysis**: AI can explain query execution plans and suggest optimizations for better performance.
- **Cloud-based code completion**: Leverages cloud resources for more precise autocomplete, including single lines, code blocks, and entire scripts.
- **Cloud database integration**: Connect directly to AWS, Azure, and Google Cloud accounts to browse and connect to databases.
- **Free tier and local models**: A free tier offers unlimited local completions, and you can connect OpenAI-compatible local endpoints (Ollama, LM Studio) for offline use.

## TablePlus

[TablePlus](https://tableplus.com/) is considered by many as the best SQL client on Mac. It's a modern, native database management tool that supports multiple databases. It has integrated text-to-SQL AI capabilities to help developers generate queries from natural language.

![](/content/blog/top-text-to-sql-query-tools/tableplus.webp)

Unlike DataGrip which charges an extra price for its comprehensive AI Assistant features, TablePlus follows a BYOK (Bring Your Own Key) model, allowing you to pick your preferred LLM provider. While it only provides text-to-SQL generation rather than the full suite of AI features offered by DataGrip, this approach gives users more flexibility in choosing their AI backend.

![](/content/blog/top-text-to-sql-query-tools/tableplus-llm.webp)

## SQL Chat

![](/content/blog/top-text-to-sql-query-tools/sqlchat.webp)

[SQL Chat](https://www.sqlchat.ai/) is a chat-based SQL client, which uses natural language to communicate with the database to implement operations such as query, modification, addition, and deletion of the database. Instead of navigating across many UI controls, SQL chat applies a much more intuitive chat-based interface, bringing more flexible and comfortable user experience.

SQL Chat supports the following databases and will add more over time:

- MySQL
- PostgreSQL
- MSSQL
- TiDB Cloud

## Vanna.ai

![](/content/blog/top-text-to-sql-query-tools/vanna.webp)

[Vanna.ai](https://vanna.ai/) is a personalized AI SQL agent that transforms natural language questions into actionable database insights. In late 2025, Vanna released **Vanna 2.0**, a complete architectural rewrite that evolved from a simple SQL generation library into a production-ready, user-aware agent framework.

Key features of Vanna 2.0:

- **Agent-based architecture**: New Agent-based API replacing the legacy VannaBase class methods, with better support for agentic models like Claude 4.5 and GPT-5.
- **User-aware components**: Identity flows through every layer with user context automatically available in tools, SQL queries, and audit logs.
- **Enterprise security**: Row-level security with queries filtered based on user permissions, group-based access control, and comprehensive audit logging for compliance.
- **Web-first approach**: Built-in `<vanna-chat>` component and server with streaming and rich UI components.
- **NVIDIA NIM integration**: Accelerated inference using NVIDIA NIM microservices for improved performance.

The platform supports major databases including PostgreSQL, MySQL, Snowflake, BigQuery, Redshift, SQLite, Oracle, SQL Server, DuckDB, ClickHouse, and more. For LLMs, it supports OpenAI, Anthropic Claude, Google Gemini, Azure OpenAI, Ollama, and others.

## SQLAI.ai

[SQLAI.ai](https://www.sqlai.ai/) is divided into several SQL generators, each serving specific purposes:

- **Explain SQL Queries**: Offers explanations with summaries, output visualizations, and detailed query breakdowns.
- **Format SQL Query**: Formats SQL queries for improved readability and reduced error-proneness.
- **Analyze Your Data**: Allows you to upload CSV data and ask AI questions.
- **Generate SQL Query, Fix SQL Queries, Optimize SQL Query, etc.**

![](/content/blog/top-text-to-sql-query-tools/sqlai.webp)

In addition to the core generators, SQLAI.ai includes tools to facilitate your work. The available tools vary depending on the generator.

## Honorable mention

### Postgres.new

[Postgres.new](https://postgres.new/) is an WASM-based in-browser PostgreSQL sandbox with AI assistance. It allows users to operate with PostgreSQL directly in a web browser, without having to install or set up the database locally.

![](/content/blog/top-text-to-sql-query-tools/postgresnew.webp)

Postgres.new is currently in early alpha with limited functionality. It may have limitations compared to a full native PostgreSQL installation.

### DBHub (MCP Server)

Unlike standalone SQL clients, [DBHub](https://dbhub.ai) brings text-to-SQL capabilities directly into your existing development workflow through the MCP standard, eliminating the need to switch between tools.

![](/content/blog/top-text-to-sql-query-tools/dbhub.webp)

DBHub is a universal database MCP (Model Context Protocol) server that enables any MCP client such as Claude, Cursor, VS Code, and Codex to have text-to-SQL capability. It acts as a bridge between AI assistants and databases, allowing developers to use natural language to query and explore their databases directly from their development tools.

As of early 2026, DBHub has over 100K downloads and 2K GitHub stars. It supports multiple databases including PostgreSQL, MySQL, MariaDB, SQL Server, and SQLite. Key features include:

- **Custom tools**: Define reusable, parameterized SQL operations in configuration files.
- **Built-in web interface**: Visual way to execute queries, run custom tools, and view request traces without requiring an MCP client.
- **Security-first design**: Read-only mode and safety controls to prevent accidental data modifications.

Compared with other database MCP servers, DBHub comes with an admin console to view configured data sources and trace SQL requests.

![](/content/blog/top-text-to-sql-query-tools/dbhub-console.webp)

## Techniques and Benchmarks

Cloud hyperscalers are increasingly integrating Text-to-SQL capabilities across their database offerings. Google Cloud, for example, has rolled out this feature to BigQuery, Cloud SQL, and AlloyDB, and recently published [guidance](https://cloud.google.com/blog/products/databases/techniques-for-improving-text-to-sql) on effective AI prompting techniques for SQL generation. Meanwhile, the research community has developed evaluation frameworks like [BIRD-Bench](https://bird-bench.github.io/) to measure Text-to-SQL accuracy.

## Security Considerations

When Anthropic introduced MCP, the Postgres server was one of the original reference implementations. Security reviews of early MCP-based database servers have shown that simply wrapping user-supplied SQL in a read-only transaction is not sufficient if the server also accepts semicolon-delimited statements, because additional statements may execute outside the intended protection. When evaluating MCP-based database tools, prioritize implementations with robust security controls such as proper SQL parsing or allow-listing, strict read-only enforcement, and clear configuration for which databases and schemas can be accessed.

Text-to-SQL tools improve efficiency and make it easier to interact with databases. Although they use state-of-the-art LLM models, the results may sometimes be inaccurate. If you want to use them in production, make sure to double-check before you hit **RUN**.


An overview of the most popular text-to-SQL AI tools in 2026, including DataGrip, TablePlus, SQL Chat, Vanna.ai, SQLAI.ai, Postgres.new, and DBHub MCP server. Compare features and capabilities to find the best SQL AI assistant for your workflow.

Top 5 Text-to-SQL Query Tools in 2026

An extensive comparison between Postgres and MySQL on performance, features, security, extensibility, usability, architecture, ecosystem, industry best practices.

Postgres vs. MySQL: a Complete Comparison in 2026


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database DevSecOps tool that is a Liquibase/Flyway alternative. We update the post every year.

</HintBlock>

| Update History | Comment                                                 |
| -------------- | ------------------------------------------------------- |
| 2023/06/27     | Initial version.                                        |
| 2025/05/23     | Update for 2025.                                        |
| 2026/02/03     | Flyway 2026, Liquibase 5.0, pricing/licensing changes.  |

When looking for a database CI/CD and schema migration change tool, Flyway (Redgate) and Liquibase are two common options. Understanding the differences between these two tools can help potential users choose the one that best meets their needs.

## What Flyway and Liquibase have in common

- Java-based, providing Java SDK and CLI.
- Provide open-source community version and closed-source commercial version.
- Native SQL supported, however Liquibase requires XML changelog to define the changes.
- Database CI/CD with version control system #GitOps (see [Database GitOps configuration](#database-gitops-configuration)).
- Auto SQL check (see [Auto SQL check](#sql-auto-check)).
- Schema Synchronization (see [Sync schema](#sync-schema)).
- Tiered pricing, offering both free and paid plan.
- Open source, both has long history.
  ![star-history](/content/blog/flyway-vs-liquibase/star-history-flyway-vs-liquibase.webp)

## What are the differences between Flyway and Liquibase?

While both Flyway and Liquibase are tools for database CI/CD, there are some key differences between the two. The following table summarizes the differences between Flyway and Liquibase.

|                                                        | Flyway                          | Liquibase                              |
| ------------------------------------------------------ | ------------------------------- | -------------------------------------- |
| [Product position](#product-position)                  | Schema Change & Version Control | Schema Change & Version Control        |
| [Developer interface](#developer-interface)            | CLI + Flyway Desktop            | CLI                                    |
| [Supported databases](#supported-databases)            | 22 Only SQL + MongoDB (preview) | 50 SQL & NoSQL DB                      |
| [Programming language and installation](#installation) | Java + JVM                      | Java + JVM                             |
| [Change execution](#change-execution)                  | SQL script + CLI                | Changelog (SQL, XML, JSON, YAML) + CLI |
| [Change orchestration](#change-orchestration)          | Numbering of SQL files          | Changelog + Flow                       |
| [Database GitOps](#database-gitops-configuration)      | ✅                              | ✅                                     |
| [SQL auto check](#sql-auto-check)                      | ✅                              | ✅                                     |
| [Change history](#change-history)                      | ✅                              | ✅                                     |
| [Sync schema](#sync-schema)                            | ✅                              | ✅                                     |
| [Rollback](#rollback)                                  | ✅                              | ✅                                     |
| [Schema drift detection](#schema-drift-detection)      | ✅                              | ✅                                     |

### Product position

- **Flyway**: A database schema change and version control tool.
  ![flyway-position](/content/blog/bytebase-vs-flyway/flyway-position.webp)

- **Liquibase**: A database schema change and version control tool. It helps you track, manage, and automate changes to your database.
  ![liquibase-position](/content/blog/flyway-vs-liquibase/liquibase-position.webp)

### Developer interface

- **Flyway**: A command-line tool with Java API, Maven plugin, and Gradle plugin. Flyway Desktop has been enhanced with full Schema Model view and Git history tracking, available for SQL Server, PostgreSQL, MySQL, and other supported databases.

- **Liquibase**: Primarily a command-line tool. The Liquibase Hub GUI was sunset in May 2023, with focus shifting to CLI and integration capabilities.

### Supported databases

- **Flyway**: 22+ SQL databases with support for:

  - MongoDB (Native Connectors mode only; JDBC support removed in 2025)
  - SinglestoreDB
  - Google Cloud Spanner
  - EnterpriseDB
  - TimescaleDB
  - Postgres 18, SQL Server 2025, Oracle 26ai (added 2025)
  - Full compatibility for all databases in Community version

- **Liquibase**: 50+ SQL and NoSQL databases with enhanced support for:
  - Databricks (added 2025)
  - Google BigQuery (including DATABASECHANGELOGHISTORY table support)
  - Continued MongoDB support
  - Snowflake with OAuth/PKI authentication

### Installation

- **Flyway**: Java-based tool requiring JVM installation. In 2025, configuration has moved to unified flyway.toml format, deprecating the older JSON format.

- **Liquibase**: Java-based tool requiring JVM installation. **Liquibase 5.0+ requires Java 17 minimum** (Java 8/11 deprecated). MacOS .dmg installer was deprecated in favor of Homebrew installation.

### Change execution

- **Flyway**: CLI or GitOps. Users write SQL files and then run command `flyway migrate`. In 2025, enhanced capabilities for script migrations and callbacks have been added to the Community version.

![flyway-change](/content/blog/bytebase-vs-flyway/flyway-change.webp)

- **Liquibase**: CLI or GitOps. Users specify the changes by defining a `changelog` and then run a command. In 2025, flow enhancements provide more sophisticated orchestration capabilities.

![liquibase-changelog](/content/blog/flyway-vs-liquibase/liquibase-changelog.webp)
![liquibase-update](/content/blog/flyway-vs-liquibase/liquibase-update.webp)

### Change orchestration

- **Flyway**: Number the SQL files in the order you want them to be executed.

  ![liquibase-change-order](/content/blog/flyway-vs-liquibase/flyway-change-order.webp)

- **Liquibase**: Specify the order of changes in the changelog file. In 2025, Flow capabilities have been enhanced with conditionals and more advanced orchestration features.

  ![liquibase-change-order](/content/blog/flyway-vs-liquibase/liquibase-change-order.webp)

Liquibase continues to be more flexible as it can specify arbitrary orders, and the enhanced [flow file](https://docs.liquibase.com/liquibase-pro/flow/home.html) capabilities in 2025 make orchestrating complex steps even more powerful.

### Database GitOps configuration

- **Flyway**: Configure with VCS CI/CD workflow manually. In 2025, integration with Test Data Manager and Docker for dedicated development databases improves the GitOps workflow.

- **Liquibase**: Configure with VCS CI/CD workflow manually. In 2025, enhanced validation and error messaging improve the GitOps experience.

  ![flyway-ci-cd](/content/blog/bytebase-vs-flyway/flyway-ci-cd.webp)
  ![liquibase-gitlab-gitops](/content/blog/flyway-vs-liquibase/liquibase-gitlab-gitops.webp)

### SQL auto check

SQL auto check helps developers write less buggy SQL and save DBAs manual review efforts.

- **Flyway**: Code Analysis with enhanced drift detection in 2025
- **Liquibase**: SQL Quality check with improved Policy Checks in 2025

#### Supported plan

- **Flyway**: Enterprise only (Teams discontinued in 2025)
- **Liquibase**: Paid tiers only (Starter and above)

#### Number of rules

- **Flyway**: 10+ general rules or integrate SQLFluff for more rules
- **Liquibase**: Enhanced policy checks with more comprehensive validation in 2025

#### How to configure

- **Flyway**: Predefined, users may set them active or not. In 2025, improved configuration through unified flyway.toml.

  ![flyway-code-analysis-rules](/content/blog/bytebase-vs-flyway/flyway-code-analysis-rules.webp)

- **Liquibase**: Predefined, users may set levels while calling. Python-based Custom Policy Checks available in paid tiers.

  ![liquibase-quality-check](/content/blog/flyway-vs-liquibase/liquibase-quality-check.webp)

#### How to trigger

- **Flyway**: Run `flyway check -code ....` command to produce a report.
  ![flyway-check-code](/content/blog/bytebase-vs-flyway/flyway-check-code.webp)
  ![flyway-code-analysis-report](/content/blog/bytebase-vs-flyway/flyway-code-analysis-report.webp)

- **Liquibase**: Run `check` command or manually integrate this command in automation. Users may go to admin console to view logs.
  ![liquibase-validate-result](/content/blog/flyway-vs-liquibase/liquibase-validate-result.webp)

### Change history

- **Flyway**: Run `flyway info` to show `flyway_schema_history` table. In 2025, enhanced info filters provide more granular control.

  ![flyway-info](/content/blog/bytebase-vs-flyway/flyway-info.webp)

- **Liquibase**: Simple Database Change Logs. In 2025, new parameters for the `history` command and improved Operation Reports.

  ![liquibase-logs](/content/blog/flyway-vs-liquibase/liquibase-logs.webp)

### Sync schema

- **Flyway**: Via Flyway Desktop, generate a migration script to bring the target database schema in sync. In 2025, the new backup-based baseline approach allows restoring a backup file instead of creating a database from scratch in the baseline script.

  ![flyway-sync-schema](/content/blog/bytebase-vs-flyway/flyway-sync-schema.webp)

- **Liquibase**: Support `diff-changelog` to compare databases and create a deployable changelog to sync. In 2025, enhanced Drift Report capabilities improve schema synchronization.

### Rollback

- **Flyway**: Write rollback scripts manually. Enterprise only (Community does not support rollback).

- **Liquibase**: Support `rollback-one-changeset` or `rollback`. Enhanced Rollback Report in paid tiers.

### Schema drift detection

- **Flyway**: Run `flyway check -drift ...` to produce a report indicating difference between target database and the one created by the migrations applied by Flyway. Enhanced in 2025 with more comprehensive code analysis.

- **Liquibase**: Run `liquibase diff --format=json` to produce a report indicating difference between target database and source database. In 2025, improved Drift Report capabilities with better visualization.

## Flyway enhancements in 2025-2026

Key features released in [2025](https://www.red-gate.com/blog/redgate-flyway-2025-year-in-review):

- **State-based deployments** - Work the way you want with declarative schema definitions
- **Backups as baseline** - Use database backups instead of lengthy baseline scripts, solving issues with invalid objects or external server references
- **Improved drift detection** - Snapshots and drift resolution scripts to identify and resolve schema drift
- **Database support** - Added support for Postgres 18, SQL Server 2025, Oracle 26ai

[January 2026 updates](https://www.red-gate.com/blog/redgate-flyway-product-updates-january-2026):

- **AI-powered migration descriptions** - Flyway Desktop 8 generates descriptions based on migration script contents
- **New shadow database dialog** - Option to let Flyway create shadow databases automatically
- **Schema model view** - View all object definitions in Flyway Desktop
- **SQLFluff 4.0.0** - Upgraded from 3.5.0

**Important licensing change**: Since May 2025, Flyway only offers Community (free) and Enterprise (paid). The Teams tier was discontinued for new customers.

## Liquibase enhancements in 2025-2026

[Liquibase 5.0](https://docs.liquibase.com/community/release-notes/5-0) was released in September 2025 as a major release:

- **Java 17 minimum requirement** - Java 8 and 11 deprecated for improved performance and security
- **Liquibase Secure 5.0** - Certified enterprise distribution with integrated extensions, drivers, and policies
- **Liquibase Secure Developer** - New VS Code extension for enterprise users
- **AI Changelog Generator** - MCP Server powered tool to convert natural language to production-ready changelogs (Private Preview)
- **Databricks support** - Added database connection support

Flow and automation enhancements:

- **Conditional Flow File logic** - Based on action exit codes with new `continueOnError` property
- **Snowflake OAuth/PKI authentication** - Simplified authentication for Snowflake's token-based requirements
- **New policy check** - `FormattedSqlHeaderRequired` ensures proper SQL file formatting
- **Python checks enhancements** - New `get_config_value()` helper function

**Important licensing change**: Liquibase Community has adopted the [Functional Source License (FSL)](https://docs.liquibase.com/community/release-notes/5-0) to ensure long-term sustainability while keeping the software free for developers.

## Pricing

Flyway uses a per-user licensing model, while Liquibase uses a per-application/database-type licensing model.

| Aspect               | Flyway                            | Liquibase                                        |
| -------------------- | --------------------------------- | ------------------------------------------------ |
| **Free Tier**        | Community                         | Community (FSL license)                          |
| **Paid Tiers**       | Enterprise only                   | 4 tiers: Starter, Growth, Business, Enterprise   |
| **Licensing Model**  | Per-user                          | Per-application + database types                 |
| **Entry-Level**      | Enterprise (contact for quote)    | Starter: up to 5 apps, 1 DB type                 |
| **Mid-Tier**         | —                                 | Growth (10 apps, 3 DB types), Business (25 apps) |
| **Enterprise**       | Unlimited (custom pricing)        | Unlimited apps & DB types, 24/7 support          |
| **Pricing**          | Contact sales                     | Contact sales (all tiers)                        |

## Summary

Both Flyway and Liquibase are Java-based, offering with SDKs and CLI support, and follow a [migration-based](/blog/database-version-control-state-based-vs-migration-based/#migration-based-version-control-imperative) approach to schema changes.
They also leverage the same open-source monetization strategy. While Flyway is generally considered more developer-friendly, Liquibase offers a broader set of advanced features.

The key difference is that Liquibase introduces the Changelog and Flow concepts, enabling users to specify explicit migration ordering, preconditions, labels, and contexts. In contrast, Flyway determines migration order based solely on file naming conventions.

Both tools offer a Git-like experience for database migrations. However, if you're seeking a GitHub/GitLab-style experience with a user-friendly GUI and team collaboration features, consider our own Bytebase. Continue reading for a side-by-side comparison with each tool:

- [Bytebase vs. Liquibase](/blog/bytebase-vs-liquibase/)
- [Bytebase vs. Flyway](/blog/bytebase-vs-flyway/)


When looking for a database CI/CD and schema migration change tool, Flyway and Liquibase are two common options. Understanding the differences between these two tools can help potential users choose the one that best meets their needs.

Flyway (Redgate) vs. Liquibase in 2026


Database changes are easy. Deploying them is where things go wrong.

A missing migration, a schema drift, or a change applied to the wrong environment can break production fast — often without anyone noticing until users complain or dashboards go red.

DevOps teams rely on a small set of tools to deploy database changes. On the surface, many of these tools look similar. In practice, they solve very different problems.

This article breaks down the database deployment tools teams actually use, and how they differ in how changes are defined, applied, and coordinated.

## What we mean by "Database Deployment Tools"

Before listing tools, it's important to be clear about scope.

This article focuses on general-purpose database deployment tools — tools designed to manage and deploy database schema changes directly.

We intentionally exclude:

- **ORM-driven tools** where the database schema is derived from application models
  (for example, Prisma Migrate or Drizzle)
- **CI/CD systems and infrastructure tools**
  (GitHub Actions, GitLab CI, Terraform)
- **Framework-specific migration systems**
  (Rails or Django migrations)

Many of these tools can run SQL. That alone doesn't make them database deployment tools in the sense discussed here.

## A simpler way to understand the landscape

Most confusion around database deployment tools comes from mixing different concerns together.

In reality, there are three separate questions:

1. **How are database changes defined?**
   As migrations, or as a desired schema state?
2. **How are those changes executed?**
   Almost always via a CLI.
3. **How are deployments coordinated across teams and environments?**
   Reviews, approvals, rollout, audit, and visibility.

Once you separate these, the tooling landscape becomes much easier to reason about.

## CLI-based database deployment tools

Most database deployment tools today are **CLI-based**.

You run a command locally or in CI. The tool connects to a database and applies changes.
Where tools differ is not how they execute, but how database changes are defined.

### Migration-based tools (CLI)

Migration-based tools deploy database changes as explicit migration scripts, usually written in SQL.

Each migration represents a small, incremental change. The tool tracks what has already been applied and runs the rest in order.

Some tools use simple version numbers. Others use dependencies. Both approaches are still migration-based.

**Common traits**

- Explicit migration files
- Predictable execution order
- Familiar mental model
- Easy to adopt

**Tools you'll see most often**

- **Flyway** Probably the most widely used migration tool. Straightforward versioned SQL files, broad database support, and easy CI integration.

- **Liquibase** Common in larger organizations. Supports SQL migrations and higher-level change definitions. Powerful, but heavier and more complex.

- **Goose** Uses plain SQL with explicit up/down sections. Often seen in Go-based systems.

- **Sqitch** Still migration-based, but uses dependencies instead of linear version numbers. Better suited for complex or parallel database development.

**Where migration-based tools work well**

- Application-driven schema changes
- Incremental database evolution
- Teams comfortable owning SQL migrations

**Where they fall short**

- Cross-team coordination
- Visibility across environments
- Approval, audit, and change control

At scale, migrations often become "just another pipeline step" — even though their impact is much higher than application code.

### Declarative / state-based tools (CLI)

Declarative tools take a different approach.

Instead of writing step-by-step migrations, you define what the schema should look like, and the tool figures out how to get there.

Execution still happens via a CLI.

**Representative tools**

- **Atlas** Compare the desired schema state with the actual database schema, generates the diff, and applies the required SQL.

- **pgschema** A Terraform-style declarative schema migration tool for Postgres. Works directly with schema files and the target database — no migration table or shadow database required.

**Where this works well**

- Teams comfortable with declarative workflows
- Tightly controlled environments
- Enforcing schema consistency

**Tradeoffs**

- Less explicit control over each change
- Requires trust in diff generation
- Riskier for large, long-lived production databases

Migration-based and declarative tools differ in how changes are defined, but they share the same execution model: a CLI applying changes directly to the database.

## Database deployment orchestration and control

CLI tools are good at one thing: applying changes.

They don't answer questions like:

- Who is allowed to deploy this?
- Has this change been reviewed?
- Which environments are affected?
- What happened after deployment?
- How do multiple teams avoid stepping on each other?

That's where orchestration/Bytebase comes in.

It is SQL-first and does not introduce a proprietary DSL. Database changes are written in native SQL, just like with migration tools.

In practice:

- Migration-based workflows are the primary model
- Declarative capabilities are also available for schema comparison and drift detection

What makes Bytebase different is not how SQL is written, but how deployments are coordinated.

Bytebase provides multiple ways to deploy the same SQL:

- A **GUI** for review, approval, and visibility
- An **API** for automation and internal platform integration
- A **GitOps workflow** for pull-request-driven deployments

Rather than replacing CLI tools, Bytebase acts as a **control plane** on top of them — managing policy, approval, rollout, and audit across environments.

## Putting it together

Seen clearly, the landscape is smaller than it first appears:

- **CLI-based tools** define and apply database changes
  - Migration-based: Flyway, Liquibase, Goose, Sqitch
  - Declarative: Atlas, pgschema
- **Orchestration tools** manage how those changes move through environments
  - Bytebase

There's no single "best" database deployment tool. Small teams may be fine with a simple migration tool. As systems grow, coordination and visibility become just as important as execution.

Database deployment isn't about running SQL.

It's about making changes **predictable, visible, and safe** as teams and systems scale.

Once you separate (1) how changes are defined, (2) how they're executed, and (3) how they're coordinated, choosing the right tools becomes much easier.


Database deployment tools for devops teams in 2026.

Database Deployment Tools for DevOps Teams in 2026


Schema changes are a normal part of software development. Columns get renamed, tables evolve, and teams clean up technical debt all the time.

The problem usually isn't the change itself.
It's **who else depends on it**.

Without proper **schema change detection and notification**, a small, reasonable change can quietly break analytics pipelines — often without anyone noticing until dashboards are already wrong.

This article walks through a very common scenario and explains why it keeps happening.

## A Setup Most Teams Will Recognize

The stack looks roughly like this:

- **PostgreSQL** running the production application
- **Fivetran** or **Airbyte** syncing tables into **Snowflake**
- **dbt** transforming that data for analytics
- Dashboards used by Finance and leadership

This setup works well. It's also fragile in ways that aren't obvious until something breaks.

## The Change That Starts It All

A backend developer notices a column called `amt` and decides to rename it to something clearer:

```sql
ALTER TABLE orders RENAME COLUMN amt TO total_amount;
```

They update the application code. Tests pass. The change is deployed.

From the application's point of view, everything is fine.

## What Breaks (And Why It's Not Obvious)

### The Data Sync Runs

A few hours later, the scheduled sync kicks in.

Fivetran or Airbyte expects a column called `amt`. Instead, it finds `total_amount`.

Depending on configuration, the sync may:

- Fail outright
- Or create a new column and treat the old one as deleted

In the second case, historical data often shows up as `NULL`. No error. Just missing values.

### dbt Starts Failing

Downstream, dbt models still reference the old column name:

```sql
SELECT amt FROM orders;
```

The warehouse responds with:

`invalid identifier 'AMT'`

Transforms stop running.

### Dashboards Are Either Broken — or Worse

By the time someone from Finance opens a dashboard:

- It may not load at all
- Or revenue shows as zero for yesterday

At this point, it's unclear what changed or when. The deploy happened yesterday. The impact shows up today.

Cue Slack messages.

## Why This Happens

Incidents like this usually come down to a few common gaps:

- **No visibility** — The backend dev had no idea downstream pipelines depended on that column name

- **No guardrails** — Nothing blocked or warned about the breaking change

- **Schema as implicit contract** — The column name was the API, but nobody documented or enforced it

Each one on its own is understandable. Together, they're how small changes turn into real incidents.

## Why Schema Change Detection and Notification Matter

Schema change detection isn't about stopping people from making changes.

It's about:

- Seeing what changed
- Understanding who might be affected
- Letting the right people know early

But once a schema change reaches production, notification alone is no longer enough.

At that point, the schema has already changed, and the question becomes: what should run next?

In mature workflows, a successful schema change actively triggers downstream pipelines:

- Data sync jobs refresh with the new schema
- dbt models are validated or updated
- Data quality checks are re-run
- Dashboards and derived tables are verified

Instead of waiting for the next scheduled run — or worse, for someone to notice broken data — the schema change itself becomes the trigger.

More advanced teams treat schema changes as first-class pipeline events:

- A migration is applied successfully
- The system emits a schema change event
- Downstream pipelines react automatically via webhooks or integrations

## How Bytebase Helps

Bytebase sits at the control point where schema changes are planned, reviewed, and applied — which makes it a natural place to turn schema changes into events.

Before deployment, Bytebase helps teams:

- Run SQL Review against the change automatically
- Require approval before changes reach production
- Keep a clear record of who approved what, and why

After deployment, Bytebase closes the loop.

When a schema change is successfully applied, Bytebase can emit **a post-deploy schema change signal** with rich metadata about the change (such as the database, environment, project, affected tables and columns, executor, and timestamp). This signal is sent as a **webhook** to downstream systems:

- Bytebase applies the schema change
- Bytebase posts a webhook to an endpoint
- That endpoint triggers pipeline actions, such as:
  - Running or validating data pipelines
  - Opening or updating dbt pull requests
  - Running data quality checks
  - Notifying the correct data owners with precise context via IM (e.g. Slack)

This shifts teams from reacting to schema changes after something breaks to coordinating and automating around them.

Instead of discovering schema changes through missing data or failed dashboards, teams see — and act on — schema changes the moment they happen.

Schema change detection and notification are essential for data pipelines. This article explains why a simple column rename can break your data pipeline and how to prevent it.

Schema Change Detection and Notification: Why a Simple Column Rename Can Break Your Data Pipeline


When a Postgres query runs slowly, the problem is rarely the SQL syntax itself. The real cause is usually **how Postgres chooses to execute the query**. That decision is captured in the **Postgres query plan**.

A query plan shows which tables are scanned, which indexes are used, how joins are performed, and how much work Postgres expects each step to take. Learning to read and reason about query plans is one of the most effective ways to diagnose performance problems and prevent regressions.

This article focuses on three practical skills:

1. How to read a Postgres query plan
2. How to troubleshoot common performance issues
3. How to fix query plan problems safely

---

## What Is a Postgres Query Plan?

A **Postgres query plan** is the execution strategy chosen by Postgres’ cost-based optimizer for a SQL statement.

When you run:

```sql
EXPLAIN SELECT * FROM orders WHERE customer_id = 42;
```

Postgres does not execute the query. Instead, it shows the steps it **would** take, including scan methods, join algorithms, and cost estimates.

Using:

```sql
EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 42;
```

Postgres executes the query and reports **actual execution time, row counts, and loops**, which makes it the most reliable way to understand real performance.

## How Postgres Builds a Query Plan

Postgres builds a query plan in several stages:

1. Parsing – validates SQL and resolves tables, columns, and functions
2. Rewriting – expands views and applies rules and security policies
3. Planning – evaluates execution strategies using table statistics and selects the lowest-cost plan
4. Execution – runs the chosen plan, consuming CPU, memory, and I/O

`EXPLAIN` shows the result of planning, while `EXPLAIN ANALYZE` includes execution.

| Command                    | Purpose                               |
| -------------------------- | ------------------------------------- |
| EXPLAIN                    | Shows estimated plan only             |
| EXPLAIN ANALYZE            | Executes query and shows real metrics |
| EXPLAIN (ANALYZE, BUFFERS) | Adds memory and disk usage            |
| EXPLAIN (ANALYZE, VERBOSE) | Shows internal details                |

Use `EXPLAIN ANALYZE` in **staging** or with care in **production**.

## How to Read a Postgres Query Plan

### Read bottom-up, not top-down

Postgres executes query plans **from the leaf nodes upward**.

```text
Hash Join
  -> Seq Scan on orders
  -> Index Scan on customers
```

Execution order: Scan `orders` -> Scan `customers` -> Join the results

### Key plan nodes you should recognize

| Node              | Meaning                          |
| ----------------- | -------------------------------- |
| Seq Scan          | Full table scan                  |
| Index Scan        | Reads rows via an index          |
| Bitmap Index Scan | Index lookup + heap access       |
| Nested Loop       | Repeats inner scan per outer row |
| Hash Join         | Builds an in-memory hash table   |
| Merge Join        | Joins sorted inputs              |

Most performance issues involve a small subset of these nodes.

### The 5 fields to check first

| Field                | Why it matters          |
| -------------------- | ----------------------- |
| **Actual Time**      | Real execution cost     |
| **Actual Rows**      | What actually happened  |
| **Estimated Rows**   | Planner expectation     |
| **Loops**            | Hidden work multipliers |
| **Scan / Join type** | Execution strategy      |

Large gaps between estimated and actual rows often indicate deeper problems.

## How to Troubleshoot Common Postgres Query Plan Problems

### Problem 1: Sequential Scan on a Large Table

**Symptoms**

- Seq Scan over millions of rows
- Long execution time

**Common causes**

- Missing index
- Low-selectivity filter
- Planner believes index access is more expensive

### Problem 2: Nested Loop With High Loop Count

**Symptoms**

- Nested Loop
- Inner scan runs thousands of times

**Common causes**

- Missing index on join keys
- Poor join order

### Problem 3: Large Gap Between Estimated and Actual Rows

**Symptoms**

```text
estimated rows = 10
actual rows = 100000
```

**Common causes**

- Outdated statistics
- Skewed data distribution
- Correlated columns

### Problem 4: Query Plan Changes After Deployment

**Symptoms**

- Same SQL suddenly slower
- Different plans across environments

**Common causes**

- Data growth
- Missing ANALYZE
- Schema or index changes

## How to Fix Postgres Query Plan Issues

### Fix 1: Add or Adjust Indexes (Carefully)

Indexes help when:

- Filters are selective
- Join columns are frequently used
- `ORDER BY` matches index order

Avoid indexing low-selectivity columns or adding indexes blindly.

### Fix 2: Refresh and Improve Statistics

```sql
ANALYZE;
```

Targeted:

```sql
ANALYZE orders (customer_id, created_at);
```

Extended statistics for correlated columns:

```sql
CREATE STATISTICS orders_stats
ON customer_id, status
FROM orders;
```

Accurate statistics lead to better plans.

### Fix 3: Rewrite the Query

Effective rewrites include:

- Pushing filters earlier
- Reducing result sets
- Replacing correlated subqueries
- Avoiding `SELECT *`

Small SQL changes can lead to **very different query plans**.

### Fix 4: Verify Improvements With EXPLAIN ANALYZE

Always confirm:

- Execution time improves
- Estimates align better with reality
- No new regressions appear

Never rely on estimated cost alone.

### Fix 5: Prevent Query Plan Regressions

Many performance issues appear **after deployment**.

Best practices:

- Review query plans during SQL review
- Compare plans across environments
- Detect plan changes as part of CI/CD

## When Postgres Query Plans Matter Most

- Slow queries
- Performance regressions
- Schema migrations
- Large datasets
- Production stability

If SQL is part of your delivery pipeline, **query plan review should be too**.

### Final takeaway

A Postgres query plan is a window into how Postgres thinks. Teams that understand and review query plans proactively fix performance problems **before** they reach production, **not after** users complain.

A comprehensive guide to understanding and analyzing Postgres query plans.

Postgres Query Plan: How to Read and Fix Performance Issues


## Introduction

Terraform has firmly established itself as the leading infrastructure as code (IaC) tool, enabling teams to define and provision infrastructure through declarative configuration files.

While Terraform's command-line interface serves as its primary interaction method, UI tools have emerged to address collaboration challenges, visualization needs, and governance requirements in enterprise environments.

This article explores the top four Terraform UI tools in 2025, examining their features, use cases, and how they can enhance your infrastructure management workflows.

## HashiCorp Terraform Cloud

![](/content/blog/top-terraform-ui-tools/hashicorp-terraform.webp)

Terraform Cloud is HashiCorp's official SaaS platform for Terraform, designed to bridge the gap between individual practitioners and enterprise teams.

**Key Features:**

- **Remote Terraform execution** with centralized state management
- **Version-controlled workflows** with integration to GitHub, GitLab, Bitbucket, and Azure DevOps
- **Collaborative plan reviews** with commenting functionality
- **Role-based access control** for enterprise users
- **Policy as Code** enforcement with Sentinel and OPA
- **Cost estimation** for planned infrastructure changes
- **Private module registry** for sharing reusable Terraform modules

**Best For:**

Organizations seeking an official, fully managed solution that scales from individual projects to enterprise-wide deployments. The tight integration with HashiCorp's ecosystem makes it particularly valuable for teams already invested in other HashiCorp tools.

**Pricing:**

Terraform Cloud offers three tiers:

- **Free**: For individuals and small teams (up to 5 users)
- **Team & Governance**: For mid-sized teams requiring collaboration features ($20 per user/month)
- **Business**: For enterprises needing advanced security and compliance features ($70 per user/month)

## Semaphore

![](/content/blog/top-terraform-ui-tools/semaphore.webp)

[Semaphore](https://github.com/semaphoreui/semaphore) is an open-source alternative to Terraform Cloud that provides a web UI for Terraform operations with a focus on team-based workflows and self-hosted deployment.

**Key Features:**

- **Web-based UI** for Terraform plan and apply operations
- **Task-based automation** for complex infrastructure workflows
- **Role-based access control** with fine-grained permissions
- **Audit logging** for compliance and tracking
- **Template-based deployment** with reusable components
- **Integration with Git repositories** for version control
- **Self-hosted architecture** for enhanced security and control

**Best For:**

Organizations that prefer self-hosted solutions with complete control over their infrastructure automation platform. Semaphore is particularly well-suited for teams with security requirements that prevent using cloud-based solutions.

**Pricing:**

Semaphore is open source and free to use, with the following options:

- **Community Edition**: Free and self-hosted
- **Enterprise Support**: Available through consulting partnerships

## Atlantis

![](/content/blog/top-terraform-ui-tools/atlantis.webp)

[Atlantis](https://github.com/runatlantis/atlantis) takes a unique approach by integrating Terraform workflows directly into pull request processes, with a lightweight server that automates Terraform plan and apply operations based on Git events.

**Key Features:**

- **Pull request automation** for Terraform workflows
- **Comment-driven operations** in GitHub, GitLab, or Bitbucket
- **Terraform plan visualization** within pull requests
- **Lock management** to prevent concurrent operations
- **Custom workflow definitions** for different repository structures
- **Webhook integration** with most Git providers
- **Container-based deployment** for easy hosting

**Best For:**

Developer-centric teams that prioritize Git-based workflows and want to integrate infrastructure changes directly into their existing code review processes. Atlantis shines in environments where developers manage infrastructure alongside application code.

**Pricing:**

Atlantis is completely open-source and free to use, with a vibrant community providing support and extensions.

## Scalr

![](/content/blog/top-terraform-ui-tools/scalr.webp)

[Scalr](https://www.scalr.com/) offers a comprehensive Terraform management platform designed for enterprise scale, with robust governance and collaboration features.

**Key Features:**

- **Multi-cloud management** across providers
- **Hierarchical organization** of environments (account, environment, workspace)
- **OPA-based policy engine** for governance
- **VCS integration** with major providers
- **Self-service infrastructure** through service catalog
- **Drift detection** and remediation
- **Advanced role-based access control** with custom roles
- **Private module registry** with approval workflows
- **Comprehensive API** for automation

**Best For:**

Large enterprises with complex organizational structures requiring granular control over Terraform deployments across multiple teams and environments. Scalr's hierarchical approach to infrastructure management makes it ideal for organizations with sophisticated governance requirements.

**Pricing:**

Scalr offers both self-hosted and SaaS options:

- **SaaS**: Starting at $50 per user/month with volume discounts
- **Self-hosted**: Custom enterprise pricing based on deployment size

## Feature Comparison

| Feature             | Terraform Cloud | Semaphore   | Atlantis      | Scalr               |
| ------------------- | --------------- | ----------- | ------------- | ------------------- |
| Deployment Model    | SaaS            | Self-hosted | Self-hosted   | SaaS or Self-hosted |
| State Management    | Built-in        | External    | External      | Built-in            |
| VCS Integration     | Comprehensive   | Basic       | Comprehensive | Comprehensive       |
| Module Registry     | ✅              | ❌          | ❌            | ✅                  |
| Policy Enforcement  | Sentinel/OPA    | Basic       | ❌            | OPA                 |
| Cost Estimation     | ✅              | ❌          | ❌            | ✅                  |
| API Availability    | Full            | Limited     | Limited       | Full                |
| Team Collaboration  | Advanced        | Basic       | PR-based      | Advanced            |
| Free Tier           | ✅ (limited)    | ✅          | ✅            | ❌                  |
| Enterprise Features | Comprehensive   | Basic       | Limited       | Comprehensive       |

## Choosing the Right Tool

When selecting a Terraform UI tool, consider the following factors:

1. **Team Size and Structure**: Larger, distributed teams benefit from the collaboration features in Terraform Cloud or Scalr.

2. **Governance Requirements**: Organizations with strict compliance needs should look to Terraform Cloud or Scalr's policy engine.

3. **Workflow Integration**: Developer-centric teams may prefer Atlantis for its tight integration with the pull request process.

4. **Deployment Preferences**: If you need to keep everything on-premises, Semaphore or Atlantis provide self-hosted options.

5. **Budget Constraints**: Open-source tools like Atlantis and Semaphore offer cost-effective solutions for teams with limited budgets.

## Conclusion

The Terraform ecosystem now offers diverse UI tools to enhance IaC workflows. Terraform Cloud provides the official, polished experience with direct HashiCorp integration. Semaphore offers a robust self-hosted alternative. Atlantis excels with Git-integrated workflows for developer-centric teams. Scalr delivers enterprise governance for complex organizations.

As IaC adoption grows, these tools will become increasingly vital for team collaboration while maintaining proper governance. Choose based on your specific team size, security requirements, existing workflows, and budget constraints.


Explore the leading Terraform UI tools of 2026 that enhance IaC workflows with visualization, collaboration, and governance features to streamline infrastructure management.

Top 4 Terraform UI Tools 2026


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database Management Software. We update the post once per year.

</HintBlock>

| Update History | Comment          |
| -------------- | ---------------- |
| 2024/01/16     | Initial version. |
| 2025/03/16     | Update for 2025. |
| 2026/01/20     | Update for 2026. |

PostgreSQL is the DBMS of the Year 2023 and the runner-up in 2024. On the other hand,
database schema migration in Postgres is still PITA. In this post, we are reviewing several open source
database schema migration tools for Postgres.

[![starhistory](/content/blog/top-open-source-postgres-migration-tools/star-history.webp)](https://star-history.com/#bytebase/bytebase&xataio/pgroll&liquibase/liquibase&flyway/flyway&graphile/migrate&fabianlindfors/reshape&Date)

## Bytebase

[Bytebase](https://github.com/bytebase/bytebase) is an all-in-one database DevOps and CI/CD solution. It's like GitHub/GitLab that provides
a GUI workspace for developers and DBAs to collaborate on database changes. It's written in Go and TypeScript.

![issue-detail](/content/blog/top-open-source-postgres-migration-tools/issue-detail.webp)

It also provides configurable SQL lint rules to detect SQL anti-patterns such as `NOT NULL` enforcement.

![sql-review](/content/blog/top-open-source-postgres-migration-tools/sql-review.webp)

Apart from taking care of database change management, Bytebase also provides a SQL Editor with data
access control, dynamic data masking and audit logs to control the query path.

![sql-editor](/content/blog/top-open-source-postgres-migration-tools/sql-editor.webp)

## Liquibase

[Liquibase](https://github.com/liquibase/liquibase) is arguably the most well-known product in this segment. It's CLI-based and written in Java.

In Liquibase, the schema migrations are organized as `Changeset` and `Changelog`. Probably due to its age and root in Java, the most commonly used form is XML (YAML and JSON support have been added later):

![_](/images/products/liquibase/liquibase-xml.webp)

Plain SQL is also supported with proper annotation:

![_](/images/products/liquibase/liquibase-sql.webp)

## Flyway

[Flyway](https://github.com/flyway/flyway) is another open source project with a long history and a large customer base. Its core product includes the CLI and a Java library.

Flyway has been acquired by Redgate, while it still retains a developer-oriented approach. This can be
felt from its website.

![_](/images/products/flyway/flyway-migration.webp)

Liquibase and Flyway are closely matched in their offerings. The key distinction between them is their target audiences: Liquibase is oriented towards serving enterprise clients, whereas Flyway presents itself as more developer-friendly and approachable.

## Reshape

[Reshape](https://github.com/fabianlindfors/reshape) takes a novel approach to achieve zero-downtime
schema migrations. It's CLI based and written in Rust. Reshape schema migration consists of 3 phases:

1. **Start migration** (`reshape migration start`): Sets up views and triggers to ensure both the new and old schema are usable at the same time.

1. **Roll out application**: Your application can be gradually rolled out without downtime. The existing deployment will continue using the old schema whilst the new deployment uses the new schema.

1. **Complete migration** (`reshape migration complete`): Removes the old schema and any intermediate data and triggers.

The author is now working on [ReshapeDB](https://reshapedb.com/), a new database built from the ground up to make zero-downtime schema and data migrations as simple and safe as possible.

![_](/images/products/reshape/reshape.webp)

FWIW, [pgroll](https://github.com/xataio/pgroll) is another Postgres schema
migration tool inspired by Reshape. pgroll is CLI based and written in Go. It's built by [Xata](https://xata.io/), the serverless database platform built on the top of PostgreSQL.

![_](/images/products/pgroll/pgroll-migration-flow.webp)

## graphile-migrate

[graphile-migrate](https://github.com/graphile/migrate) is an [opinionated](https://github.com/graphile/migrate?tab=readme-ov-file#opinions) SQL-powered productive roll-forward migration tool for PostgreSQL.

graphile-migrate is also CLI-based, written in TypeScript. It can also work with [PostGraphile](https://www.graphile.org/postgraphile/) to generate GraphQL schemas upon database changes.

Another novel idea graphile-migrate introduces is the `commit`, `uncommit` commands. It enables a
git-like workflow when developing against the local database.

- `graphile-migrate commit` commits the current migration into the `committed/` folder, resetting the
  current migration. Resets the shadow database.

- `graphile-migrate uncommit` Moves the latest commit out of the committed migrations folder and back to the current migration
  (assuming the current migration is empty-ish). Removes the migration tracking entry from **ONLY** the local database.

## pgschema

[pgschema](https://github.com/pgplex/pgschema) is a new CLI-based tool, written in Go. It features a terraform-like declarative schema migration workflow:

- **Dump** a Postgres schema in a developer-friendly format with support for all common objects
- **Edit** a schema to the desired state
- **Plan** a schema migration by comparing desired state with current database state
- **Apply** a schema migration with concurrent change detection, transaction-adaptive execution, and lock timeout control

[![asciicast](https://asciinema.org/a/vXHygDMUkGYsF6nmz2h0ONEQC.svg)](https://asciinema.org/a/vXHygDMUkGYsF6nmz2h0ONEQC)

## Summary

| Name                                                    | Interface | Language        | Postgres Only | Highlight                                                          |
| ------------------------------------------------------- | --------- | --------------- | ------------- | ------------------------------------------------------------------ |
| [Bytebase](https://github.com/bytebase/bytebase)        | GUI       | Go + TypeScript | N             | All in one for team collaboration.                                 |
| [Liquibase](https://github.com/liquibase/liquibase)     | CLI       | Java            | N             | Long-standing.                                                     |
| [Flyway](https://github.com/flyway/flyway)              | CLI       | Java            | N             | Developer-friendly.                                                |
| [Reshape](https://github.com/fabianlindfors/reshape)    | CLI       | Rust            | Y             | Zero-downtime based on view.                                       |
| [pgroll](https://github.com/xataio/pgroll)              | CLI       | Go              | Y             | Inspired by Reshape and more accessible via its database platform. |
| [graphile-migrate](https://github.com/graphile/migrate) | CLI       | TypeScript      | Y             | Opinionated with git-like workflow.                                |
| [pgschema](https://github.com/pgplex/pgschema)          | CLI       | Go              | Y             | Terraform-like declarative schema migration workflow               |

## Further Readings

- [Top Postgres Extensions](/blog/top-postgres-extension/)
- [Top Postgres GUI Clients](/blog/top-postgres-gui-client/)
- [Postgres vs. MySQL](/blog/postgres-vs-mysql)
- [Postgres vs. MongoDB](/blog/postgres-vs-mongodb)
- [MySQL Migration Tools](/blog/top-open-source-mysql-migration-tools)


What are the top open source Postgres schema migration tools.

The Anatomy of Postgres Permission ERROR 42501


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database Management Software. We update the post every year.

</HintBlock>

| Update History | Comment          |
| -------------- | ---------------- |
| 2025/03/14     | Initial version. |
| 2025/05/28     | Add Devtron.     |
| 2026/01/19     | Update for 2026. |

## What are the criteria?

When evaluating free and open-source Kubernetes dashboard solutions, several key criteria are essential to consider:

- **Open Source:** The software should be freely available and modifiable, allowing for community contributions and transparency.
- **Free:** The software should be free to use, modify, and distribute.

![star-history](/content/blog/top-open-source-kubernetes-dashboard/star-history.webp)

## Kubernetes Dashboard

![k8s-dashboard](/content/blog/top-open-source-kubernetes-dashboard/k8s-dashboard.webp)

Kubernetes Dashboard ([repo](https://github.com/kubernetes/dashboard)) is the official web-based UI for Kubernetes clusters. It allows users to manage applications running in the cluster and troubleshoot them, as well as manage the cluster itself.

**Key Features**

- **Cluster Overview:** Displays an overview of cluster health, workloads, and resources.
- **Workload Management:** Create, update, and delete Kubernetes objects such as Deployments, StatefulSets, and Jobs.
- **Pod & Container Monitoring:** View logs, resource consumption, and running status of individual pods.
- **Service & Networking Control:** Manage Services, Ingress, and Network Policies.
- **Storage Management:** View and manage Persistent Volumes (PVs) and Persistent Volume Claims (PVCs).
- **Role-Based Access Control (RBAC):** Supports authentication and access control using Kubernetes RBAC.

## Headlamp

![headlamp](/content/blog/top-open-source-kubernetes-dashboard/headlamp.webp)

[Headlamp](https://headlamp.dev/) ([repo](https://github.com/headlamp-k8s/headlamp)) an easy-to-use and extensible Kubernetes web UI, which can also be used as a desktop app. It's developed by Kinvolk (now part of Microsoft). It's a Cloud Native Computing Foundation (CNCF) Sandbox project.

Headlamp was created to blend the traditional feature set of other web UIs/dashboards (i.e., to list and view resources) with added functionality.

**Key Features**

- Vendor-independent / generic Kubernetes UI
- Works in-cluster, or locally as a desktop app
- Multi-cluster
- Extensible through plugins
- UI controls reflecting user roles (no deletion/update if not allowed)
- Clean & modern UI
- Cancellable creation/update/deletion operations
- Logs, exec, and resource editor with documentation
- Read-write / interactive (actions based on permissions)

## Lens

![lens](/content/blog/top-open-source-kubernetes-dashboard/lens.webp)

Designed for Developers and DevOps Engineers, [Lens](https://k8slens.dev/) ([repo](https://github.com/lensapp/lens)) provides an intuitive context-aware UI.

<HintBlock type="info">

Lens Desktop was originally an open-source Kubernetes IDE, but the open-source version has been retired and is no longer actively maintained. The Lens Desktop product is now developed and maintained by Mirantis, and contributions to Lens are now made via the Lens Extension API rather than direct modifications to the core product.

</HintBlock>

**Key Features**

- **Multi-Cluster Management:** Easily manage multiple Kubernetes clusters.
- **Real-Time Monitoring:** Built-in Prometheus for live CPU, memory, and network metrics.
- **Integrated Terminal:** Direct kubectl access with auto-complete.
- **Workload & Resource Management:** View and control pods, deployments, and services.
- **Helm Chart Support:** Install and manage Helm applications.
- **RBAC & Security Insights:** Manage access roles and detect security issues.
- **Lens Extensions API:** Customize with plugins and integrations.
- **Desktop App:** Available for Windows, macOS, and Linux.
- **Event Alerts & Troubleshooting:** Get real-time alerts and logs for debugging.

## Rancher

![rancher](/content/blog/top-open-source-kubernetes-dashboard/rancher.webp)

[Rancher](http://rancher.com/) ([repo](https://github.com/rancher/rancher)) is an open source container management platform built for organizations that deploy containers in production. Rancher makes it easy to run Kubernetes everywhere, meet IT requirements, and empower DevOps teams

<HintBlock type="info">

Rancher was developed by Rancher Labs, founded in 2014, to simplify Kubernetes management. In December 2020, it was acquired by SUSE, which continues to develop and support it as an open-source solution for multi-cluster, cloud-native, and edge Kubernetes management.

</HintBlock>

**Key Features**

- **Multi-Cluster Management:** Centrally manage Kubernetes clusters across cloud, on-prem, and edge.
- **Cluster Provisioning:** Deploy Kubernetes clusters on AWS, Azure, Google Cloud, and bare metal.
- **RBAC & Security:** Centralized Role-Based Access Control (RBAC) and security policies.
- **Built-in Monitoring & Logging:** Integrated Prometheus, Grafana, and Fluentd for insights.
- **Application Deployment:** Manage workloads using Helm charts, Rancher Apps, and GitOps.
- **CIS Benchmarking:** Automate security scans and compliance checks.
- **Rancher Fleet:** GitOps-based multi-cluster application management.
- **Edge Kubernetes Support:** Manage lightweight Kubernetes (K3s) for edge computing.

## KubeSphere

![kubeSphere](/content/blog/top-open-source-kubernetes-dashboard/kubesphere.webp)

[KubeSphere](https://github.com/kubesphere/kubesphere) ([repo](https://kubesphere.io/)) is a distributed operating system for cloud-native application management, using Kubernetes as its kernel. It provides a plug-and-play architecture, allowing third-party applications to be seamlessly integrated into its ecosystem. KubeSphere is also a multi-tenant container platform with full-stack automated IT operation and streamlined DevOps workflows.

**Key Features**

- 🧩 **Extensible Architecture** – Plugin-based and customizable.
- 🕸 **Cluster Provisioning** – Deploy on any infrastructure, including air-gapped setups.
- 🔗 **Multi-Cluster Management** – Centralized control for multiple Kubernetes clusters.
- 🤖 **DevOps & GitOps** – Built-in Argo CD, Jenkins, and CI/CD pipelines.
- 🔎 **Observability** – Integrated monitoring, logging, alerts, and auditing.
- 🌐 **Service Mesh** – Istio-based traffic control and microservices visualization.
- 💻 **App Store** – One-click Helm-based app deployment.
- 💡 **Edge Computing** – Manage applications on edge devices with KubeEdge.
- 🗃 **Storage & Networking** – Supports Ceph, NFS, OpenELB, Calico, Flannel, Kube-OVN.
- 🏢 **Multi-Tenancy & RBAC** – Secure workspaces with fine-grained access control.
- 🧠 **GPU Workload Scheduling** – GUI-based GPU management and monitoring.

## K9s

![k9s](/content/blog/top-open-source-kubernetes-dashboard/k9s.webp)

[K9s](https://k9scli.io/) ([repo](https://github.com/derailed/k9s)) provides a terminal UI (TUI) to interact with your Kubernetes clusters. The aim of this project is to make it easier to navigate, observe and manage your applications in the wild. K9s continually watches Kubernetes for changes and offers subsequent commands to interact with your observed resources.

**Key Features**

- **Terminal-Based UI** – Fast and lightweight, runs entirely in the command line.
- **Real-Time Monitoring** – Auto-refreshing view of pods, deployments, and resources.
- **Context-Aware Navigation** – Quickly switch between namespaces and cluster resources.
- **Built-in Log Viewer** – Live pod logs with filtering and search capabilities.
- **Hotkeys & Shortcuts** – Streamlined workflows for quick actions and debugging.
- **YAML Editing & Execution** – Modify Kubernetes resources without leaving the terminal.
- **Resource Metrics** – View CPU, memory, and network usage directly in the UI.
- **RBAC & Security Insights** – Manage user permissions and policies.
- **Plugin Support** – Extend K9s functionality with custom plugins.
- **Works with Any Kubernetes Distribution** – Compatible with K3s, AKS, EKS, GKE, and more.

## Devtron

![devtron](/content/blog/top-open-source-kubernetes-dashboard/devtron.webp)

[Devtron](https://devtron.ai/) ([repo](https://github.com/devtron-labs/devtron)) is an open-source, Kubernetes-native DevOps platform that gives teams a complete web dashboard for building, deploying, observing and securing their cloud-native applications. By unifying CI/CD, GitOps, observability and security in a single interface, Devtron removes the operational toil of stitching together disparate tools and lets developers focus on shipping reliable code.

**Key Features**

- **Web-Based Kubernetes Dashboard** – Visual, single-pane interface that replaces complex `kubectl` commands and surfaces cluster state, workloads and health at a glance.
- **End-to-End CI/CD & GitOps** – Drag-and-drop pipelines with declarative GitOps synchronisation enable automated, auditable releases from commit to production.
- **Multi-Cluster Visibility** – Manage and deploy workloads across multiple clusters and clouds from one console, with aggregated views of nodes, namespaces and services.
- **AI-Driven Troubleshooting** – Built-in AI analyses pod and application errors to suggest fixes and cut mean-time-to-resolution.
- **Progressive Delivery Strategies** – One-click canary, blue-green and rolling updates powered by Flagger and Istio to deploy changes safely.
- **Integrated DevSecOps** – Automated image, code and manifest scanning with policy gates to stop insecure builds before they ship.
- **Helm & Chart Hub** – Browse, customise and deploy Helm charts, with version management and diffing directly in the console.
- **Real-Time Metrics & Logs** – Stream application logs and view CPU, memory and network metrics without leaving the dashboard.
- **Fine-Grained RBAC & SSO** – Integrates with enterprise identity providers and lets admins define role-based permissions at namespace or resource level.
- **Extensible Ecosystem** – Hooks and integrations connect Devtron with popular tools across the SDLC for cost insights, tracing, alerting and more.


Top open-source Kubernetes dashboard solutions include K9s, Lens, and KubeDashboard.

Top Free, Open Source Kubernetes Dashboard in 2026


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database DevSecOps tool. We update the post gradually.

</HintBlock>

This is a series of articles about database schema change / database schema migration:

1. [What is a Database Schema?](/blog/what-is-database-schema)
1. [What is a Database Schema Migration?](/blog/what-is-database-migration)
1. [How to Handle Database Migration / Schema Change?](/blog/how-to-handle-database-schema-change)
1. Top Database Schema Migration Tools in 2026 (this one)

---

It's year ~~2024~2025~2026, database schema migration is still the riskiest area in application development - it's tough, risky, and painful. Database schema migration tools exist to alleviate pain, and have come a long way: from the basic CLI tools to GUI tools, from simple SQL GUI clients to the all-in-one collaboration database platform.

## Command-Line Clients (CLI) - mysql / psql

`mysql` and `psql` are the native CLI for MySQL and PostgreSQL respectively. You can send commands or queries directly to MySQL or PostgreSQL servers from the command line.

![_](/content/blog/top-database-schema-change-tool-evolution/mysql.webp)

Although the CLI interface is simple, sometimes it's intimidating to the beginners.

## GUI

### phpMyAdmin, pgAdmin

phpMyAdmin and pgAdmin are classic SQL clients. phpMyAdmin is already an established tool for managing MySQL and MySQL-compatible databases (e.g. MariaDB), while pgAdmin is the PostgreSQL counterpart, which has now evolved to pgAdmin 4. Compared with CLI tools, phpMyAdmin and pgAdmin provide a UI for running and executing SQL queries, making them user friendlier.

![_](/content/blog/top-database-schema-change-tool-evolution/gui.webp)

### DBeaver

DBeaver's initial release came in 2010, this open-source SQL client supports a whole range of databases (SQL, NoSQL, and cloud DBs), and keeps up with the latest trends in the AI realm by incorporating OpenAI's GPT-capabilities, allowing natural language to be converted into SQL. In 2025, DBeaver has expanded its AI capabilities with more advanced schema optimization suggestions and enhanced query performance analysis.

![_](/content/blog/top-database-schema-change-tool-evolution/dbeaver.webp)

### DataGrip

DataGrip is another commercial SQL client developed by JetBrains. DataGrip has powerful code completion and refactoring tools for efficient database development. The AI Assistant, which became a standard feature in 2025, supports text2sql, schema optimization, code explanation, and automated migration script generation with backward compatibility analysis.

![_](/content/blog/top-database-schema-change-tool-evolution/datagrip.webp)

### Navicat

Navicat was first released back in 2001. It only supported MySQL back then, but later added more databases. Although Navicat's UI looks a bit outdated, it has complete functionality and provides a smooth overall experience when operating databases. In 2025, Navicat introduced a modernized interface with improved cloud integration features.

![_](/content/blog/top-database-schema-change-tool-evolution/navicat.webp)

GUI based SQL client is a good complementary to CLI based sql client. Fundamentally, they work in the same way. You connect to the database, and execute some SQL. Quick and convenient, on the flip side, lack of control, both in terms of database change management and data security. This [post](/blog/stop-using-navicat/) goes into more details using Navicat as an example.

## GitOps / Database-as-Code

To better manage and source control database schema changes, several tools have introduced the code change process into database changes, known as Database-as-Code.

### Liquibase

Liquibase started out in 2006 as an open-source library for tracking, managing and applying database changes. Fun fact: Liquibase was acquired by a company called Datical in 2012 and rebranded to Datical, but then changed their name back to Liquibase in 2020 (wise move, have you seen anyone mention Datical on a forum?).

![_](/content/blog/top-database-schema-change-tool-evolution/liquibase.webp)

Liquibase's main product is a Java-based CLI that allows developer teams to integrate database schema migration into their CI/CD workflows through the CLI. As of 2025, Liquibase has strengthened its enterprise offerings with enhanced security features and improved support for regulated industries requiring detailed change tracking.

### Flyway

Flyway shares similarities with Liquibase in many ways: they are both long-established, both have a customer base, and are open-source projects. Its core product is a CLI and a Java library.

The commercial entity behind Flyway is Redgate (acquired in 2019). It's got 3 editions: community, team and enterprise. You can tell that they have marked a clear boundary between the open-source and commercial offerings: Flyway's website feels more casual overall, while Redgate's way more professional.

![_](/content/blog/top-database-schema-change-tool-evolution/flyway-redgate.webp)

Liquibase and Flyway are neck and neck. The main difference lies in their respective positioning: Liquibase is more enterprise oriented, while Flyway is more developer oriented.

### Sqitch

Sqitch is a purely open-source project with no commercial offerings that's been on the market since 2012. It is pure CLI and does not have a UI.

![_](/content/blog/top-database-schema-change-tool-evolution/sqitch.webp)

Unlike Java-based Liquibase and Flyway, Sqitch is developed using Perl. In addition, Sqitch has its own philosophy on how to manage database schema changes: Liquibase and Flyway both use file naming conventions to order schema migrations (convention over configuration).

![_](/content/blog/top-database-schema-change-tool-evolution/schema-migration.webp)

While Sqitch adopts an explicit approach to allow user to specify the order in the sqitch plan.

![_](/content/blog/top-database-schema-change-tool-evolution/appschema.webp)

### Atlas

![_](/content/blog/top-database-schema-change-tool-evolution/atlas.webp)

Atlas is an open-source tool built by Ariga, and promotes the term "database schema-as-code".

You can tell that they have drawn lots of inspo from HashiCorp and dubbed itself "Terraform for Database Migrations" upon its first appearance on Hacker News. They also invented Atlas HCL based on HCL (HashiCorp Configuration Language).

Atlas uses the modern programming language Go (unlike Liquibase/Flyway, which uses Java), and centers around CLI (similar to Liquibase/Flyway/Sqitch), but it also has a lightweight UI. Atlas Cloud is a commercial offering of the open-source version.

## All in One: GUI + GitOps / Database-as-Code + Collaboration

### Prisma

ORMs like Prisma target the domain of how code interacts with data. Although this is more of a backend topic, Prisma is an ORM with frontend roots. Frontend engineers are probably not the most proficient in SQL. To lower the barrier for managing database schemas, Prisma invents their own DSL to define data models.

![_](/content/blog/top-database-schema-change-tool-evolution/prisma.webp)

DSL is state-based (declarative), which describes the end state of the database schema instead of incremental changes, which is different from Liquibase/Flyway/Sqitch. This way, Prisma is able to provide a more comprehensive view of database management throughout the entire application development cycle.

Prisma ORM is open source and free to use, and their Data Platform offers a cloud-based collaborative platform with some advanced features (it's clear that their ambition goes beyond just an ORM and schema migration tool).

### Bytebase

Bytebase is an open-source database DevSecOps tool, covering database management scenarios from changes, queries, security, to governance with a visual web-based collaboration workspace

**Visual change workflow**

![_](/content/blog/top-database-schema-change-tool-evolution/bytebase-issue.webp)

Bytebase provides a web-based UI where developers and DBAs can collaborate to work on database changes through the same interface.

**Database-as-Code**

To better accommodate the working habits of developers, Bytebase has integrated capabilities into [code repositories such as GitLab and GitHub](/blog/integrate-sql-review-into-github/). With the GitOps workflow enabled, developers can submit database change files to familiar code repositories, and once review is completed and committed to the repo, the deployment will be automatically triggered by Bytebase. No need to switch between multiple tools!

**Team Collaboration & Management**

You can define different [roles](https://docs.bytebase.com/concepts/roles-and-permissions/) for members at two levels: Workspace and Project. You can assign different roles to your team members, so that they have different permissions for different projects; or configure approval workflows for each project, such as specifying specific DBAs or QA responsible for this specific project.

![_](/content/blog/top-database-schema-change-tool-evolution/bytebase-roles.webp)

The difference between Prisma and Bytebase lies in the target audience. Prisma is mainly aimed at front-end/full-stack developers, while Bytebase is more focused on back-end and DBAs. Both products provide collaboration capabilities, with Prisma focusing on the collaboration between developers in a single project, while Bytebase targets the entire engineering organization, i.e. developers and DBAs / Platform Engineering / Ops teams. In 2025, Bytebase has enhanced its collaboration features and expanded its role-based permissions system to provide even more granular control for engineering organizations.

## To Sum Up

| Feature              | Liquibase | Flyway           | Atlas            | Prisma         | Bytebase |
| -------------------- | --------- | ---------------- | ---------------- | -------------- | -------- |
| Open Source          | ✅        | ✅               | ✅               | ✅             | ✅       |
| Commercial Offering  | ✅        | ✅               | ✅               | ✅             | ✅       |
| Web UI               | ❌        | ✅ (lightweight) | ✅ (lightweight) | ✅             | ✅       |
| Rollback Support     | ✅        | Limited          | ✅               | Limited        | ✅       |
| CI/CD Integration    | ✅        | ✅               | ✅               | ✅             | ✅       |
| Team Collaboration   | Limited   | Limited          | Limited          | ✅             | ✅       |
| Role-Based Access    | ❌        | ❌               | Limited          | Limited        | ✅       |
| Cloud-Native Support | Limited   | Limited          | ✅               | ✅ (expanding) | ✅       |

If you are operating the database as an individual, the classic CLI or GUI SQL clients like Navicat will suffice. If you prefer integration with code repos, there are solutions like Liquibase and Flyway. However, for a GUI and project collaboration capabilities similar to Jira or GitLab, your options are Prisma and Bytebase. And Bytebase is the only tool that offers organization-wide management capabilities to ensure data security and governance, in addition to making database changes more efficient and safer.

This wraps our database schema change series. Here at Bytebase, we are trying to deliver a schema change experience as close as to the code change experience, from both the individual perspective and the organization perspective. To give it a try, [start locally with a single command](https://docs.bytebase.com/get-started/deploy-with-docker/).

## Further Readings

- [Top Postgres Extensions](/blog/top-postgres-extension/)
- [Top Postgres GUI Clients](/blog/top-postgres-gui-client/)
- [Top Open Source Postgres Migration Tools](/blog/top-open-source-postgres-migration-tools/)


Database schema migration have come a long way, from CLI to GUI, and all the way to GitOps/Database-as-Code. Let's review the state-of-the-art of database schema change / migration tools in 2026.

Top Database Schema Migration Tools to Avoid Change Outage 2026


Database schema compare tool enables you to identify differences in the object structure within relational databases, and synchronize your specific objects across multiple databases. It is usually used in the following scenarios:

- Merge database changes from your private branch to your team's main branch
- Maintain schema consistency across homogeneous databases
- Build a new database test environment
- Deploy database schema changes to production based on application requirements
- Troubleshoot database problems when the service is down

## mysqldbcompare

[mysqldbcompare](https://github.com/mysql/mysql-utilities/blob/master/scripts/mysqldbcompare.py) is a command-line utility from MySQL Utilities that compares the structure and data of two databases. It identifies differences in tables, views, stored procedures, triggers, and other database objects, and can generate SQL statements to synchronize them. The tool supports comparing databases across different servers or on the same server, with options to control comparison depth, output format, and which server's schema should be considered authoritative.

A typical workflow:

1. Use `mysqldump --no-data --routines --triggers` to create schema-only snapshots
1. Version control these snapshots
1. Utilize `mysqldbcompare` to compare schemas and generate diff migration scripts
1. Organize migration scripts with naming convention (e.g., `V1.0.1__description.sql`)
1. Deploy migration scripts

## MySQL Workbench

[MySQL Workbench](https://dev.mysql.com/doc/workbench/en/) is the official GUI for MySQL development from Oracle. It includes schema comparison and synchronization utilities. It enables you to compare and synchronize schema between models, databases and SQL files. These three types can be the destination, source, or both. The following figure shows the source is model and the destination is database.

![workbench](/content/blog/top-mysql-schema-compare-tools/workbench.webp)

In addition, MySQL Workbench allows you to create a report showing the differences in catalog between the compared objects, as the next figure shows.

![report](/content/blog/top-mysql-schema-compare-tools/catalog-report.webp)

It is available on Windows, Linux and Mac OS X.

## dbForge Schema Compare for MySQL

[dbForge Schema Compare for MySQL](https://www.devart.com/dbforge/mysql/schemacompare/) is dbForge's flagship product for MySQL database development and management. It allows you to compare and synchronize database schema between databases and SQL files. dbForge Schema Compare for MySQL has two distinct advantages over other comparion tools:

- Diff and synchronize MySQL databases quickly and efficiently, even for extra-large ones
- Automate routine tasks via CLI to schedule your comparison and synchronization tasks

![dbforge](/content/blog/top-mysql-schema-compare-tools/dbforge.webp)

dbforge is only available on Windows.

## Liquibase

[Liquibase](https://www.liquibase.com/) is an open source command-line tool, which helps you track, version, and deploy database changes with declarative and imperative approaches. It provides three Diff-based commands to discover the differences between databases and database snaphots: `diff`, `diff-changelog`, and `generate-changelog`. You can refer to the section of [Database inspection commands](https://docs.liquibase.com/commands/home.html#database-inspection-commands) for more details. The biggest advantage of Liquibase comparision feature is allow you to integrate them into various workflows.

![liquibase](/content/blog/top-mysql-schema-compare-tools/liquibase.webp)

## Bytebase

[Bytebase](/) is an open source database DevOps tool, which is GitLab/GitHub for databases management throughout the software development lifecycle (SDLC). It offers a GUI web-based workspace for Developers and DBAs to collaborate safely and efficiently. Bytebase has a powerful comparison and synchronization feature with the following unique capabilities:

- Compare and synchronize schema from one database to multiple databases

![bytebase-source](/content/blog/top-mysql-schema-compare-tools/bytebase-source.webp)

- Allow you to select a schema version from the database change history as the source

![bytebase-target](/content/blog/top-mysql-schema-compare-tools/bytebase-target.webp)

- Provide a user-friendly way to display the differences between databases

![bytebase-diff](/content/blog/top-mysql-schema-compare-tools/bytebase-diff.webp)

- Enable you to complete the synchronization with one-click

![bytebase-issue](/content/blog/top-mysql-schema-compare-tools/bytebase-issue.webp)

Most schema compare features are available in Bytebase's Free plan.

## Summing it up

If you are used to working with command line tools, Liquibase and Flyway are excellent choices. If you prefer the fine-grained control during the synchronization process, there are solutions like MySQL Workbench and dbForge Schema Compare for MySQL. On the other hand, if you need a fully-fledged GUI and a more streamlined experience to compare, review and deploy database schema, Bytebase will be a great fit.

## Further Readings

- [Top MySQL GUI Clients](/blog/top-mysql-gui-client)
- [Top Free, Open Source SQL Clients](/blog/top-open-source-sql-clients)


MySQL schema compare tool is used to diff and synchronize schemas between MySQL databases. This article reviews the top 4 common tools in this category.

Top 4 MySQL Schema Compare Tool to Diff and Sync Database 2026


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database DevSecOps tool that can manage PostgreSQL. We update ~once per year.

</HintBlock>

| Update History | Comment                           |
| -------------- | --------------------------------- |
| 2023/07/18     | Initial version.                  |
| 2025/03/18     | Update for 2025.                  |
| 2025/05/23     | Add VS Code PostgreSQL extension. |
| 2026/01/16     | Update for 2026.                  |

Postgres (or PostgreSQL) is one of the most advanced open-source relational databases on the market (alongside [MySQL](/blog/postgres-vs-mysql/)).

It's not a must to use a GUI for managing your Postgres, it can be beneficial if you aren't a veteran at database management or if you simply prefer a graphical interface for database administration tasks.

Here, we gathered 7 of the most popular Postgres GUI Clients out there.

## The Official: pgAdmin

[pgAdmin](https://www.pgadmin.org/) is the Postgres counterpart to MySQL's phpMyAdmin, it is considered the go-to solution for Postgres users in need of a GUI. Similar to phpMyAdmin, it is open-source. The project was initiated by Dave Page, a Postgres core team member, and is backed by EnterpriseDB, a company that provides commercial Postgres solutions. The latest version is pgAdmin 4, which is a complete rewrite of pgAdmin, built using Python and Javascript/jQuery.

pgAdmin is compatible with Windows, Mac, and Linux. Aside from the standard features you'd expect from a database GUI tool (creating and managing databases, running queries, editing tables, managing users and permissions, etc.), it also has monitoring tools to monitor the status of database operations.

![pgadmin](/content/blog/top-postgres-gui-client/pgadmin.webp)

## Mac only

### TablePlus

[TablePlus](https://tableplus.com/) first greeted the world in 2017 and is the newest addition to the list. The UI is simple and sleek without any redundant modules, making it quick and easy to get started with.

![tableplus-ui](/content/blog/top-postgres-gui-client/tableplus-ui.webp)

It supports a wide range of relational databases and some NoSQL databases. Initially, it was exclusive to macOS (just like Postico), but it is now available on Windows, Linux, and even iOS.

![tableplus](/content/blog/top-postgres-gui-client/tableplus.webp)

Currently, TablePlus offers two plans: a free tier with no trial time (but a limited feature) and a paid subscription model that provides extended features.

TablePlus is not open-source, but their other product is DBngin, which can spin up a local database server (Postgres, MySQL, or Redis) on your Mac. It can then be connected to TablePlus for you to visually manage your local databases.

### Postico

[Postico](https://eggerapps.at/postico2/) is a native Mac app for Postgres databases (and some compatible databases including Amazon Redshift, CockroachDB, Greenplum, and others). It's a Mac-only app, and the UI reflects it. The person behind Postico is the same developer who built [postgres.app](https://www.bytebase.com/blog/free-tools-to-start-local-database-on-mac/), which starts a Postgres server on your Mac.

![postico](/content/blog/top-postgres-gui-client/postico.webp)

The latest version is Postico 2 (the prior and first edition is Postico 1.5, which was first released back in Feb 2015), and the tasks that Postico is competent at include:

- Designing database.
- Importing, entering, and editing data.
- Searching, viewing, and analyzing data.

One thing to note though, Postico is not (yet) capable of tasks such as backup and restore, user and permission management, or database monitoring. Postico is probably not your best option if you have database administration needs. However, if you are a business analyst who uses Mac, this is the tool for you.

## Cross-platform

### DataGrip

[DataGrip](https://www.jetbrains.com/datagrip/) is a powerful, cross-platform IDE by JetBrains. It also comes with a comprehensive AI Assistant features.

![datagrip-for-pg](/content/blog/top-postgres-gui-client/datagrip.webp)

### Navicat

[Navicat](https://navicat.com/) was initially released in 2002 as a simple application exclusive to MySQL on Windows. It has since expanded its compatibility to include macOS and Linux and added support for a series of other mainstream databases. Now they even have a dedicated Navicat for Postgres (which comes at a lower cost compared to the premium version).

![navicat-for-pg](/content/blog/top-postgres-gui-client/navicat-for-pg.webp)

Note: Navicat is not open-source software and does not offer a free version, and it's not news that they have [a piracy issue](/blog/stop-using-navicat), so make sure you visit the official channels.

Other than that, Navicat is a comprehensive database management tool for your database administration needs: the feature list includes data modeling and design, data synchronization between databases, backup and restore, data import/export, and SQL Development (it has a built-in SQL editor for users to write and optimize SQL queries).

![navicat](/content/blog/top-postgres-gui-client/navicat.webp)

### DBeaver

[DBeaver](https://dbeaver.io/) and Navicat are similar in many ways, but their biggest difference is that unlike Navicat, which only provides commercial versions, DBeaver offers both open-source and commercial products. The open-source version fulfills the basic database management and administration needs, while the commercial version expands its capabilities with advanced features such as data modeling, collaboration, and security.

![dbeaver](/content/blog/top-postgres-gui-client/dbeaver.webp)

DBeaver started as a hobby project in 2010 and was open-sourced for use in 2013. In early 2023, they announced their first round of funding - they have gone a long way!

### VS Code

![vs-code](/content/blog/top-postgres-gui-client/vs-code.webp)

Microsoft announced brand-new [PostgreSQL extension](https://techcommunity.microsoft.com/blog/adforpostgresql/announcing-a-new-ide-for-postgresql-in-vs-code-from-microsoft/4414648) for Visual Studio Code (VS Code). It comes with core features like intellisense, schema visualization, database explorer, query history. It also integrates with Copilot and Entra ID to offer a seamless experience.

## Governance built-in: Bytebase

While the aforementioned options provide a user-friendly interface for database operations, there may be instances where your organization requires an additional layer of control over database queries, changes, and administrative actions. In such cases, you can explore [Bytebase](/), our [open-source](https://github.com/bytebase/bytebase) Database DevSecOps platform designed specifically for teams. Bytebase aims to centralize control and safeguard your organization's most valuable asset - database data.

![bytebase-sql-editor](/content/blog/top-postgres-gui-client/bytebase-sql-editor.webp)

## Futher Readings

- [Top Open Source Postgres Migration Tools](/blog/top-open-source-postgres-migration-tools/)
- [Top Postgres Extension](/blog/top-postgres-extension)
- [Postgres vs. MySQL](/blog/postgres-vs-mysql)
- [Postgres vs. MongoDB](/blog/postgres-vs-mongodb)


Postgres GUI clients makes it safer and easier to manage Postgres databases by providing a provide a visual graphical interface. In this post, we are taking a look at the most common Postgres GUI Clients.

Top 4 Free Docker Desktop Alternatives 2026


Backing up PostgreSQL is essential for reliability and disaster recovery. The open-source ecosystem offers several strong tools - each with unique design goals, languages, and ideal use cases.

Below is a curated list of the top PostgreSQL backup solutions, compared across multiple dimensions: language and ideal use cases.

[![star-history](/content/blog/top-open-source-postgres-backup-solution/star-history.webp)](https://www.star-history.com/#databasus/databasus&wal-g/wal-g&pgbackrest/pgbackrest&EnterpriseDB/barman&pgmoneta/pgmoneta&eduardolat/pgbackweb)

## Databasus

- **GitHub:** https://github.com/databasus/databasus
- **Language:** Go + TypeScript
- **Maintainer:** Community
- **License:** Apache 2.0
- **Interface:** Web UI
- **Supported Databases:** PostgreSQL, MySQL, MariaDB, MongoDB
- **Best For:** Teams needing a self-hosted, web-based backup management solution

Databasus (rebranded from Postgresus) is a self-hosted database backup management tool with a modern web UI. It offers flexible scheduling (hourly, daily, weekly, monthly, or cron), multiple storage destinations (S3, Google Drive, NAS, SFTP), AES-256-GCM encryption, and team features like role-based access control and audit logging. Notifications are supported via email, Telegram, Slack, Discord, and webhooks.

![databasus](/content/blog/top-open-source-postgres-backup-solution/databasus.webp)

## WAL-G

- **GitHub:** https://github.com/wal-g/wal-g
- **Language:** Go
- **Maintainer:** Community
- **License:** Apache License 2.0
- **Interface:** CLI
- **Supported Databases:** PostgreSQL, MySQL/MariaDB, MS SQL Server, MongoDB (beta), Redis (beta)
- **Best For:** Cloud-native and multi-database environments

WAL-G is an archival and restoration tool for databases in the Cloud, the spiritual successor to [WAL-E](https://github.com/wal-g/wal-e). It's built for performance, with parallel compression, encryption, and seamless cloud storage integration. Its multi-database support makes it ideal for teams managing mixed database stacks.

## pgBackRest

- **GitHub:** https://github.com/pgbackrest/pgbackrest
- **Language:** C
- **Maintainer:** Crunchy Data
- **License:** MIT License
- **Interface:** CLI
- **Supported Databases:** PostgreSQL only
- **Best For:** Mission-critical, high-performance PostgreSQL workloads

pgBackRest, is designed for speed, reliability, and flexibility. It supports full, differential, and incremental backups with parallel processing, compression, encryption, and seamless cloud storage integration. Trusted by production teams worldwide, it's one of the most capable PostgreSQL backup tools for both on-premises and cloud environments.

## Barman

- **GitHub:** https://github.com/EnterpriseDB/barman
- **Language:** Python
- **Maintainer:** EnterpriseDB (EDB)
- **License:** GNU GPL 3
- **Interface:** CLI
- **Supported Databases:** PostgreSQL only
- **Best For:** Enterprises requiring centralized backup management

Barman (**B**ackup **A**nd **R**ecovery Manager) is an open-source administration tool for disaster recovery of PostgreSQL servers written in Python. It allows your organisation to perform remote backups of multiple servers in business critical environments to reduce risk and help DBAs during the recovery phase.

## pgmoneta

- **GitHub:** https://github.com/pgmoneta/pgmoneta
- **Language:** C
- **Maintainer:** Community
- **License:** BSD-3-Clause
- **Interface:** CLI (daemon mode)
- **Supported Databases:** PostgreSQL only
- **Best For:** Teams needing efficient daemon-based backup with incremental support

pgmoneta is a backup and restore solution for PostgreSQL, named after the Roman Goddess of Memory. It runs as a daemon and supports full and incremental backups (PostgreSQL 14+), multiple compression options (gzip, zstd, lz4, bzip2), AES encryption, and WAL shipping. It features Prometheus monitoring integration, remote management capabilities, and TLS v1.2+ support. Built with a process-based model using shared memory and libev for efficient network operations.

## pgBackWeb

- **GitHub:** https://github.com/eduardolat/pgbackweb
- **Language:** Go + JavaScript
- **Maintainer:** Community
- **License:** AGPL-3.0
- **Interface:** Web UI
- **Supported Databases:** PostgreSQL (via pgBackRest)
- **Best For:** Teams preferring visual management for pgBackRest backups

pgBackWeb provides a user-friendly web dashboard on top of pgBackRest. It enables visual monitoring, restore operations, and scheduling — ideal for smaller teams or users who prefer GUI-based workflows instead of the command line.

![pgbackweb](/content/blog/top-open-source-postgres-backup-solution/pgbackweb.webp)

## Comparison Table

| Tool           | Language        | License      | Interface    | Multi-DB Support | Cloud Storage       |
| -------------- | --------------- | ------------ | ------------ | ---------------- | ------------------- |
| **Databasus**  | Go + TypeScript | Apache 2.0   | Web UI       | ✅               | ✅                  |
| **WAL-G**      | Go              | Apache 2.0   | CLI          | ✅               | ✅                  |
| **pgBackRest** | C               | MIT          | CLI          | ❌               | ✅                  |
| **Barman**     | Python          | GNU GPL 3    | CLI          | ❌               | ✅                  |
| **pgmoneta**   | C               | BSD-3-Clause | CLI (daemon) | ❌               | ❌                  |
| **pgBackWeb**  | Go + JavaScript | AGPL-3.0     | Web UI       | ❌               | ✅ (via pgBackRest) |

## Choosing the Right Tool

- **Databasus** — great for teams wanting a self-hosted web UI with multi-database support.
- **WAL-G** — best for multi-database, cloud-native setups.
- **pgBackRest** — the most complete solution for high-performance and cloud-integrated PostgreSQL.
- **Barman** — fits enterprise environments with strict compliance.
- **pgmoneta** — lightweight daemon-based solution with incremental backup and monitoring support.
- **pgBackWeb** — ideal if you prefer a GUI for pgBackRest.

Each project offers a unique trade-off between simplicity, scalability, and ecosystem support. Pick based on your environment’s scale, cloud strategy, and team expertise.


Top open-source Postgres backup solutions for 2026.

Top 7 Free, Open Source SQL Clients to Make Database Management Easier 2026


Logs are one of the most important, and most misunderstood parts of running **Postgres** in production. When configured well, Postgres logs help you debug incidents, understand performance issues, and gain visibility into database behavior. When configured poorly, they create noise, increase load, and still fail to answer the questions you care about.

This guide explains **what Postgres logs are, how logging works, which settings matter most, and how to use logs effectively in real systems**, without overcomplicating things.

---

## What Postgres logs are

**Postgres logs record what the database server is doing.** They capture server-side events such as:

- SQL statements (optionally)
- Slow queries
- Errors and warnings
- Connection and disconnection events
- Lock waits and deadlocks
- Background activity like checkpoints and autovacuum

They are primarily an **operational tool**. Engineers rely on Postgres logs to answer questions like:

- Why is this query slow?
- Why are connections spiking?
- What caused this error?
- Is the database under unusual load?

Postgres logs are generated by the database server itself and are independent of application logs.

## How Postgres logging works (high level)

By default, Postgres writes log output to `stderr`. When the `logging_collector` is enabled, Postgres captures this output and writes it to log files on disk.

Key points to know:

- Logs are written on the database server, not the client
- Log files can be plain text or structured (`csvlog`)
- Log rotation can be based on time or file size
- Retention is your responsibility — Postgres does not manage it automatically

#### Common log formats

- **Plain text logs**
  Easy to read, harder to parse at scale.

- **CSV logs (`csvlog`)**
  Structured, machine-friendly, and much easier to ingest into log systems.

If you plan to analyze logs beyond manual inspection, `csvlog` is usually the better choice.

## The most important Postgres logging parameters

Postgres exposes many logging options, but only a subset matter for most production systems. It’s best to think about them by **intent**, not alphabetically.

### 🔍 Query performance and slow queries

This is where most teams start — and where most mistakes happen.

#### `log_min_duration_statement`

Logs statements that run longer than a specified duration (in milliseconds).

```conf
log_min_duration_statement = 500
```

This logs queries that take longer than 500 ms and is often the **best default** for production systems.

#### `log_duration`

Logs how long every statement took.

Useful for debugging, but very noisy if left on permanently.

#### `log_statement`

Controls which statements are logged.

Values include `none`, `ddl`, `mod` and `all`.

```conf
log_statement = none
```

Setting `log_statement = all` in production is rarely a good idea — it generates huge volumes of logs and can noticeably impact performance.

### 🔌 Connections and authentication

Connection-related logs help diagnose spikes, leaks, and unexpected traffic.

Key settings are `log_connections`, `log_disconnections` and `log_hostname`.

These are especially useful when:

- Using connection pools
- Investigating sudden increases in load
- Debugging authentication issues

They are typically safe to enable, but still generate noticeable volume on busy systems.

### ⚠️ Errors, locks, and failures

These settings help explain *why* queries are blocked or failing.

- `log_error_verbosity`
  Controls how much detail is logged for errors.

- `log_lock_waits`
  Logs statements that wait longer than `deadlock_timeout` for locks.

- `deadlock_timeout`
  How long Postgres waits before checking for deadlocks.

Together, these settings are invaluable for diagnosing:

- Lock contention
- Deadlocks
- Long-running transactions blocking others

### 🧠 Context: making Postgres logs readable

Logs without context are frustrating. This is where `log_line_prefix` matters.

```conf
log_line_prefix = '%m [%p] %u@%d '
```

This adds:

- timestamp
- process ID
- user
- database name

Without a proper prefix, it becomes very difficult to correlate log lines with:

- applications
- users
- specific databases

A good `log_line_prefix` dramatically improves log usability with minimal overhead.

## Common Postgres logging mistakes

Even experienced teams run into these.

### Logging everything

Turning on full statement logging in production creates massive noise and real performance cost — often without delivering useful insight.

### Enabling too many options at once

Logging connections, disconnections, durations, statements, and background activity simultaneously can overwhelm both storage and humans.

### Missing context

Logs without user, database, or timestamp information are hard to interpret and nearly impossible to analyze at scale.

### No retention strategy

Keeping logs forever increases cost and risk. Keeping them too briefly makes post-incident analysis impossible.

### Collecting logs but never using them

Logs should answer questions. If nobody looks at them, they’re just expensive text files.

## Performance impact of Postgres logging

Logging is not free.

Each logged event adds:

- CPU overhead
- Disk IO
- Additional load on log processing pipelines

High-volume logging can amplify existing performance issues, especially under load.

**Practical rules of thumb:**

- Prefer slow-query logging over full statement logging
- Increase logging temporarily during incidents
- Validate logging changes in staging before production
- Scale logging back down once the issue is resolved

## How teams use Postgres logs in practice

In real systems, Postgres logs are commonly used to:

- Identify slow endpoints by spotting recurring slow queries
- Investigate failed deployments or migrations
- Diagnose connection storms and pool misconfigurations
- Understand background activity like checkpoints or autovacuum
- Correlate application errors with database-side failures

They work best when paired with clear operational workflows and disciplined configuration changes.

## Postgres logging best practices quick summary

- Log slow queries, not everything
- Always configure `log_line_prefix`
- Prefer structured logs (`csvlog`) for analysis
- Rotate and retain logs intentionally
- Treat logging as a tuning tool, not a permanent dump

## Final thoughts

Postgres logs are one of the most powerful tools you have for understanding database behavior — but only if they’re configured with intent. Start minimal, focus on slow queries and errors, and increase verbosity only when you need answers.

Good logging doesn’t mean more logs.
It means **the right logs, at the right time, for the right questions**.


Postgres Logs Explained: A Practical Guide for Production


PostgreSQL uses MultiXact structures to track row-level locks shared by multiple transactions. Each MultiXact contains members pointing to the transactions holding the lock. The problem: MultiXactOffset—the pointer into the member storage—was a 32-bit value, capping total members at 2^32 (~4 billion) before wraparound. Unlike [transaction ID wraparound](/blog/postgres-transaction-id-wraparound), which is well-documented and widely monitored, MultiXact member exhaustion catches most teams off guard.

This isn't just theoretical. In May 2025, [Metronome experienced four separate outages](https://metronome.com/blog/root-cause-analysis-postgresql-multixact-member-exhaustion-incidents-may-2025) when their PostgreSQL cluster exhausted MultiXact member space during a large data migration. The insidious part: their monitoring showed MultiXact ID usage under 50%, but PostgreSQL enforces a separate, poorly-documented member space limit. When it hit, all writes failed with "This command would create a multixact" errors, and recovery required hours of emergency vacuuming across their 30TB cluster.

Postgres 19 eliminates this limit entirely with 64-bit MultiXactOffset, introduced by Heikki Linnakangas in commit [bd8d9c9b](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=bd8d9c9bdfa0c2168bb37edca6fa88168cacbbaa), which widened MultiXactOffset to 64 bits.

## What Changes

The 64-bit expansion has two practical effects:

**No more wraparound emergencies.** The 32-bit limit meant high-concurrency workloads—especially those with foreign keys where multiple transactions lock the same rows—could exhaust member space unexpectedly. When `n` transactions share-lock a row, PostgreSQL creates O(n) MultiXacts with O(n²) members. The new 64-bit offset eliminates this ceiling entirely.

**Simplified operations.** PostgreSQL previously required aggressive anti-wraparound vacuuming specifically for member space. This is now removed. You still need regular vacuuming for disk space management, but the emergency "vacuum or die" scenario for member exhaustion is gone.

The change requires a pg_upgrade rewrite of `pg_multixact` SLRU files when upgrading to Postgres 19, but this happens automatically during the upgrade process.

## Closing Thoughts

The MultiXact member limit was a hidden operational risk—poorly documented, invisible to standard monitoring, and catastrophic when hit. Postgres 19 eliminates it entirely rather than just raising the threshold. While the timeline for 64-bit transaction IDs remains unclear, widening MultiXactOffset to 64 bits is a concrete step toward eliminating all wraparound-related risks in PostgreSQL.

## References

- [Metronome: Root Cause Analysis of PostgreSQL MultiXact Member Exhaustion](https://metronome.com/blog/root-cause-analysis-postgresql-multixact-member-exhaustion-incidents-may-2025)
- [Postgres commit bd8d9c9b: Widen MultiXactOffset to 64 bits](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=bd8d9c9bdfa0c2168bb37edca6fa88168cacbbaa)


Postgres 19 widens MultiXactOffset to 64 bits, eliminating the hidden 4 billion member limit that caused catastrophic outages for high-concurrency workloads.

Postgres 19 Feature Preview: 64-bit MultiXactOffset


Schema changes are a normal part of software development. Columns get added, data types change, indexes appear or disappear. But in MySQL, even a simple `ALTER TABLE` can block queries, slow down replicas, or cause downtime if not planned well.

This guide explains a practical way to manage MySQL schema migrations - simple to follow and focused on the MySQL behaviors that matter in real production environments.


## Core Best Practices

These ideas apply broadly to database migrations. They form a reliable foundation before considering MySQL-specific behavior.

### Keep schema changes traceable

Teams need a clear record of how the database evolves.
Whether you use Git, a migration tool, or a structured changelog, the key is:

> **Every schema change should be recorded, reviewable, and reproducible.**

This prevents hidden production changes and environment drift.

### Use small, incremental, backward-compatible steps

A safer approach is the known pattern:

1. **Expand** - add new columns or tables without removing anything
2. **Migrate** - backfill data and update application logic
3. **Contract** - remove old fields only after everything is stable

The contract step usually happens days later because dropping a column is irreversible. Teams wait until all parts of the system clearly use the new structure and real traffic shows no issues.

### Automate checks and workflows

Automation reduces mistakes and increases consistency:

- **SQL review rules:** catch unsafe SQL before execution.
- **CI checks:** run automated tests and static checks on migration scripts.
- **Staging verification:** confirm the migration behaves correctly with realistic data.
- **Consistent environment promotion:** apply migrations in the same order across Dev -> Test -> Prod.

These steps create a predictable path from development to production.

## Key MySQL Behaviors to Know

MySQL has several characteristics that directly affect schema migrations. Understanding these early avoids surprises.

- **Some schema changes rebuild the whole table** - Certain `ALTER TABLE` operations create a new copy of the table, which can slow things down or block writes.

- **Metadata locks can block queries** - A schema change must wait for ongoing queries to finish. While waiting, it can block new queries behind it.

- **Large changes can cause replica lag** - Big ALTERs or heavy backfills often make replicas fall behind, affecting reads and failover.

- **Some MySQL features are harder to modify** - Examples include ENUM changes, JSON indexes, and FOREIGN KEYS on large tables.

## A Practical Migration Workflow for MySQL

This workflow reflects what developers actually do, with MySQL’s behavior in mind.

### 1 Plan the change carefully

Before writing SQL, understand the impact:

- Will the operation **rebuild** the table?
- How **large** is the table in production?
- Are there **long-running queries** that could block the migration?
- How will **replicas** respond?
- Can the work be **split** into smaller, safer steps?

A few minutes of planning prevents many issues later.

### 2 Write simple and predictable migration scripts

Good migration scripts are easy to understand and review:

- One logical change at a time
- Clear and descriptive naming
- Avoid mixing schema and heavy data updates
- Break changes into steps when needed (for example, add column -> backfill -> update code)

Older MySQL versions often rebuild tables when adding a column with a default value, so separating operations reduces risk.

### 3 Test the migration with real impact in mind

Testing should consider both correctness and performance:

- Run first in Dev, then in Staging
- Use staging data that matches production size
- Measure how long the migration takes
- Observe whether replicas fall behind
- Test application behavior on the changed tables

A change that seems instant on a local machine may take minutes - or more - on real data.

### 4 Deploy with caution and observability

During deployment:

- Promote the migration across environments in order
- Check for blocking queries before running DDL
- Monitor:
  - metadata lock waits
  - slow queries
  - replica lag

Because MySQL DDL is not transactional, rolling back a schema change is usually not possible. To reduce risk:

- Use roll-forward migrations to fix issues quickly
- Keep reliable backups and test restore procedures
- Make data migrations reversible when possible

A careful rollout with a clear recovery plan makes deployments much safer.

## Tooling and Zero-Downtime Options

Some tools exist specifically to work around MySQL’s migration challenges.

### [pt-online-schema-change](https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html)  

Creates a shadow table and copies data gradually, reducing locking. Useful for large and busy tables.

### [gh-ost](https://github.com/github/gh-ost)

Uses binary logs instead of triggers and works well on high-write workloads.
Often safer for large-scale online schema changes.

### [Native MySQL Online DDL](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl.html)

MySQL 8.0 supports more fast operations, but testing is still required to avoid unexpected behavior.

### Migration frameworks and workflow tools

These tools help manage versioning, ordering, and execution of schema changes:

- [Flyway](https://flywaydb.org/)
  Tracks versions and applies schema changes in order.

- [Liquibase](https://www.liquibase.org/)
  Uses declarative change sets and supports rollback logic and more structured migration workflows.

- [Bytebase](/)
  Provides a workflow for schema changes, SQL review, and environment promotion, supporting both GUI-based changelog and GitOps modes.
  Also integrates with gh-ost for safer online schema changes on large tables.

## Conclusion

Safe MySQL schema migration comes from understanding how MySQL behaves and following a clear, consistent workflow. With small changes, proper testing, and basic awareness of locking and replication, teams can update their schemas with far less risk and uncertainty.

A guide to MySQL schema migration best practice.

MySQL Schema Migration Best Practice


Logical replication in Postgres has a long-standing limitation: while table data replicates perfectly, the sequences backing `SERIAL` and `IDENTITY` columns stay at their initial values on the subscriber. This creates a problem during upgrades or failover—promote a subscriber to primary, and the next `INSERT` tries to generate an ID that already exists, triggering a primary key violation.

Postgres 19 solves this with automatic sequence synchronization, introduced across three commits by Amit Kapila:

1. [96b37849](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=96b37849734673e7c82fb86c4f0a46a28f500ac8) added publication support for sequences

1. [f0b3573c](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=f0b3573c3a) introduced the `REFRESH SEQUENCES` command

1. [5509055d](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=5509055d6956745532e65ab218e15b99d87d66ce) implemented the synchronization worker.

## Usage

Postgres 19 adds `FOR ALL SEQUENCES` to publications:

```sql
-- Publisher
CREATE PUBLICATION upgrade_pub FOR ALL TABLES, ALL SEQUENCES;

-- Subscriber
CREATE SUBSCRIPTION upgrade_sub
    CONNECTION 'host=publisher-db dbname=production'
    PUBLICATION upgrade_pub;
```

A new sequencesync worker automatically synchronizes sequences in batches. Sequences track their state (INIT → READY) in `pg_subscription_rel`.

```sql
SELECT srsubid, srrelid::regclass, srsubstate
FROM pg_subscription_rel
WHERE srsubstate IN ('i', 'r');  -- 'i' = INIT, 'r' = READY
```

Sequences only sync when you explicitly run `REFRESH SEQUENCES`. Build `REFRESH SEQUENCES` into your cutover runbook.

```sql
ALTER SUBSCRIPTION upgrade_sub REFRESH SEQUENCES;
```

## Closing Thoughts

Postgres is systematically closing logical replication gaps. Postgres 18 added generated column support. Now Postgres 19 adds sequence synchronization, the largest hurdle in zero-downtime upgrade and failover workflows.

The remaining major limitation is DDL replication—schema migrations still require manual coordination. But for upgrade and failover scenarios, this is less critical than sequence synchronization. When promoting a subscriber, you typically want identical schemas already in place. Missing sequence values cause immediate production failures; managing DDL changes is standard operational procedure. Let's save this for the future release.

## References

- [Postgres logical replication restrictions](https://www.postgresql.org/docs/18/logical-replication-restrictions.html)


Postgres 19 introduces automatic sequence synchronization for logical replication, eliminating manual tracking and making major version upgrades and failover significantly easier.

Postgres 19 Feature Preview: Sequence Synchronization for Logical Replication


Most development teams work with several environments — Dev->Test->UAT->Staging->Prod, some also manage multiple tenants in production. Application code moves through this workflow fairly smoothly. Databases are a different story. They hold state, evolve over time, and can’t simply be redeployed or rolled back. As a result, promoting database changes across multiple environments often turns into a recurring source of friction.

## What does Database Multi-Environment Deployment Mean?

For databases, **multi-environment deployment** means taking the same change and rolling it out in a controlled way across:

- the environment pipeline (Dev → Test → UAT → Staging → Prod), and

- all relevant production databases (tenants, regions, clusters).

A typical flow looks like this:

1. Apply the change in Dev.

1. If it looks good, promote to Test/UAT/Staging.

1. Finally, promote to Prod.

1. In Prod, repeat the same change for each tenant/region that uses its own database.

Every step has to respect existing data, constraints, and workload. That mix of multiple environments plus multiple Prod instances is what makes database deployment very different from shipping a stateless service.

## Challenges of Database Change Deployment

In practice, database deployments hit the same problems again and again:

- **Environment drift**
  Someone runs a quick fix in Prod, or a migration is skipped in Test. Schemas no longer match.

- **Unclear migration state**
  Nobody is 100% sure which changes have run in which environment or tenant.

- **Out-of-order and conflicting changes**
  Feature branches introduce overlapping migrations; a hotfix lands in Prod before it exists in Dev.

- **Prod behaves differently**
  A trivial change in Dev can lock big tables or blow up queries in Prod because of larger data volume.

- **Different deployment paths**
  Not every service uses the same Dev→Test→Stage path, and some have extra staging steps.

- **Rollback is hard**
  Rolling back schema and data safely is much harder than rolling back a container image.

These are the reasons "just run this SQL everywhere" stops working beyond a certain scale.

## Patterns for Handling Multi-Environment Database Changes

### 1. Manual copy/paste scripts

The simplest pattern looks like this:

1. A developer writes SQL and runs it in Dev.

1. When it’s ready, they copy the same script to Test, UAT, Staging, Prod (and each tenant) — either running it themselves or raising a ticket so an admin/DBA can run it.

Even with admin approval in the loop, the failure modes are obvious:

- scripts get edited between environments

- someone forgets one step or one tenant

- too many tickets to approve, DBAs are overwhelmed

- different environments or tenants quietly end up with slightly different schemas

It’s fine for a small team and a handful of databases. It doesn’t age well.

### 2. GitOps: single source of truth

Another common pattern is to manage database changes the same way as application code:

- **Create once** – write a migration script for each change.
- **Keep it in Git** – store migrations in a repo (either the app repo or a dedicated DB repo).
- **Review it** – use pull requests for SQL review and approval.
- **Promote it** – automation applies the same migration, in order, to Dev, Test, UAT, Staging, Prod, and then any tenant databases.

In a typical GitOps setup, branches map to environments: a `dev` branch deploys to Dev, `test` to Test, `staging` to Staging, and `main` (or `prod`) to Production. Merging from one branch to the next triggers the corresponding deployment pipeline.

Teams tend to choose this model when they:

- want Git as the single source of truth
- prefer PR-based review
- already have CI/CD pipelines in place
- need a clear versioned change history

Manual copy/paste plus tickets, and GitOps with pipelines, are just different patterns. Which one works better depends on team size, existing tooling, and how much process you actually want.

## Process Enhancements

Once a team has picked a basic deployment pattern—manual scripts with tickets, GitOps with pipelines, or a mix—there are a few simple add-ons that make life much easier. These don’t replace your process; they sit beside it.

### 1. Review process (SQL review and approval)

Before a change reaches Stage or Prod, it’s worth running it through:

- automated checks (linting, basic safety rules)
- a human approval step for higher-risk changes

### 2. Diff tools (schema sync)

Instead of writing every change from scratch, you can let a diff tool compare two schemas and generate the SQL:

- "Make Staging look like Test"
- "Bring this tenant up to the reference schema"

### 3. Drift detection

Drift is when the actual database no longer matches what you think it should be:

- a hotfix was applied directly in Prod
- a tenant missed a deployment
- a migration failed halfway

### 4. Automated rollback

Because databases are stateful, rollback usually isn't a clean "revert" button. Automating rollback means:

- generating forward-fix migrations automatically when a deployment fails
- triggering restore from pre-deployment backups or restore points
- executing reversible down-migrations where they exist

The key enhancement is moving from "someone manually fixes it" to "the system handles common rollback scenarios automatically."

### 5. Change history

When something breaks, you need to know **what changed, where, and when**.

Common sources:

- Git history for migration files
- Change history with change tickets
- audit logs

It doesn't matter whether the trigger was "merge PR" or "DBA clicked deploy" — having a single place to look back is what saves time. This is essential for debugging and for compliance.

## Closing Thoughts

Multi-environment database deployment is hard not because the concepts are complicated, but because the system is stateful, long-lived, and spread across many environments and tenants.

The patterns that work share a few traits:

- there's a single source of truth for changes

- sql linting and review process are required

- ad-hoc SQL is minimized

- diff, review, and drift detection keep schemas aligned

- there's a clear history and automated rollback in place

Once those basics are in place, database deployment starts to feel much closer to application deployment—quiet, repeatable, and mostly boring, which is exactly what you want.

<HintBlock type="info">
Bytebase is built to handle these workflows, supporting both GUI-based and GitOps patterns with all the process enhancements described above.
</HintBlock>


A guide to multi-environment deployments in databases.

Database Multi-Environment Deployments - The Challenges and Patterns


## What is Enterprise Database Management?

Enterprise Database Management (EDM) is the practice of organizing, securing, and optimizing database systems across large organizations. It encompasses the tools, processes, and strategies needed to manage multiple database platforms while ensuring data integrity, security, regulatory compliance, and business continuity. Unlike basic database administration, EDM coordinates cross-functional teams—developers, DBAs, security, and platform engineers—to handle complex database ecosystems at scale:

- **Multiple databases**: PostgreSQL, MySQL, Oracle, SQL Server, MongoDB—all at once
- **Cross-functional teams**: Developers, DBAs, security, platform teams—all need access
- **Regulatory compliance**: GDPR, HIPAA, PCI-DSS, SOC2—all enforced
- **Scale**: Petabytes of data, global distribution
- **Business continuity**: Zero downtime, instant recovery

## Brief History of Enterprise Database Management

### Standalone Databases (1970s-1990s)

Early databases were command-line only. IBM created IMS (1966) for NASA, then System R (1974-1977) introduced SQL. Oracle commercialized SQL in 1979. DBAs typed commands, everyone else waited. This couldn't scale.

### Rise of Database Management Tools (1990s-2010s)

Databases alone weren't enough. Oracle launched [Oracle Enterprise Manager (OEM)](https://www.oracle.com/enterprise-manager/) in 1996, Microsoft followed with [SQL Server Management Studio (SSMS)](https://learn.microsoft.com/en-us/ssms/). Both dominant vendors creating management tools wasn't coincidence—enterprises demanded better control. These tools replaced command lines with visual interfaces for monitoring, backups, and optimization.

### Cloud Era (2010s-2022)

Cloud providers changed the game. AWS RDS, Azure SQL Database, Google Cloud SQL—they offered simplified versions of what OEM and SSMS provided. One-click backups, automatic failovers, built-in monitoring. No more infrastructure management. DBAs could focus on data, not servers. But these cloud consoles still operated in silos, managing databases in isolation from development pipelines.

### The AI Shift (2023-Present)

Large language models transformed expectations. Natural language to SQL, AI-assisted schema reviews, automated migration generation—development workflows finally got intelligent assistance. On the operations side: predictive scaling, anomaly detection, self-healing databases. AI also tackled governance—sensitive data discovery, automated compliance checks, intelligent access recommendations.

The breakthrough: AI bridges the gap between operations and development. Database changes that took days now happen in hours, with AI handling the translation between what developers want and what databases need.

## The Gap Between Operations and Development Workflows

These tools handled operations well but ignored development needs. No version control integration. No CI/CD pipelines. Security teams managed access separately. Platform teams couldn't automate provisioning.

Cloud made this problem more obvious. Applications deployed in minutes, databases still took days. Developers had CI/CD for code, but database changes remained manual tickets. Cloud promised agility—databases became the bottleneck.

The core problem: **these tools focused on operational tasks, not collaborative workflows**. Modern enterprises need all teams working together—traditional tools never delivered that.

The AI shift adds new complexity: validating AI-generated changes, protecting sensitive data from model exposure, and deciding when to trust automation versus human judgment.

In our next article, we'll explore the evolution of enterprise database management in detail—tracing its journey from mainframe-era centralized systems through client-server architectures to cloud-native platforms and the AI shift, and examining how these changes are reshaping database management for modern development teams.


Enterprise database management is a comprehensive approach to organizing, securing, and optimizing database systems and workflows at scale, ensuring data integrity, availability, and compliance across large organizations.

What is Enterprise Database Management (EDM)?


## Introduction

Sensitive data discovery is the first step in protecting PII, PHI, and other regulated information. Before you can mask, encrypt, or restrict access to sensitive data, you need to find it.

Open source tools for this task span a spectrum:

- **NLP libraries** - Building blocks for custom detection pipelines
- **Lightweight CLI scanners** - Quick, targeted scans for developers and CI pipelines
- **Full data platforms** - Comprehensive metadata management with classification as one feature

This article covers four tools across this spectrum, from NLP foundations to enterprise data catalogs with built-in classification.

## spaCy

[spaCy](https://github.com/explosion/spaCy) is the industrial-strength NLP library that powers many sensitive data detection tools.

![spaCy](/content/blog/top-open-source-sensitive-data-discovery-tools/spacy.webp)

spaCy provides named entity recognition (NER) that can identify persons, organizations, locations, and other entity types in text. Both PiiCatcher and OpenMetadata use spaCy under the hood for ML-based PII detection. If you need maximum flexibility, you can build custom detection pipelines directly with spaCy, though the tools below provide ready-to-use solutions.

## PiiCatcher

[PiiCatcher](https://github.com/tokern/piicatcher) is a focused CLI scanner that detects PII in databases and tags findings in data catalogs.

![piicatcher](/content/blog/top-open-source-sensitive-data-discovery-tools/piicatcher.webp)

**Detection approach:** PiiCatcher uses two methods - regex pattern matching against column names, and NLP-based analysis of sample data using spaCy. This dual approach catches both obviously-named columns (e.g., `email`, `ssn`) and columns with generic names but sensitive content.

**Data source support:** PostgreSQL, MySQL, SQLite, Redshift, Athena, Snowflake, BigQuery.

**Key strength:** Native integration with data catalogs. PiiCatcher can automatically tag discovered PII in DataHub or Amundsen, bridging the gap between standalone scanning and catalog-based governance.

**Best for:** Teams wanting a lightweight scanner that feeds into their existing data catalog.

## Hawk-Eye

[Hawk-Eye](https://github.com/rohitcoder/hawk-eye) is a broad-spectrum scanner covering databases, cloud storage, and files - including images and videos via OCR.

![hawk-eye](/content/blog/top-open-source-sensitive-data-discovery-tools/hawk-eye.webp)

**Detection approach:** Pattern matching with configurable fingerprints defined in YAML. Supports OCR for images and documents (350+ file types including DOCX, PDF, images, videos).

**Data source support:** MySQL, PostgreSQL, MongoDB, CouchDB, Redis, S3, Google Cloud Storage, Firebase, Slack, Google Drive, local filesystem.

**Key strength:** Breadth of coverage. Unlike database-only scanners, Hawk-Eye finds PII across your entire data footprint - useful when sensitive data leaks into unstructured storage.

**Best for:** Security teams auditing diverse data sources beyond just databases.

## OpenMetadata

[OpenMetadata](https://github.com/open-metadata/OpenMetadata) is a unified metadata platform with auto-classification as a core governance feature.

![openmetadata](/content/blog/top-open-source-sensitive-data-discovery-tools/open-metadata.webp)

**Detection approach:** Auto-Classification workflow powered by spaCy with configurable confidence levels (0-100). The system identifies PII and either auto-applies tags or suggests them for review. Runs as a separate workflow from metadata ingestion, so you can tune classification independently.

**Data source support:** 84+ connectors spanning databases, dashboards, messaging, and pipelines.

**Key strength:** Tight integration between classification and governance workflows. Tags flow into data quality rules, access policies, and team collaboration features. The no-code profiler makes classification accessible to non-engineers.

**Best for:** Teams wanting a modern, API-first platform where classification drives downstream governance policies.

**Alternative:** [DataHub](https://github.com/datahub-project/datahub) is another open-source metadata platform, but its auto-classification feature only supports Snowflake and has been [marked as deprecated](https://docs.datahub.com/docs/metadata-ingestion/docs/dev_guides/classification). If you're using DataHub, consider pairing it with PiiCatcher for broader classification coverage.

## Comparison

| Tool             | Language      | Primary Use Case                         | Detection Method                                            | Data Source Support                                              | Deployment         | License                   |
| ---------------- | ------------- | ---------------------------------------- | ----------------------------------------------------------- | ---------------------------------------------------------------- | ------------------ | ------------------------- |
| **spaCy**        | Python        | NLP library / building block             | Named entity recognition (NER), ML models                   | N/A (text processing only)                                       | pip                | MIT                       |
| **PiiCatcher**   | Python        | CLI scanner for databases                | Regex + NLP (spaCy)                                         | PostgreSQL, MySQL, SQLite, Redshift, Athena, Snowflake, BigQuery | pip, Docker        | Apache 2.0                |
| **Hawk-Eye**     | Python        | Multi-source scanner (DBs, cloud, files) | Pattern matching + OCR                                      | MySQL, PostgreSQL, MongoDB, Redis, S3, GCS, Firebase, Slack      | pip, Docker        | LGPL 2.1 + Commons Clause |
| **OpenMetadata** | Java / Python | Data platform with governance            | Auto-classification workflow (spaCy), confidence thresholds | 84+ connectors                                                   | Docker, Kubernetes | Apache 2.0                |

![star-history](/content/blog/top-open-source-sensitive-data-discovery-tools/star-history.webp)

Choose based on your needs:

- **spaCy** - Build custom detection pipelines with maximum flexibility
- **PiiCatcher** - Catalog-integrated database scanning
- **Hawk-Eye** - Broad coverage across databases, cloud storage, and files
- **OpenMetadata** - Classification within a full metadata platform

Start lightweight, then graduate to full platforms as governance requirements grow.

## From Discovery to Protection

Finding sensitive data is only half the challenge - you then need to protect it. [Bytebase provides dynamic data masking](https://docs.bytebase.com/security/data-masking/overview) that can be driven by classification results.

Once you've identified PII columns using tools above, you can call the Bytebase REST/gRPC API to apply masking policies programmatically. This creates an automated pipeline: scan → classify → mask, ensuring discovered sensitive data is protected without manual intervention.


A guide to open source sensitive data discovery tools - from lightweight CLI scanners like PiiCatcher and Hawk-Eye to full data platforms like OpenMetadata.

Top Open Source Sensitive Data Discovery Tools in 2025


Database audit logging is now a core security expectation, with standards like SOC 2, ISO 27001, GDPR, HIPAA, and PCI DSS requiring a complete record of **who accessed what, when, and from where**.

Yet building a consistent audit trail across different database engines is still challenging. This article explains why, what "good" looks like, and how to design a reliable auditing strategy.

## Why Audit Logging Matters

Audit logging provides the answers to the most critical operational and security questions:

- **Who** accessed the data?
- **What** did they query or modify?
- **When** did it occur?
- **From where** did the access originate?

This information is essential for:

- Detecting unauthorized access
- Investigating security incidents
- Meeting compliance requirements
- Understanding schema and data evolution
- Establishing accountability across engineering teams

Without reliable audit logs, organizations lack visibility at the exact moment it matters most.

## The Real-World Pain Today (Across All Major Databases)

All major relational databases — MySQL, PostgreSQL, SQL Server, Oracle and cloud-managed variants like AWS RDS, Google Cloud SQL, and Azure Database — provide audit capabilities. However, *how* they provide these capabilities varies dramatically, and implementing them correctly requires deep expertise.

Here are common issues teams encounter:

### MySQL (Community Edition) — Example

MySQL CE’s general and slow logs are **all-or-nothing and extremely noisy**.
Selective auditing (especially for non-root users) requires additional plugins that introduce configuration complexity and variability across environments.

### PostgreSQL — Example

PostgreSQL relies on extensions such as `pgaudit` for structured auditing.
While powerful, these extensions require **careful tuning** to avoid overwhelming log volume while still capturing all critical operations — including SELECTs.

### Cloud Databases (AWS RDS, Google Cloud SQL, Azure Database) — Example

Cloud platforms wrap underlying engine audit logs into provider-specific formats.
Teams often struggle with:

- inconsistent event types
- missing or partial SQL text
- difficulty correlating logs across mixed engines or environments

**In short:**

> Audit information exists everywhere — but it’s fragmented, inconsistent, and often incomplete.

## What a Good Audit Log Should Capture

A reliable audit log must capture **every database action**, not just modifications.
In modern security models, **access is just as important — and often more important — than change**.

### A robust audit log includes:

- **Real human identity**
  No shared admin or application accounts. Every query must map to an actual person.

- **Full query text**, including:

  - **DDL** (all schema changes)
  - **DML** (INSERT, UPDATE, DELETE)
  - **SELECT** (all read operations — because viewing sensitive data is a high-risk event)

- **Authentication events**
  Both successful logins and failed login attempts.

- **Permission changes**
  The audit log must record any permissions granted or revoked for specific users.

- **Execution outcome**
  Whether the operation succeeded, failed, or was rejected.

- **Optional contextual metadata**
  Such as ticket/issue ID, environment, deployment reference, or any policy configurations or changes.

A complete record of SELECT queries ensures you always know *exactly who viewed what data*, which is a mandatory capability under many security and privacy frameworks.

## Approaches to Audit Logging

Teams typically rely on one or more of the following auditing methods:

### 1. Engine-native auditing

Each database engine includes its own audit features.

**Pros:**

- High fidelity
- Deeply integrated with database internals

**Cons:**

- Different for every engine
- Easily becomes noisy without tuning
- Harder to unify across environments

### 2. Cloud provider audit logs

Cloud platforms provide audit streams for their managed databases.

**Pros:**

- Easy to enable
- Centralized in cloud logging services
- Integrated with monitoring tools

**Cons:**

- Inconsistent formats and event coverage
- SQL text may be missing
- Hard to correlate across multi-cloud or multi-engine stacks

### 3. Proxy / workflow-based auditing

SQL is routed through a centralized gateway or workflow before executing.

**Pros:**

- Unified audit trail across all engines
- Automatically tied to real human identity
- Can embed metadata (ticket ID, environment)
- Ensures DDL, DML, *and SELECT* are always captured

**Cons:**

- Requires routing SQL through a central component

*For example:*
A workflow platform like **Bytebase** produces complete, contextual audit logs because all SQL flows through a single, identity-aware pipeline.

## Recommended Best Practices

Regardless of database engine or auditing method, strong audit practices share the same foundations:

- **Use individual identities** — never share DB accounts.
- **Record all DDL, DML, and SELECT** — access visibility is non-negotiable.
- **Store logs off-host** — prevents tampering or accidental deletion.
- **Apply retention policies** (90, 180, or 365+ days).
- **Integrate logs into a SIEM** for alerting and correlation (Datadog, Splunk, CloudWatch, Grafana).
- **Treat default engine settings cautiously** — they often require substantial tuning.

A minimal-noise, high-fidelity audit log is better than a noisy one that nobody can use.


Database Audit Logging - The Practical Guide


[HITRUST CSF](https://hitrustalliance.net/) (Common Security Framework) is a widely adopted security framework in healthcare and other regulated industries. For organizations already familiar with HIPAA, the relationship is straightforward:

- HIPAA defines **what** must be protected — electronic Protected Health Information (ePHI) — but leaves the implementation details vague.
- HITRUST provides the **how** — prescriptive, certifiable controls that satisfy HIPAA requirements and give organizations clear implementation guidance.

Since databases are the primary storage for sensitive health data, database operations are a critical area for HITRUST compliance. This post covers the key database controls, associated risks, and how to implement them effectively.

## Access Control

Only authorized users should access sensitive data, with verifiable identity and appropriate privileges. This is foundational to HITRUST compliance.

**Key controls:**

- **Role-Based Access Control (RBAC)** — Assign permissions based on job function rather than individual users. Clinical staff access patient records; billing staff access financial data; DBAs manage infrastructure.
- **Least Privilege** — Grant the minimum access needed for each task. A developer debugging an issue doesn't need write access to production.
- **Just-in-Time Access** — Provide temporary elevated access that automatically expires. This reduces the window of exposure for privileged operations.
- **Authentication** — Verify user identity through MFA, SSO, and LDAP/AD integration. Shared credentials make accountability impossible.

**Risks without these controls:**

- Unauthorized access to ePHI
- Over-privileged accounts creating unnecessary exposure
- No accountability when incidents occur

## Audit Logging

All access and changes to sensitive data must be recorded, centralized, and analyzable. Audit trails are essential for compliance investigations, certification assessments, and real-time threat detection.

**Key controls:**

- **Query logging** — Track who accessed what data and when. This provides the evidence trail auditors require.
- **Change tracking** — Record all DDL and DML operations with before/after states. Know exactly what changed and who made the change.
- **Log retention** — Maintain audit trails for the required retention period (6+ years for HIPAA-related compliance).
- **SIEM integration** — Stream database audit logs to your SIEM (Security Information and Event Management) system. Correlate database activity with other security events for unified threat detection and incident response.

**Risks without these controls:**

- Unable to investigate breaches or demonstrate compliance
- No accountability for data access or modifications
- Database activity siloed from broader security monitoring
- Gaps in audit trails causing certification failures

## Change Management

Database changes must be controlled, reviewed, and traceable. Uncontrolled changes are a common source of both security incidents and compliance failures.

**Key controls:**

- **Approval workflows** — Require review and approval before changes reach production. No direct production access without oversight.
- **Risk-based classification** — Apply different approval paths based on change risk. A column rename needs less scrutiny than dropping a table.
- **Rollback capability** — Maintain the ability to revert changes when issues arise. This reduces the blast radius of mistakes.
- **Change history** — Keep a complete record of what changed, who approved it, and why. This is critical for audit evidence.

**Risks without these controls:**

- Unreviewed changes introducing vulnerabilities or breaking compliance
- No ability to trace when or how data was altered
- Emergency changes bypassing security controls with no oversight

## Data Protection

Sensitive data must be protected from unauthorized disclosure, both at rest and during access.

**Key controls:**

- **Data masking** — Hide sensitive fields like SSN, diagnosis codes, or payment information from users who don't need full access. A support engineer troubleshooting can see order status without seeing payment details.
- **Data classification** — Identify and label sensitive data to apply appropriate controls. You can't protect what you haven't identified.
- **Encryption** — Protect data at rest and in transit. This is table stakes for any compliance framework.
- **Secret management** — Secure handling of database credentials and connection strings. Credentials in code or logs are a common breach vector.

**Risks without these controls:**

- Sensitive data exposed to users who only need partial access
- No visibility into where sensitive data resides across your databases
- Credentials leaked through code repositories or application logs

## How Bytebase Can Help

[Bytebase](/) is a database DevSecOps platform that addresses the heterogeneous database controls required for HITRUST compliance:

| Control Domain    | Bytebase Features                                                                                                                                                                                                                                                                                                                  |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Access Control    | [RBAC](https://docs.bytebase.com/administration/roles), [Just-in-Time access](https://docs.bytebase.com/security/database-permission/overview), [MFA](https://docs.bytebase.com/administration/2fa), [SSO](https://docs.bytebase.com/administration/sso/overview), [SCIM](https://docs.bytebase.com/administration/scim/overview/) |
| Audit Logging     | [Audit logging](https://docs.bytebase.com/security/audit-log)                                                                                                                                                                                                                                                                      |
| Change Management | [Change workflows](https://docs.bytebase.com/change-database/plan), [risk-based approval](https://docs.bytebase.com/change-database/approval), [rollback](https://docs.bytebase.com/change-database/rollback-data-changes), [change history](https://docs.bytebase.com/change-database/change-history)                             |
| Data Protection   | [Dynamic data masking](https://docs.bytebase.com/security/data-masking/overview), [data classification](https://docs.bytebase.com/security/data-masking/data-classification), [secret manager](https://docs.bytebase.com/get-started/connect/overview#secret-manager-integration)                                                  |

HITRUST provides the prescriptive controls that turn HIPAA's requirements into actionable implementation. Databases sit at the center of compliance — they store the sensitive data these frameworks exist to protect. The right tooling makes achieving and maintaining HITRUST compliance practical without sacrificing development velocity.


A practical guide to HITRUST database compliance covering access control, audit logging, change management, and data protection.

Database Compliance for HITRUST: Controls, Risks, and Best Practices


A table without a primary key seems harmless until your system grows and problems appear. Missing primary keys commonly lead to **duplicate data, broken CDC pipelines, replication stalls, and inconsistent analytics**.

Bytebase [SQL Review includes the rule](https://docs.bytebase.com/sql-review/review-rules#require-primary-key):

> Bytebase considers this rule to be violated if the SQL tries to create a no primary key table or drop the primary key. If the SQL drops all columns in the primary key, Bytebase also considers that this SQL drops the primary key.

## Real Incidents Caused by Missing Primary Keys

**PostgreSQL logical replication breaks**
Without a PK, Postgres cannot apply updates/deletes during logical replication (or Debezium).
Reference: [TIL: Creating tables without primary keys CAN cause updates and deletes to fail in Postgres](https://abhinavomprakash.com/posts/replica-identities/)

**Matomo production replication stalled**
A single table without a primary key caused MySQL master–slave replication to stop.
Reference: [Master–Slave Replication Stalls Because of Missing Primary Key](https://forum.matomo.org/t/master-slave-replication-stalls-because-of-missing-primary-key/36251)

**GitLab reported schema inconsistencies**
GitLab engineers found tables without primary keys leading to environment drift and maintenance issues.
Reference: [Database schema missing many primary keys - breaks replication](https://gitlab.com/gitlab-org/gitlab-ce/-/issues/51964)

## Why Missing Primary Keys Are Dangerous

### **1. Duplicate rows slip in**

Without a PK, the database cannot enforce uniqueness.
Accidental duplicates appear, corrupting analytics and reports.

### **2. CDC systems can’t track row changes**

Tools like Debezium and Kafka Connect need a stable row identity.
No PK → they can’t emit correct update/delete events.

### **3. Replication may stop or diverge**

Both MySQL and PostgreSQL depend on primary keys during replication.
A missing PK can cause replication to stall or go out of sync.

### **4. Upserts don’t work correctly**

`INSERT … ON CONFLICT`, `MERGE`, and UPSERT patterns require a PK.
Without one, the database can’t resolve conflicts reliably.

### **5. Debugging becomes guesswork**

Without a unique row identifier, you can’t target a specific record.
Deleting, fixing, or investigating a single row becomes unsafe.

## How to Fix Tables Without Primary Keys?

### **1. Add a surrogate primary key**

```sql
ALTER TABLE events
ADD COLUMN id BIGSERIAL PRIMARY KEY;
```

### **2. Use a natural composite key if appropriate**

```sql
ALTER TABLE order_items
ADD PRIMARY KEY (order_id, line_number);
```

### **3. Combine surrogate PK with a unique business key**

```sql
ALTER TABLE shipments
ADD COLUMN id BIGSERIAL PRIMARY KEY,
ADD CONSTRAINT shipments_unique UNIQUE (tracking_number);
```


Learn why requiring a primary key is important and how the "Require Primary Key" review rule protects your production database.

SQL Review Rule Explained - Require Primary Key


Bytebase [SQL Review](https://docs.bytebase.com/change-database/review) contains a rule to enforce including "NOT VALID" option when adding "CHECK" constraints:

> Adding a CHECK constraint needs to verify the existing data and requires ACCESS EXCLUSIVE table lock. This blocks read and write, which may cause business interruption. It is recommended to add the "NOT VALID" option to validate new data and manually validate existing data after the change is completed.

PostgreSQL allows you to add `CHECK` constraints to enforce data quality rules on a table. This is useful, but adding a CHECK constraint in the wrong way can block reads and writes and cause unexpected downtime.
This SQL Review rule ensures that all new CHECK constraints are created using the safe, non-blocking approach.

## How CHECK Constraints Work

A CHECK constraint ensures that a column’s value satisfies a given condition, for example:

```sql
CHECK (amount > 0)
```

PostgreSQL will reject any insert or update that violates this rule.

To add a CHECK constraint, you typically run:

```sql
ALTER TABLE orders
ADD CONSTRAINT orders_positive CHECK (amount > 0);
```

This enforces the rule on future writes, but it also immediately validates all existing rows.
This is where the risk comes from.

## Why Adding CHECK Constraints Can Cause Downtime

When PostgreSQL validates an existing table, it must scan every row to confirm the constraint is not violated. During this step, it acquires an `ACCESS EXCLUSIVE` lock. This lock is the strongest one in PostgreSQL and can block:

- Reads
- Writes
- Other schema changes

On large tables or busy production databases, the lock can cause:

- Query timeouts
- Application errors
- Service degradation
- Full outages

This SQL Review rule prevents teams from accidentally introducing blocking schema changes during deployments.

## The Safe Approach: Use `NOT VALID`

PostgreSQL offers a safer method to add CHECK constraints to existing tables.
The idea is to separate constraint creation from constraint validation.

**Step 1: Create the constraint without validating existing rows**

```sql
ALTER TABLE orders
ADD CONSTRAINT orders_positive CHECK (amount > 0) NOT VALID;
```

- Only a brief catalog lock is required
- The constraint is enforced for all new inserts and updates
- Existing rows are not scanned yet

**Step 2: Validate at a convenient time**

```sql
ALTER TABLE orders
VALIDATE CONSTRAINT orders_positive;
```

Validation uses a lighter lock that does not block reads and writes.
You can perform this step during low-traffic periods or as part of a controlled rollout.

## Example

Unsafe pattern (not allowed):

```sql
ALTER TABLE accounts
ADD CONSTRAINT check_balance CHECK (balance >= 0);
```

Safe pattern (recommended):

```sql
ALTER TABLE accounts
ADD CONSTRAINT check_balance CHECK (balance >= 0) NOT VALID;
```

Validate separately:

```sql
ALTER TABLE accounts
VALIDATE CONSTRAINT check_balance;
```

This ensures the constraint is applied safely without disrupting user traffic.

## Summary

Adding CHECK constraints without `NOT VALID` can block reads and writes and lead to downtime.
This SQL Review rule enforces the two-step PostgreSQL best practice:

- Add the constraint using `NOT VALID`
- Validate it later using a non-blocking operation

By following this pattern, teams can ensure constraint correctness without risking production stability.


Learn why enforcing NOT VALID in CHECK constraints is important and how the "Enforce NOT VALID in CHECK" review rule protects your production database.

SQL Review Rule Explained - Enforce NOT VALID in CHECK


Audit logging is a cornerstone of database security and compliance. Whether you’re tracking who changed what, investigating anomalies, or preparing for an audit, PostgreSQL gives you several ways to record activity at different levels of detail.

In this guide, we’ll walk through the most practical approaches — from PostgreSQL’s built-in logging to advanced tools like pgAudit and Bytebase — to help you choose the right setup for your organization.

## 1. Native PostgreSQL Logging

PostgreSQL comes with a [logging subsystem](https://www.postgresql.org/docs/current/runtime-config-logging.html) out of the box. It’s often the first step in building an audit trail.

**Key settings**

You can enable and configure logging in your `postgresql.conf` file:

```ini
logging_collector = on
log_statement = 'all'          # Options: none, ddl, mod, all
log_line_prefix = '%m [%p] %u@%d '   # Timestamp, process ID, user, database
log_duration = on
log_destination = 'csvlog'
```

This setup tells PostgreSQL to collect all executed SQL statements, include who ran them and when, and record query durations.

**Pros**

- Simple to enable, built-in, no extension needed.
- Useful for performance analysis and basic audit visibility.

**Cons**

- Unstructured text logs — difficult to parse automatically.
- Can grow large quickly.
- May include sensitive query parameters.

You can use tools like [pgBadger](https://github.com/darold/pgbadger) to analyze these logs and generate visual reports of who executed what queries and when.

## 2. Trigger-Based Auditing

If you need to record **row-level changes** — for example, before and after values on UPDATE — you can use triggers.

**Example**

Create a table to store change history:

```sql
CREATE TABLE audit_log (
  id serial PRIMARY KEY,
  table_name text,
  action text,
  changed_by text,
  changed_at timestamptz DEFAULT now(),
  old_data jsonb,
  new_data jsonb
);
```

Then define a trigger:

```sql
CREATE OR REPLACE FUNCTION audit_trigger() RETURNS trigger AS $$
BEGIN
  INSERT INTO audit_log (table_name, action, changed_by, old_data, new_data)
  VALUES (TG_TABLE_NAME, TG_OP, current_user, row_to_json(OLD), row_to_json(NEW));
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER audit_user_table
AFTER INSERT OR UPDATE OR DELETE ON users
FOR EACH ROW EXECUTE FUNCTION audit_trigger();
```

This captures all DML operations (`INSERT`, `UPDATE`, `DELETE`) on the `users` table, recording what changed and by whom.

**Pros**

- Captures before/after data.
- Fully customizable schema.

**Cons**

- Must be defined per table.
- Can impact performance on write-heavy workloads.

For real-time change notifications, PostgreSQL offers the [tcn](https://www.postgresql.org/docs/current/tcn.html) module (Triggered Change Notification), which can be used to send `NOTIFY` events to listening clients when data changes.

## 3. Logical Replication Based Auditing

PostgreSQL's [logical replication](https://www.postgresql.org/docs/current/logical-replication.html) decodes changes from the Write-Ahead Log (WAL) and streams them in a structured format. This allows you to capture all data changes without adding triggers or modifying application code.

**How It Works**

Logical replication uses _replication slots_ and _output plugins_ (like `wal2json` or `pgoutput`) to convert WAL entries into JSON or other formats. Tools like [Debezium](https://debezium.io/) can then consume these changes and forward them to audit storage systems like Kafka, Elasticsearch, or S3.

**Example Setup**

Enable logical replication in `postgresql.conf`:

```ini
wal_level = logical
max_replication_slots = 4
max_wal_senders = 4
```

Create a replication slot:

```sql
SELECT * FROM pg_create_logical_replication_slot('audit_slot', 'wal2json');
```

Read changes:

```sql
SELECT * FROM pg_logical_slot_get_changes('audit_slot', NULL, NULL);
```

You'll receive JSON output like:

```json
{
  "change": [
    {
      "kind": "update",
      "schema": "public",
      "table": "users",
      "columnnames": ["id", "email", "updated_at"],
      "columnvalues": [42, "user@example.com", "2025-11-10 10:30:00"],
      "oldkeys": { "keynames": ["id"], "keyvalues": [42] }
    }
  ]
}
```

**Pros**

- Captures all data changes automatically, no per-table setup.
- Minimal performance impact — reads from existing WAL infrastructure.
- Works with existing tools like Debezium for streaming to external systems.
- No application code changes required.

**Cons**

- Does not capture `SELECT` queries (only data modifications).
- WAL retention can increase storage if replication slot falls behind.

This approach is ideal when you need **near-real-time change data capture (CDC)** for auditing, analytics, or event-driven architectures.

## 4. pgAudit Extension

For structured, compliance-grade audit logs, PostgreSQL’s [pgAudit](https://github.com/pgaudit/pgaudit) extension is the standard choice.
It extends native logging to provide more context and granularity, especially around read/write operations.

**Installation**

Enable the extension:

```sql
CREATE EXTENSION pgaudit;
```

Update configuration:

```ini
shared_preload_libraries = 'pgaudit'
pgaudit.log = 'READ, WRITE'
pgaudit.log_catalog = off
```

After restarting PostgreSQL, you’ll start seeing audit logs like:

```plain
AUDIT: SESSION,1,READ,SELECT,,,,"SELECT * FROM customers WHERE id=42;",<none>
```

**Benefits**

- Records who executed which statement, in which session.
- Captures both DDL and DML activity.
- Integrates with PostgreSQL’s standard log collector — no new storage model.

**Considerations**

- Logs can be verbose — use `pgaudit.log_parameter = off` to reduce noise.
- Requires proper log rotation and analysis strategy.

## 5. Bytebase

Bytebase is a Database DevSecOps platform that provides a [centralized audit trail](https://docs.bytebase.com/security/audit-log) across your PostgreSQL environments.
It records _who did what, when, and why_ — linking SQL actions to their context (issues, approvals, and deployments) while keeping sensitive data secure.

**What Bytebase Audits**

- **Query access:** logs _who queried which data_ and _when_ across SQL Editor, Admin Query, and Data Export.
- **Schema and data changes:** tracks _who made which changes_, _when they were approved_, and _through which workflow or Git commit_.
- **Governance controls:** built-in SQL review rules, approval flow, and role-based access help prevent unauthorized actions.
- **Actual end users, not just database users:** A critical advantage is that Bytebase solves the **shared database user problem**. In most applications, all queries use the same database connection user (like `app_user`), making it impossible to trace actions back to individual users using traditional database auditing. Since users operate through Bytebase, every action is attributed to the actual end user, not just a generic database account.

**Why It Matters**

- **Complete visibility** across read and write operations.
- **Privacy-safe auditing** with no sensitive data exposure.
- **Compliance-ready** logs aligned with SOC 2, ISO 27001, and GDPR.

You can also call the [API](https://docs.bytebase.com/integrations/api/audit-log) to send the audit logs to a centralized log sink.

## Conclusion

PostgreSQL offers multiple layers of auditing — from basic text logs to complete governance solutions.

**Comparison Table**

| Approach | Performance Impact | Captures SELECTs | Captures Actor | Row-Level Detail | Best For |
|----------|-------------------|------------------|----------------|------------------|----------|
| **Native Logging** | Low-Medium | ✅ | ⚠️ DB user only | ❌ | Development, debugging, basic audit trails |
| **Triggers** | Medium-High | ❌ | ⚠️ DB user only | ✅ (before/after) | Critical tables needing full change history |
| **Logical Replication** | Low | ❌ | ❌ | ✅ | Real-time CDC, event-driven systems, analytics |
| **pgAudit** | Medium | ✅ | ⚠️ DB user only | ❌ | Compliance requirements, structured logging |
| **Bytebase** | N/A (app-level) | ✅ | ✅ End user | ✅ | Centralized governance, approval workflows, team collaboration |

- Use **native logging** for baseline activity tracking (captures database user only).

- Use **triggers** for fine-grained row-level auditing on critical tables (captures database user only).

- Use **logical replication** for near-real-time change data capture without application changes.

- Use **pgAudit** for structured, compliance-grade statement logging (captures database user only).

- Add **Bytebase** for centralized auditing that tracks actual end users (not just database users), linking every action to the person who performed it with full context of approvals and workflows.

By combining these layers, you gain both the visibility and control needed for secure, compliant, and well-governed database operations.


Postgres Audit Logging Guide


> A single misunderstood `CASCADE` command can wipe out production data in seconds. The keyword appears harmless but can trigger deletions across your entire database schema.

Bytebase [SQL Review](https://docs.bytebase.com/change-database/review) contains the following rules to prevent **CASCADING** failures:

1. **Prohibit using CASCADE option for ON DELETE clauses**: The `CASCADE` option in `ON DELETE` can cause a large number of dependent objects to be deleted or modified, leading to unexpected results.

1. **Prohibit using CASCADE when removing a table**: Using the `CASCADE` option when removing a table can cause a large number of dependent objects to be deleted or modified, leading to unexpected results.

## What is CASCADE?

CASCADE appears in two contexts in SQL:

### CASCADE in Foreign Key Constraints

```sql
CREATE TABLE posts (
    user_id INT,
    FOREIGN KEY (user_id) REFERENCES users(id)
        ON DELETE CASCADE  -- Deleting a user deletes all their posts
);
```

### CASCADE in TRUNCATE Operations

```sql
TRUNCATE TABLE users CASCADE;  -- Truncates users AND all tables referencing it
```

_Note: `TRUNCATE ... CASCADE` is PostgreSQL-specific. This behavior (automatically truncating dependent tables via foreign keys) doesn't exist in MySQL, SQL Server, or other databases. Oracle has `TRUNCATE ... CASCADE` but for a different purpose (materialized views)._

This is especially dangerous—many developers misunderstand its behavior.

### The Common Misunderstanding

Imagine a team is migrating their database and wants to clean up test data from new tables. A developer runs:

```sql
TRUNCATE TABLE new_feature_table CASCADE;
```

**Their intention:** Delete test data from the new tables.

**Their assumption:** CASCADE only deletes related test data in other tables.

**What actually happened:** `TRUNCATE ... CASCADE` deleted **ALL data** in every table with foreign keys pointing to `new_feature_table`—including years of production data.

The key misunderstanding: `TRUNCATE TABLE A CASCADE` doesn't just delete related rows in table B. It truncates **entire tables** that have foreign keys pointing to table A. In complex schemas with interconnected tables, a single CASCADE command can wipe out your entire database.

## How to Protect Your Database

**1. Use explicit DELETE statements**

```sql
-- ❌ DANGEROUS: CASCADE affects unexpected tables
TRUNCATE TABLE users CASCADE;

-- ✅ SAFE: Full control over what's deleted
DELETE FROM user_sessions WHERE user_id IN (SELECT id FROM users WHERE ...);
DELETE FROM users WHERE ...;
```

**2. Implement automated SQL review**

Use Bytebase to automatically scan all SQL before execution and block dangerous `CASCADE` operations in production.

**3. Remove dangerous privileges**

```sql
REVOKE TRUNCATE ON ALL TABLES IN SCHEMA public FROM app_user;
```

**4. Test migrations thoroughly**

Run the full migration in staging with production-like data, verify the scope of impact, and test rollback procedures.

**5. Use safer foreign key options**

Consider `ON DELETE RESTRICT` or `ON DELETE SET NULL` instead of `ON DELETE CASCADE`.

## Summary

The `CASCADE` keyword can trigger automatic data deletion across your entire database. The most critical point: `TRUNCATE TABLE A CASCADE` truncates **ALL tables** with foreign keys pointing to A—not just related rows.

**Protect your database:**

- Use explicit DELETE statements for full control over data deletion
- Block CASCADE in production using Bytebase SQL review rules
- Remove TRUNCATE privileges from application users
- Always test schema changes in staging environments


Learn why CASCADE operations in SQL can lead to catastrophic data loss and how the "Prohibit CASCADE" review rule protects your production database.

SQL Review Rule Explained - Prohibit CASCADE


## From Supabase to AWS

[Supabase](https://supabase.com/) has become the go-to choice for developers who want to build fast.
It gives you everything in one place — PostgreSQL, Auth, Storage, and Edge Functions — without needing to manage infrastructure.

But as your product matures, new requirements appear that go beyond what Supabase’s all-in-one model can offer:

- **Advanced database management** – high availability, read replicas, automated backups, and fine-tuned performance.
- **Analytics and data warehousing** – integrating with data lakes and large-scale ETL pipelines.
- **Enterprise-grade compliance and security** – meeting SOC, ISO, HIPAA, or FedRAMP standards.
- **Granular IAM and networking** – unifying database access, APIs, and infrastructure under a single identity and policy system.

That's where hyperscalers like [AWS](https://aws.amazon.com/), [GCP](https://cloud.google.com/), and [Azure](https://azure.microsoft.com/) come in. They offer a **best-of-breed ecosystem** — each component is purpose-built, scales independently.

**This guide focuses on AWS**, but the migration principles and architecture patterns apply equally to other cloud providers.

We'll walk you through how to migrate each Supabase component to its AWS counterpart.

---

## Debundle Supabase

| Supabase Component   | Equivalent                                     | Notes                                                   |
| -------------------- | ---------------------------------------------- | ------------------------------------------------------- |
| **Networking & IAM** | VPC + IAM roles/policies                       | AWS foundation - must be set up first                   |
| **Auth**             | Amazon Cognito / BetterAuth / Auth0            | Centralized user management, SSO, and MFA               |
| **Storage**          | Amazon S3                                      | Object storage with IAM-based access and CloudFront CDN |
| **Functions**        | AWS Lambda + API Gateway                       | Event-driven compute for backend logic                  |
| **Realtime**         | AppSync / EventBridge / API Gateway WebSockets | Live updates, subscriptions, or event streams           |
| **Database**         | Amazon RDS / Aurora                            | Migrate last - becomes simple PostgreSQL migration      |

**Recommended migration order:**

This guide uses a **Services-First** approach — decouple Supabase-specific services first, then migrate the database last as a simple PostgreSQL migration.

1. **Networking & IAM** – set up AWS infrastructure (VPC, subnets, IAM roles).
1. **Auth** – migrate to Cognito (decouples from Supabase `auth` schema).
1. **Storage** – migrate to S3 (decouples from Supabase `storage` schema).
1. **Functions** – redeploy to Lambda (update to use Cognito + S3).
1. **Realtime** – replace with AppSync/EventBridge (removes logical replication dependency).
1. **Database** – simple PostgreSQL migration of application data only (`public` schema).

**Why this order?** By migrating services first, the Supabase `auth` and `storage` schemas become dormant. The final database migration is just your application data — lower risk, simpler cutover, and easier to validate.

Always start in **staging**, validate each part, then proceed to production.

### 1. Networking and IAM

> **Note:** The networking and identity concepts in this section apply to all major cloud providers (AWS, GCP, Azure). This guide focuses on AWS implementations, but the architecture patterns translate directly to VPC/IAM (GCP) or VNet/RBAC (Azure).

**Supabase:**
Abstracted networking and simple project-level access roles.

**AWS approach:**
Full-control networking and IAM system for isolation and compliance.

| Concept                | Supabase          | AWS Equivalent                 |
| ---------------------- | ----------------- | ------------------------------ |
| Top-level entity       | Organization      | AWS Organization               |
| Project                | Supabase Project  | AWS Account                    |
| Environment separation | Multiple projects | Separate accounts or VPCs      |
| Access control         | Role-based in app | IAM users, roles, and policies |

**Migration focus:**

1. Create VPC with public and private subnets across multiple Availability Zones.
1. Set up **Security Groups** for database, Lambda, and API Gateway.
1. Configure **Route Tables** and **NAT Gateways** for private subnet internet access.
1. Create **IAM roles** for Lambda, RDS, S3, and Cognito.
1. Set up **VPC endpoints** for AWS service access (S3, DynamoDB, etc.).
1. Use **AWS Organizations** for environment isolation (dev, staging, prod).

**Key advantages:**

- Granular control over infrastructure and networking.
- Centralized access and audit through IAM.
- Broad compliance coverage — [AWS Compliance](https://aws.amazon.com/compliance) vs [Supabase Security](https://supabase.com/security).

### 2. Auth → Amazon Cognito (or Alternatives)

**Supabase:**
[GoTrue](https://supabase.com/docs/guides/auth/auth-go-true)-based Auth with email/password and [OAuth integration](https://supabase.com/docs/guides/auth/social-login), connected to Postgres RLS.

**AWS replacement:**

- [Amazon Cognito](https://aws.amazon.com/cognito/) for user pools, federated identity, and SSO integration.
- Alternatives like [BetterAuth](https://betterauth.com/), [Auth0](https://auth0.com/), or [Clerk](https://clerk.com/) if developer-experience is a priority.

**Migration focus:**

1. Export user data (emails, metadata, OAuth IDs) from `auth.users`.
1. Import into Cognito User Pool.
1. Configure OAuth providers (Google, GitHub, etc.).
1. Update frontend SDKs and backend JWT verification.
1. Update application authorization logic (see RLS section below).
1. Require one-time user re-authentication after migration.

> **Important:** If you're using Supabase Row-Level Security (RLS) policies, they reference `auth.uid()` and won't work after migrating to Cognito. See the **Handling RLS Policies** section in the Database migration step for strategies to address this.

**Key advantages:**

- Deep IAM integration with AWS services.
- SAML/OIDC support and MFA.
- Fine-grained access control and security compliance.

### 3. Storage → Amazon S3

**Supabase:**
S3-compatible object storage managed inside Supabase, with integrated access policies and signed URLs.

**AWS replacement:**

- [Amazon S3](https://aws.amazon.com/s3/) for raw file storage.
- [CloudFront](https://aws.amazon.com/cloudfront/) for CDN delivery.

**Migration focus:**

1. Create an S3 bucket with IAM-based access.
1. Copy data using `aws s3 sync` or `rclone`.
1. Recreate folder structure and permissions.
1. Update signed URL logic to use S3 pre-signed URLs.
1. Add CloudFront for caching if needed.

**Key advantages:**

- Lifecycle policies, versioning, and encryption (SSE-KMS).
- Regional redundancy and cost-based storage tiers.
- Tight integration with Lambda, Athena, and Redshift.

### 4. Functions → AWS Lambda

**Supabase:**
[Edge Functions](https://supabase.com/docs/guides/functions/edge-functions) built with [Deno](https://deno.com/) for lightweight APIs.

**AWS replacement:**

- [AWS Lambda](https://aws.amazon.com/lambda/) for event-driven functions.
- [API Gateway](https://aws.amazon.com/api-gateway/) for HTTP endpoints.

**Migration focus:**

- Rewrite Deno functions in Node.js, Python, or Go.
- Deploy via Lambda console, CLI, or IaC (Terraform/CDK).
- Store environment variables in **Secrets Manager** or **Parameter Store**.
- Connect Lambda to S3, DynamoDB, or EventBridge as needed.

**Key advantages:**

- Multiple runtimes and deployment methods.
- Native observability via CloudWatch.
- Scales automatically with demand.

### 5. Realtime and Events → AppSync / EventBridge

**Supabase:**
Realtime engine based on Postgres logical replication and WebSockets.

**AWS replacements:**

- [AppSync](https://aws.amazon.com/appsync/) – GraphQL subscriptions for live updates.
- [EventBridge](https://aws.amazon.com/eventbridge/), [SNS](https://aws.amazon.com/sns/), or [SQS](https://aws.amazon.com/sqs/) – event-driven messaging.
- [API Gateway WebSockets](https://aws.amazon.com/api-gateway/) – persistent connections for custom protocols.

**Migration focus:**

1. Identify realtime use cases (chat, collaboration, notifications).
1. Choose appropriate AWS service per pattern.
1. Replace database-triggered realtime with event-driven design.

**Key advantages:**

- Decoupled architecture.
- Scalable pub/sub and async event flows.
- Integrates natively with Lambda and analytics pipelines.

### 6. Database → Amazon RDS / Aurora

**Supabase:**
Managed PostgreSQL with `auth`, `storage`, and `public` schemas.

**AWS replacement:**

- **Amazon RDS (PostgreSQL)** – Multi-AZ, automated backups, PITR, read replicas.
- **Amazon Aurora (PostgreSQL-compatible)** – high-performance clustered Postgres.
- **DynamoDB** – optional for NoSQL or key-value workloads.

**Why migrate last:**

By this point, Auth is on Cognito, Storage is on S3, and Realtime has been replaced with AppSync/EventBridge. The Supabase `auth` and `storage` schemas are now dormant — your application no longer uses them.

This makes the database migration simple — just a vanilla PostgreSQL migration of your application data (the `public` schema).

#### Handling Row-Level Security (RLS) Policies

Supabase uses PostgreSQL [Row-Level Security (RLS)](https://supabase.com/docs/guides/auth/row-level-security) policies heavily. These policies reference Supabase-specific functions and the `auth.users` table:

```sql
-- Example Supabase RLS policy
CREATE POLICY "Users can only see their own data"
ON public.profiles
FOR SELECT
USING (auth.uid() = user_id);
```

The problem: `auth.uid()` and `auth.jwt()` are Supabase-specific functions that won't work after migrating to Cognito.

**Migration Strategy: Replace RLS with Application-Level Authorization**

The recommended approach is to move authorization logic from the database to your application layer (Lambda functions):

- **More flexible** – Works with any auth provider (Cognito, Auth0, BetterAuth, etc.).
- **Easier to test** – Authorization logic can be unit tested independently.
- **Framework-agnostic** – Not tied to PostgreSQL-specific features.
- **Modern best practice** – Industry standard for cloud-native applications.
- **Better debugging** – Authorization failures are easier to trace in application code.

Implementation example:

```javascript
// Lambda function with application-level authorization
const getUserProfile = async (event) => {
  // Extract Cognito user ID from JWT claims
  const cognitoUserId = event.requestContext.authorizer.claims.sub;

  // Enforce authorization in application code
  const profile = await db.query('SELECT * FROM profiles WHERE user_id = $1', [cognitoUserId]);

  return profile;
};
```

**Database migration focus:**

1. **Audit RLS policies** – Identify all RLS policies in your schema (see RLS migration strategy above).
1. Set up RDS/Aurora instance in the VPC created in step 1.
1. Export `public` schema and data using `pg_dump` (ignore dormant `auth` and `storage` schemas).
1. Restore into RDS or Aurora (same Postgres version).
1. **Implement application-level authorization** – Migrate RLS logic to Lambda functions as shown above.
1. **Remove RLS policies** – Drop policies once application-level authorization is tested and deployed.
1. Recreate extensions (e.g., `pgcrypto`, `uuid-ossp`) if used in application.
1. Validate schema and queries in staging environment.
1. Update application connection strings to point to RDS/Aurora.
1. Perform final cutover during low-traffic window.

**Key advantages:**

- Performance tuning and [CloudWatch metrics](https://aws.amazon.com/cloudwatch/).
- Private networking in VPC with security groups.
- Access to AWS analytics tools ([Redshift](https://aws.amazon.com/redshift/), [Athena](https://aws.amazon.com/athena/), [Glue](https://aws.amazon.com/glue/)).

## Conclusion

Migrating from Supabase to AWS isn't just a lift-and-shift — it's a step toward scalable, enterprise-ready infrastructure.

Use the **Services-First** approach to decouple components gradually:
**Networking & IAM → Auth → Storage → Functions → Realtime → Database.**

By migrating services first, the final database migration becomes a simple PostgreSQL-to-PostgreSQL operation with just your application data — lower risk, simpler cutover, and easier to validate.

Supabase helps you **build fast**.
AWS helps you **scale next** — with advanced database management, analytics, IAM, and compliance.

When done right, the migration lays a foundation your product can grow on for years to come.


A comprehensive guide on migrating from Supabase to AWS, covering database, auth, storage, functions, realtime, and networking.

How to Migrate from Supabase to AWS


PostgreSQL’s transaction system is built around a simple but powerful idea: every transaction gets a **Transaction ID (XID)** - a 32-bit integer that powers its **MVCC (Multi-Version Concurrency Control)** engine.

MVCC lets multiple users read and write data at the same time without blocking each other.
But those transaction IDs are finite, and eventually they **wrap around**.

If not handled properly, wraparound can make data invisible or even force the entire database into emergency shutdown.
Let’s understand why.

### How Postgres Differs from Other Databases

PostgreSQL uses a **tuple-based** MVCC model - every data change creates a new version (tuple) of a row stored directly in the table.

Each tuple has two hidden fields: `xmin` and `xmax`. `xmin` is the transaction ID that created the tuple, and `xmax` is the transaction ID that deleted or replaced it.

Other databases handle versioning differently:

| Database           | MVCC type           | Where old versions are stored      | Wraparound risk |
| ------------------ | ------------------- | ---------------------------------- | --------------- |
| **PostgreSQL**     | Tuple-based         | In the main table (needs `VACUUM`) | ✅ Yes           |
| **MySQL (InnoDB)** | Undo-log-based      | Undo logs in system tablespace     | ❌ No            |
| **SQL Server**     | Version-store-based | Tempdb version store               | ❌ No            |
| **Oracle**         | Undo-log-based      | Undo segments                      | ❌ No            |

In short:
PostgreSQL keeps old rows **inline** with the table, and each one carries its own transaction ID.
That design enables powerful visibility control but also means transaction IDs must eventually be **reused** - leading to the wraparound problem.
Other databases store old versions separately, so their transaction identifiers can grow freely without ever wrapping around.

### What Is Transaction Wraparound

PostgreSQL’s transaction IDs are **32-bit integers** (`0` to `4,294,967,295`).
After the counter hits its limit, it **wraps around** back to 3 again.

You can picture XIDs as points on a **circle**, not a straight line.

![Transaction ID (XID) Wraparound](/content/blog/postgres-transaction-id-wraparound/pg-xid-cycle-circle.webp)

When the counter wraps, very old tuples may suddenly appear to have "future" XIDs and become **invisible** to all new transactions.

![Transaction ID (XID) Wraparound](/content/blog/postgres-transaction-id-wraparound/pg-xid-cycle.webp)

To prevent this, Postgres periodically **freezes** old tuples - marking them with a special `FrozenXID`, meaning "committed long ago, always visible."

![Transaction ID (XID) Wraparound](/content/blog/postgres-transaction-id-wraparound/pg-xid-freeze.webp)

### Consequences of Wraparound

If the database fails to freeze tuples in time, it risks data corruption.
When `datfrozenxid` becomes dangerously old, Postgres refuses new writes and may shut down with errors like:

```
PANIC: database is not accepting commands to avoid wraparound data loss
```

#### Real-world cases

- **Sentry (2024):**
  Sentry’s Postgres database stopped accepting writes after autovacuum couldn’t keep up with freezing old transaction IDs. The system hit the wraparound limit, forcing emergency manual vacuuming and downtime.
  [https://blog.sentry.io/transaction-id-wraparound-in-postgres/](https://blog.sentry.io/transaction-id-wraparound-in-postgres/)

- **Mailchimp/Mandrill (2016):**
  A busy shard’s autovacuum fell behind, triggering wraparound protection and halting writes. Recovery required truncations and manual vacuums, leading to roughly 40 hours of outage.
  [https://mailchimp.com/what-we-learned-from-the-recent-mandrill-outage/](https://mailchimp.com/what-we-learned-from-the-recent-mandrill-outage/)

These cases show that wraparound isn’t theoretical - it’s one of the few PostgreSQL maintenance failures that can completely stop production systems.

### How to Monitor and Prevent Wraparound

#### 1. Autovacuum is the First Line of Defense

Autovacuum automatically scans tables and freezes tuples before they age out.
Key parameters:

- `autovacuum_freeze_max_age` – threshold for wraparound prevention
- `vacuum_freeze_table_age` – when to start freezing during normal vacuum
- `vacuum_freeze_min_age` – minimum XID age before freezing allowed

If autovacuum is off or slow, wraparound danger grows silently.

#### 2. Check XID Age with SQL

To see how close your database is to wraparound:

```sql
SELECT datname, age(datfrozenxid) AS xid_age
FROM pg_database
ORDER BY xid_age DESC;
```

For table-level detail:

```sql
SELECT relname, age(relfrozenxid) AS xid_age
FROM pg_class
WHERE relkind = 'r'
ORDER BY xid_age DESC;
```

⚠️ **Guidelines**

- Above **1.5 billion** → warning zone
- Above **2 billion** → database may lock writes

#### 3. Cloud Provider Recommendations

**AWS RDS / Aurora**
Use `postgres_get_av_diag` to monitor autovacuum health and aging tables —
[https://aws.amazon.com/blogs/database/prevent-transaction-id-wraparound-by-using-postgres_get_av_diag-for-monitoring-autovacuum/](https://aws.amazon.com/blogs/database/prevent-transaction-id-wraparound-by-using-postgres_get_av_diag-for-monitoring-autovacuum/)

**Google Cloud SQL**
Cloud SQL provides a Recommender for High Transaction ID Utilization -
[https://cloud.google.com/sql/docs/postgres/recommender-high-transactionid-utilization](https://cloud.google.com/sql/docs/postgres/recommender-high-transactionid-utilization)

General advice:

- Never disable autovacuum.
- Schedule manual `VACUUM FREEZE` during off-peak hours.
- Avoid long-running idle transactions that block freezing.

### The Challenge of Moving to 64-bit Transaction IDs

At first glance, the easiest fix would be to make transaction IDs **64-bit** instead of 32-bit.
That would raise the ceiling from 4 billion transactions to roughly **18 quintillion** - effectively eliminating wraparound forever.

This idea has been discussed for years, with real prototypes already attempted:

- **Early discussions (2018–2019):**
  [https://www.postgresql.org/message-id/flat/DA1E65A4-7C5A-461D-B211-2AD5F9A6F2FD%40gmail.com](https://www.postgresql.org/message-id/flat/DA1E65A4-7C5A-461D-B211-2AD5F9A6F2FD%40gmail.com)
  Developers debated whether to store full 64-bit IDs or use a **hybrid scheme** (16-bit epoch + 48-bit XID) to retain compatibility.

- **Experimental patch for Postgres 15 (2021):**
  [https://www.postgresql.org/message-id/flat/CACG=ezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe=pyyjVWA@mail.gmail.com](https://www.postgresql.org/message-id/flat/CACG=ezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe=pyyjVWA@mail.gmail.com)
  It proved feasible but caused major ripple effects:

  - Every tuple grows by 8 bytes (`xmin` + `xmax`).
  - Index and WAL formats must be redesigned.
  - Replication and visibility logic rely on 32-bit arithmetic.

- **Community view:**
  [https://news.ycombinator.com/item?id=19083745](https://news.ycombinator.com/item?id=19083745)
  Developers agreed the change would *solve wraparound permanently* but **break on-disk compatibility**, forcing every database to migrate storage format.

For now, the community focuses on improving **autovacuum efficiency** and **wraparound monitoring**, accepting that 32-bit XIDs remain part of the architecture - at least until a cleaner migration path emerges.

### Best Practices

- ✅ Keep autovacuum **enabled and tuned**
- ✅ Monitor XID age regularly
- ✅ Vacuum frequently on high-write tables
- ✅ Avoid long-running transactions
- ✅ Run `VACUUM FREEZE` during maintenance windows
- ✅ Partition or archive old data to reduce bloat

What is Postgres Transaction ID (XID) Wraparound and how to monitor and prevent it.

Postgres Transaction ID (XID) Wraparound


Application code has long enjoyed the benefits of CI/CD pipelines—automated testing, version control, and structured deployment processes. Yet databases, which are often the most critical component of an application, frequently lag behind with manual, error-prone change processes.

A well-designed CI/CD pipeline for database schema migrations reduces deployment errors, improves auditability, and enables faster iteration. This guide walks you through the essential components, implementation patterns, and tooling options for building a database CI/CD pipeline.

## Why Database CI/CD Matters

Database changes carry unique risks compared to application deployments:

- **State persistence**: Unlike stateless application code, databases hold critical state. A bad migration can corrupt data permanently.
- **Downtime impact**: Schema changes often require table locks, directly affecting availability.
- **Coordination complexity**: Database changes must coordinate with application deployments—deploy schema changes too early and old code breaks; too late and new code breaks.
- **Limited rollback**: While application code can be rolled back instantly, database rollbacks are complex. You can't always undo a `DROP COLUMN` that deleted data.

A CI/CD pipeline addresses these risks through:

- **Automated validation**: Catch syntax errors, missing indexes, and unsafe operations before production
- **Consistent process**: Same deployment process across all environments eliminates "works on my machine" issues
- **Audit trail**: Track who approved what change, when it deployed, and what SQL executed
- **Controlled rollout**: Test migrations in dev/staging with production-like data before touching production
- **Reduced coordination overhead**: Automation reduces the cognitive load of manual deployments

According to the [6 levels of database automation](/blog/database-automation-levels), most organizations operate at Level 0-1 (manual changes or ticketing systems). Level 3-4 (streamlined and integrated) provides automated deployments with SQL review and approval workflows.

## Core Components of a Database CI/CD Pipeline

A complete database CI/CD pipeline consists of six essential components:

### 1. Change Planning

The pipeline begins with defining what needs to change. This includes:

- **Schema migrations (DDL)**: CREATE, ALTER, DROP statements for tables, indexes, and other schema objects
- **Data modifications (DML)**: INSERT, UPDATE, DELETE operations
- **Target scope**: Single database, multiple databases, or database groups

Modern database CI/CD platforms support both UI-driven and GitOps workflows. Choose based on your team's needs:

- **UI-Driven**: Visual interface for teams preferring centralized control and multi-level approvals
- **GitOps**: Code-first approach integrated with Git providers (GitHub, GitLab, Bitbucket) for developer-centric teams

### 2. Automatic SQL Review

Before any change reaches production, automated SQL review validates the migration:

**Syntax Validation**

- Catch SQL errors before deployment
- Verify database compatibility

**Schema Rules**

- Enforce naming conventions
- Validate data types and constraints
- Check for required fields

**Performance Checks**

- Identify missing indexes
- Detect inefficient queries
- Flag full table scans

**Security Policies**

- Prevent unsafe operations (DROP TABLE in production)
- Detect potential data exposure
- Enforce access controls

**Backward Compatibility**

- Ensure changes won't break existing applications
- Verify migration reversibility

SQL Review policies can be configured at the environment or project level, allowing you to enforce different standards for development versus production environments.

### 3. Approval Process

Changes must go through an approval workflow before deployment. Effective approval systems offer:

**Risk-Based Routing**

- ✅ Low-risk changes (dev environment, backward-compatible): Automatic approval or minimal review
- ⚠️ Moderate-risk changes: Single approver review
- 🚨 High-risk changes (production DDL, large data updates): Multi-level approval

**Role-Based Authorization**

- DBA approval for schema changes
- Security team approval for permission changes
- Manager approval for production deployments

**Integration Options**

- Built-in approval within the database CI/CD platform
- Pull request reviews in GitHub/GitLab/Bitbucket
- External ticketing systems (ServiceNow, Jira)

The approval process should be flexible enough to handle both planned releases and emergency hotfixes without becoming a bottleneck.

### 4. Multi-Environment Rollout Pipeline

Database changes must progress through environments in a controlled manner:

**Environment Chain**

```plain
Development → Testing → Staging → Production
```

**Stage Configuration**

- Define custom environment chains
- Configure different database groups per stage
- Set environment-specific policies

**Deployment Execution**

- Parallel execution across database groups
- Automatic retry for transient failures

**Gated Progression**

- Manual gates for critical environments
- Automatic promotion for lower environments
- Smoke tests between stages

### 5. Rollback Capabilities

Even with thorough testing, things can go wrong. Robust rollback capabilities are essential:

**DML Rollback**

- One-click recovery for UPDATE/DELETE operations
- Automatic backup before risky data changes

**Schema Rollback**

- Generate reverse migration scripts
- Test rollback procedures in lower environments
- Document rollback steps

Not all database changes are easily reversible (e.g., column drops destroy data). Document irreversible changes and ensure stakeholder awareness before deployment.

### 6. Schema Drift Detection

Schema drift occurs when changes are made outside the CI/CD pipeline—a common problem in organizations with mixed practices.

**Drift Detection Features**

- Continuous monitoring of database schemas
- Alerts when unexpected changes detected
- Comparison against expected state
- Integration with change management workflow

When drift is detected, the system should:

1. Notify relevant teams immediately
1. Document the unexpected change
1. Provide options to either incorporate into version control or revert

## Real-World Example: Adding a New Column

Here's how a schema change flows through a CI/CD pipeline, with technical considerations at each stage:

**1. Developer Creates Migration**

```sql
-- V042__add_user_email_verified_column.sql
-- Add nullable column first to avoid rewriting the entire table
ALTER TABLE users ADD COLUMN email_verified BOOLEAN DEFAULT NULL;

-- Backfill in batches for large tables (assuming 10M+ rows)
-- UPDATE users SET email_verified = FALSE WHERE email_verified IS NULL;
-- (Run separately in batches to avoid long-running transactions)

-- Add index - considerations:
-- - MySQL: Can cause table locks, consider ALGORITHM=INPLACE
-- - PostgreSQL: Use CREATE INDEX CONCURRENTLY to avoid blocking writes
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_email_verified
ON users(email_verified) WHERE email_verified = FALSE;
-- Partial index: only index unverified users for query efficiency
```

**2. Pull Request and Review**

Automated SQL review catches potential issues:

- ✅ Pass: Partial index reduces index size
- ⚠️ Warning: Index creation may take 10+ minutes on production (12M rows)
- ✅ Pass: No foreign key constraints that could cause cascading locks

Manual review considerations:

- Index cardinality: Will this index be selective enough? (Expected: 5% false, 95% true)
- Deployment timing: Index creation doesn't block reads but may impact replication lag
- Application compatibility: New column defaults to NULL, application must handle

**3. Merge and Automatic Deployment to Dev**

```bash
# CI/CD pipeline runs:
- git merge main
- ./migrate.sh dev  # Applies migration to dev environment
- npm run test:integration  # Verifies application handles new column
```

Migration completes in 2 seconds on dev (10K rows). No issues detected.

**4. Staging Deployment**

Staging has production-like data volume (10M rows):

- Migration takes 8 minutes (index creation)
- Replication lag spikes to 45 seconds during index build
- QA verifies:
  - Application code reads email_verified correctly
  - Performance of queries using the new index
  - NULL handling for existing rows

**5. Production Deployment**

DBA review checklist:

- ✅ Confirmed: Staging migration succeeded
- ✅ Confirmed: No application errors in staging
- ✅ Plan: Deploy during low-traffic window (2 AM PST)
- ✅ Plan: Monitor replication lag during index creation
- ✅ Rollback: Can drop column and index if needed

Deployment execution:

```sql
-- PostgreSQL: Monitor execution time and locking
SELECT pid, query, state, wait_event_type, wait_event
FROM pg_stat_activity
WHERE query LIKE '%idx_users_email_verified%';

-- Check index creation progress
SELECT phase, blocks_done, blocks_total,
       round(100.0 * blocks_done / NULLIF(blocks_total, 0), 1) AS percent_done
FROM pg_stat_progress_create_index;

-- If issues arise, can cancel concurrent index creation:
-- DROP INDEX CONCURRENTLY idx_users_email_verified;

-- MySQL: Monitor execution time and locking
-- SHOW PROCESSLIST;  -- Check for blocked queries
-- SELECT * FROM information_schema.innodb_trx;  -- Check transactions
```

Migration completes in 11 minutes. Application deployment follows after confirming no database issues.

**Key Takeaways from This Example:**

- Nullable columns avoid expensive table rewrites on large tables
- Index creation strategy differs by database (CONCURRENTLY, ALGORITHM=INPLACE)
- Production deployments need specific timing based on table size and traffic patterns
- Always verify migrations at production scale in staging first

## Choosing the Right Tools

While this guide focuses on concepts and processes, implementation requires tooling. Three widely-used open-source options for database schema migration are Bytebase, Liquibase, and Flyway. Here's how they compare across the core CI/CD components:

| Component                     | Bytebase                                                 | Liquibase                                                   | Flyway                                                                                          |
| ----------------------------- | -------------------------------------------------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| **Interface**                 | Web GUI, API, Terraform                                  | CLI, Java API, Maven/Gradle                                 | CLI, Java API, Maven/Gradle                                                                     |
| **Installation**              | ⭐ Single binary (Go), Docker, K8s                       | Requires JVM                                                | Requires JVM                                                                                    |
| **Change Planning**           | ⭐ UI-driven or GitOps with project/issue model          | Changelog files (XML/YAML/SQL) + CLI                        | SQL migration files + CLI                                                                       |
| **Batch Changes**             | ⭐ Multi-environment, Multi-tenant with Database Groups  | Manual scripting required                                   | Manual scripting required                                                                       |
| **SQL Review**                | ⭐ 200+ built-in SQL Review rules (Free)                 | Policy Checks (Pro plan only) - custom rules can be created | Code Analysis (Teams/Enterprise plans) - supports Regex and SQLFluff rules                      |
| **Approval Workflow**         | ⭐ Risk-based custom approval with multi-stage flow      | Not a built-in feature                                      | Not a built-in feature                                                                          |
| **Multi-Environment Rollout** | ⭐ Automated pipeline with environment-specific policies | Manual orchestration via scripts                            | Manual orchestration via scripts                                                                |
| **Rollback**                  | ⭐ Auto-generated rollback statements for DDL/DML        | Automatic for some operations, manual for others            | Undo migrations (Teams/Enterprise plans); auto-generation of undo scripts in Enterprise edition |
| **Schema Drift Detection**    | ⭐ Automatic detection with alerts                       | Not a built-in feature                                      | Drift reports (Enterprise plan) - manual check via CLI command                                  |
| **GitOps**                    | Manual CI/CD integration                                 | Manual CI/CD integration                                    | Manual CI/CD integration                                                                        |
| **Change History**            | ⭐ Full history with diffs, issue tracking, audit logs   | Database changelog table                                    | Database migration table                                                                        |
| **Webhook Integration**       | ⭐ Slack, Teams, Discord, and more                       | Not a built-in feature                                      | Not a built-in feature                                                                          |
| **Supported Databases**       | 20+ SQL & NoSQL                                          | ⭐ 50+ SQL & NoSQL                                          | ⭐ 50+ SQL & NoSQL                                                                              |

## When You Might NOT Need Full CI/CD

Database CI/CD adds overhead. You might not need the full pipeline if:

- **Early-stage startup**: (less than 10 databases, 5 developers) - CLI tools may suffice
- **Read-only analytical databases**: Fewer schema changes, lower risk
- **Ephemeral dev environments**: Fully automated recreation might be simpler
- **Legacy systems**: Migration effort may outweigh benefits for systems nearing replacement

Start with version control and automated deployments, then add approval workflows and observability as team size and complexity grow.

## Conclusion

Database CI/CD moves you from ad-hoc changes to systematic, auditable processes. The goal is to achieve [Level 3-4 automation](/blog/database-automation-levels)—streamlined deployments with integrated approval workflows—without overengineering.

Implementation path:

1. Version control your migrations (Level 2)
2. Add automated deployment and SQL review (Level 3)
3. Layer in approval workflows and observability (Level 4)

Choose tooling based on your architecture: platforms like Bytebase for collaboration and governance, libraries like Liquibase/Flyway for CLI-first workflows. All three are production-ready; the right choice depends on your team's size, practices, and compliance requirements.


A comprehensive guide on building a CI/CD pipeline for automated database schema migration, covering best practices, tools, and implementation strategies.

How to Build a CI/CD Pipeline for Database Schema Migration


This article focuses on my Day 1 development workflow with Postgres—the tools I use to build and ship features. I won't be covering Day 2 operations like backup, monitoring, or performance tuning in production. Instead, this is about the everyday development cycle: spinning up a local database, writing queries, managing schema changes, and running tests. These are the tools that live in my development environment and help me move from idea to deployed feature.

## Running Postgres Locally

I'm on a Mac, and my current go-to for local Postgres is [StackBricks](https://stackbricks.app/). It supports for multiple Postgres versions and other database engines beyond Postgres. Under the hood, it spins up Docker containers, which gives isolation.

I also used [DBngin](https://dbngin.com/) before, which provides a similarly smooth experience with a lightweight UI for managing local database instances.

Before discovering these tools, I was a long-time user of [Postgres.app](https://postgresapp.com/). Postgres.app is dead simple: download, click, and you have Postgres running natively. Also, I think it has the best icon among all Postgres tools.

<img src="/content/blog/how-i-ship-features-with-postgres/postgres-app.webp" alt="Postgres.app" style={{maxWidth: '200px', width: '100%', height: 'auto', display: 'block', margin: '0 auto'}} />

## SQL Editor

For writing and running SQL queries, I use [TablePlus](https://tableplus.com/). It has an elephant logo, but it actually supports a wide range of databases beyond Postgres—MySQL, Redis, SQLite, and more.

I can't point to one particular feature that makes TablePlus stand out. It's more that everything just feels smooth and easier to use compared to other tools. Part of this comes from it being built with native Mac AppKit instead of web technology—you can feel the difference in responsiveness and how it integrates with macOS. But really, it's all those small details adding up.

## Schema Migration

For schema migrations, my choice depends on the tech stack. With Go backends, I use [golang-migrate](https://github.com/golang-migrate/migrate). It's straightforward—write SQL migration files, run them up or down. No magic, just SQL.

When working with Node, I prefer [Drizzle](https://orm.drizzle.team/). The main selling point for me is that Drizzle can export the full schema. This is useful for keeping a copy of my latest database schema in the repo, which makes it easy to understand the current state without piecing together migration files.

Overall though, I think schema migration tooling for Postgres can be improved. Most tools follow the imperative migration pattern—you write a series of "ALTER TABLE" statements. That's why I built [pgschema](https://github.com/pgplex/pgschema)—a Postgres-specific declarative solution. You define what you want your schema to look like, and the tool figures out the migrations needed to get there. It's still early, but it solves a real pain point I've experienced.

## Running Integration Tests

Integration tests often need a real Postgres database.

For Node applications, I use [Testcontainers](https://testcontainers.com/modules/postgresql/). It spins up Docker containers on-demand during tests and tears them down afterward. It supports all mainstream Postgres versions.

For Go applications, I use [embedded-postgres](https://github.com/fergusstrange/embedded-postgres). Instead of spinning up Docker containers, it downloads and runs a Postgres binary directly. This saves the cost and complexity of running Docker—tests start faster, and it's one less dependency to manage.

## Type Safety

For Go applications, I use [sqlc](https://github.com/sqlc-dev/sqlc) to generate type-safe code from SQL queries. You write SQL, sqlc generates the Go code with proper types. No ORM overhead, just type-safe database access.

For TypeScript applications, I pair my database layer with [Zod](https://zod.dev/) to enforce runtime type validation. This catches type mismatches between your database schema and application code before they cause problems in production.

## For Tool Builders

As a Postgres tool builder, I heavily rely on [libpg_query](https://github.com/pganalyze/libpg_query) to parse PG statements. Most developers probably won't need it, but I want to give it a special shoutout—it's a key differentiator for Postgres compared with other database engines. It gives tool developers reliable parsing capability, which is a cornerstone to build strong tooling.

## Data Seeding

This is one area where I don't have a standard toolkit yet. I've used [Faker.js](https://fakerjs.dev/) before to generate realistic test data, but it's never become part of my regular workflow. Most of the time, I end up writing quick SQL scripts or custom seed files for each project.

I haven't found a solution that feels as smooth as the other tools in this article. If you have a data seeding approach that works well, I'd genuinely love to hear about it.

---

That's my current Postgres development toolkit. It's always evolving, what's working for you?


A practical guide to shipping features efficiently using PostgreSQL, covering development workflows, best practices, and real-world examples

How I Ship Features with Postgres


The [**Lei Geral de Proteção de Dados Pessoais (LGPD)**](https://en.wikipedia.org/wiki/General_Personal_Data_Protection_Law)—Brazil’s General Data Protection Law—came into effect in 2020. Much like Europe’s GDPR, it gives people more control over their personal data and sets out clear rules for how companies can collect, store, and use it.

But LGPD isn’t just a copy of GDPR. It reflects Brazil’s legal and cultural context, enforced by the [**Autoridade Nacional de Proteção de Dados (ANPD)**](https://www.gov.br/anpd/pt-br). For companies operating in or serving users from Brazil, LGPD compliance isn’t just a checkbox—it’s a matter of trust and legal risk.

And when it comes to data protection, the database is where it all becomes real. This is where personal data lives, where access happens, and where things can go wrong if not managed carefully.

In this post, we’ll look at LGPD through a database lens—and show how **Bytebase** helps teams bake compliance into their day-to-day workflow.

## Who LGPD Applies To

LGPD applies to _any_ organization—Brazilian or foreign—that processes personal data from individuals located in Brazil.

It defines:

- **Personal data** as any information that can identify someone, directly or indirectly.
- **Sensitive data** as information about health, biometrics, religion, political beliefs, or ethnicity.

So if your systems store or process user data from Brazil—even if your servers sit halfway across the world—you fall under LGPD.
And because the database holds the most sensitive data, it’s the first place you need strong controls.

![](/content/blog/database-compliance-for-lgpd/lgpd-persona.webp)

Under LGPD, **the same three roles from GDPR apply**:

- The **Data Subject (Titular)** is the individual whose personal data is collected.
- The **Controller (Controlador)** decides how and why data is processed.
- The **Processor (Operador)** handles data on behalf of the controller.  
  Both must designate a **Data Protection Officer (Encarregado)** as the contact point with Brazil’s ANPD and data subjects.

## LGPD Principles Meet Database Reality

Article 6 of LGPD lists ten principles for handling personal data. Many of them map directly to how databases should be managed.

| LGPD Principle                                               | What It Means for Databases                             | How Bytebase Helps                                                                                                                                                          |
| ------------------------------------------------------------ | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Purpose** – Use data only for legitimate, specific reasons | Every schema or data change should have a clear purpose | [Issue-based workflow](https://docs.bytebase.com/change-database/change-workflow) tracks who changed what, and why                                                          |
| **Security** – Prevent unauthorized access                   | Control who can access which databases                  | [RBAC](https://docs.bytebase.com/security/database-permission/overview) and [approval policies](https://docs.bytebase.com/change-database/approval) enforce least privilege |
| **Accountability** – Prove you’re following the rules        | Show evidence of controls                               | [Review workflow](https://docs.bytebase.com/change-database/review) and [immutable audit log](https://docs.bytebase.com/security/audit-log) provide an auditable trail      |
| **Necessity** – Limit processing to what’s essential         | Avoid querying or exposing extra data                   | [Dynamic data masking](https://docs.bytebase.com/security/data-masking/overview) hides sensitive fields in query results                                                    |

Instead of leaving these principles to policy documents, Bytebase brings them into your database workflow.

## Processing Data on a Legal Basis

LGPD says you can only process personal data if there’s a valid reason—such as user consent, legal obligation, or legitimate interest.

In practice, that means you need to know **why** each piece of data exists and be able to remove it when that reason no longer applies.

Bytebase makes this easier with **issue-based workflow** and **audit log**, so every change is reviewed, recorded, and tied to a legitimate purpose. If you ever need to prove compliance, you already have the evidence.

## Handling Sensitive and Anonymized Data

Sensitive data—health, biometric, or belief-related information—gets special protection under Article 11.
At the same time, LGPD encourages anonymization whenever possible.

Here’s how Bytebase helps with both:

- **Access control (RBAC)** limits who can see or modify sensitive tables.
- **Dynamic masking** hides real values in query results, perfect for lower environments.
- **Review worfklow** ensures all schema and permission changes are peer-reviewed before rollout.

It’s how you turn “we should protect sensitive data” into “we actually do.”

## Honoring Data Subject Rights

LGPD gives individuals rights to access, correct, delete, and even port their data. That sounds straightforward—until you realize it often involves multiple databases and dozens of tables.

Bytebase helps you act responsibly without chaos:

- The **database catalog** gives you visibility into where personal data lives.
- The **SQL Editor** lets you run controlled operations—like a targeted deletion—under proper review and audit.

So when users exercise their rights, you can respond accurately and securely.

## Being Ready for Incidents

If a security incident exposes personal data, LGPD requires companies to notify both the ANPD and the affected users. That’s not the time to start figuring out who had access or what changed.

Bytebase helps you prepare:

- Every SQL execution is **logged and attributed**.
- **Environment segregation** keeps production data isolated from test or staging.
- **Immutable records** support clear post-incident investigation.

With those in place, you’re not just reacting—you’re ready.

## Proving and Maintaining Compliance

LGPD’s final principle is accountability: it’s not enough to comply—you have to prove it.

With Bytebase, every schema update, permission change, and data fix goes through an **approval workflow**.
All of it is **traceable**, **reviewable**, and **exportable** for audits.
And because policies are enforced consistently across environments, compliance isn’t a one-off effort—it’s continuous.

## Closing Thoughts

Compliance with LGPD starts where your data lives. By embedding review, access control, and auditability directly into your database development process, Bytebase helps teams turn complex legal requirements into everyday best practices.

In short: **LGPD compliance isn’t a separate project—it’s how you work.** And Bytebase makes that possible.


A comprehensive overview of the essential practices and principles necessary for achieving LGPD (Brazilian General Personal Data Protection Act), compliance within database operations, and its difference with GDPR

Database Compliance for LGPD: Implications and Best Practices


A **LATERAL JOIN** allows a subquery in the `FROM` clause to reference columns from tables that appear earlier in the same `FROM` list.
PostgreSQL evaluates that subquery **once per row** of the left-side table, similar to a correlated subquery but written in `FROM`.

Use it when you need "for each row X, run this query that depends on X."

## Basic Syntax

The fundamental pattern looks like this:

```sql
FROM left_table lt
LEFT JOIN LATERAL (
  SELECT ...
  FROM ...
  WHERE ... lt.col ...
) s ON true
```

You can also write it as:

```sql
FROM lt, LATERAL (SELECT ...) s              -- CROSS JOIN LATERAL
FROM lt CROSS JOIN LATERAL (SELECT ...) s    -- explicit CROSS JOIN
```

The difference between `CROSS JOIN LATERAL` and `LEFT JOIN LATERAL` is crucial. `CROSS JOIN LATERAL` drops left rows when the subquery returns no rows, while `LEFT JOIN LATERAL` keeps all left rows and fills the right side with NULLs when there's no match.

## Why is it Called "Lateral"?

The term comes from Latin *lateralis*, meaning "of or pertaining to the side." In SQL, it refers to the subquery's ability to look **sideways** (laterally) at columns from tables that appear to its **left** in the `FROM` clause.

In the SQL standard, a *lateral derived table* is a subquery in `FROM` that can refer to columns from items that appear earlier in `FROM`. “Lateral” means the dependency runs sideways to the left, not downward like a nested subquery. So `LATERAL` just signals "this subquery may use columns from the tables before it."

## Core Example: Top-N Per Group

The classic use case is finding the top N records for each group. Suppose you want the most recent order for each customer:

```sql
SELECT c.id, c.name, o.order_id, o.placed_at, o.total
FROM customers c
LEFT JOIN LATERAL (
  SELECT *
  FROM orders o
  WHERE o.customer_id = c.id
  ORDER BY o.placed_at DESC
  LIMIT 1
) o ON true;
```

This is elegant and efficient. The `LIMIT 1` runs **per customer**, not globally. With a proper index on `(customer_id, placed_at DESC)`, PostgreSQL can grab the most recent order for each customer with minimal work. Compare this to window functions or self-joins, which often require more complex logic and can be slower.

## Common Use Cases

**Expanding JSON arrays per row.** When you have JSON data stored in a column and need to expand it into multiple rows:

```sql
SELECT p.id,
       item->>'sku' AS sku,
       (item->>'qty')::int AS qty
FROM purchase_requests p
LEFT JOIN LATERAL jsonb_array_elements(p.items_json) AS item ON true;
```

Each row's `items_json` array becomes multiple rows, one per element.

**Set-returning functions.** Functions that return multiple rows can be used with `LATERAL` (though the keyword is optional for functions):

```sql
SELECT u.id, g
FROM users u
JOIN generate_series(1, u.max_groups) AS g ON true;
```

The function `generate_series` can reference `u.max_groups` from the preceding table.

**Conversion funnel analysis.** Tracking user journeys through sequential events where each step depends on the previous one. For example, finding users who viewed a homepage, then used a demo within a week, then entered credit card details within another week. Each `LATERAL` join represents a funnel step that can reference timestamps and conditions from prior steps.

## Performance Considerations

LATERAL joins are implemented as nested loop joins. This means for N rows on the left and M rows per subquery, the complexity is O(N×M). However, with proper indexing, this can be very efficient:

| Optimization | Impact |
|:-------------|:-------|
| **Index on filtered columns** | Critical. For the "most recent order" example, an index on `(customer_id, placed_at DESC)` allows PostgreSQL to use an index scan per customer. |
| **LIMIT in subquery** | Drastically reduces work per left row by stopping early. |
| **LEFT vs CROSS JOIN** | Use `LEFT JOIN LATERAL` when you want to keep all left rows; use `CROSS JOIN LATERAL` when you only want rows with matches. |

Always use `EXPLAIN (ANALYZE, BUFFERS)` to verify your query plan and ensure indexes are being used effectively.

## Gotchas

The lateral subquery can only reference **preceding** `FROM` items. Order matters. If you write `FROM a, LATERAL (SELECT ...) s, b`, the subquery can see `a` but not `b`.

`CROSS JOIN LATERAL` drops left rows with no match. If you want to keep all left rows (like a standard `LEFT JOIN`), use `LEFT JOIN LATERAL ... ON true`.

It's easy to accidentally multiply rows. If you only need one row per left row, make sure to use `ORDER BY ... LIMIT 1` in the subquery.

## Quick Reference Patterns

**Top N items per parent:**

```sql
SELECT p.id, i.*
FROM projects p
LEFT JOIN LATERAL (
  SELECT *
  FROM issues i
  WHERE i.project_id = p.id
  ORDER BY i.priority DESC, i.created_at DESC
  LIMIT 3
) i ON true;
```

**Tokenize text per row:**

```sql
SELECT d.id, t.token
FROM docs d
LEFT JOIN LATERAL regexp_split_to_table(d.body, '\s+') AS t(token) ON true;
```

**Per-row function with parameters:**

```sql
SELECT a.id, nearby.id AS nearby_id, nearby.distance
FROM addresses a
LEFT JOIN LATERAL (
  SELECT b.id, st_distance(a.geom, b.geom) AS distance
  FROM addresses b
  WHERE b.city = a.city
  ORDER BY a.geom <-> b.geom
  LIMIT 1
) nearby ON true;
```

## Conclusion

LATERAL joins are a powerful tool for solving problems that would otherwise require complex window functions, procedural code, or inefficient self-joins. They shine in top-N-per-group scenarios, working with set-returning functions, and any situation where you need to run a parameterized query for each row of a table. Master this feature, and you'll write cleaner, faster queries.

Explain what is Postgres LATERAL JOIN and how to use it.

What is Postgres LATERAL JOIN?


Database schemas evolve constantly as applications grow and requirements change. Traditional schema modifications require downtime, disrupting user experience and causing revenue loss. MySQL's Online DDL solves this problem by allowing schema changes while the database remains operational.

## Three Algorithms: INSTANT, INPLACE, and COPY

MySQL uses three algorithms for schema changes, each with different performance characteristics and operation support.

### INSTANT: The Fastest Option

**INSTANT** applies changes instantly by modifying only table metadata. It requires no data copying, minimal resources, and works perfectly with replication. However, it supports only specific operations like adding columns with default values or modifying enums.

### INPLACE: The Balanced Approach

**INPLACE** modifies tables directly without creating temporary copies. It allows concurrent DML operations (INSERT, UPDATE, DELETE) during execution and supports more operations than INSTANT, including index creation and column drops. The trade-off is higher resource consumption.

### COPY: The Traditional Method

**COPY** creates a new table, copies all data, then swaps tables. It's the slowest method but supports all schema changes, including data type modifications. Use only when INSTANT and INPLACE aren't available.

## Operation Support Matrix

| Operation                  | INSTANT | INPLACE | COPY |
| :------------------------- | :-----: | :-----: | :--: |
| **Adding a Column**        |  ✅ \*  |   ✅    |  ✅  |
| **Dropping a Column**      |  ✅ \*  |   ✅    |  ✅  |
| **Renaming a Column**      |  ✅ \*  |   ✅    |  ✅  |
| **Changing Data Type**     |   ❌    |   ❌    |  ✅  |
| **Adding Secondary Index** |   ❌    |   ✅    |  ✅  |
| **Dropping Index**         |   ❌    |   ✅    |  ✅  |
| **Adding Primary Key**     |   ❌    |  ✅ \*  |  ✅  |
| **Dropping Primary Key**   |   ❌    |   ❌    |  ✅  |
| **Adding Foreign Key**     |   ❌    |   ✅    |  ✅  |

_Conditions apply. See [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) for details._

## Key Limitations

**Resource Impact**: INPLACE operations consume significant CPU, memory, and I/O, potentially affecting database performance on busy servers.

**Replication Lag**: The biggest limitation is replication behavior. An INPLACE operation taking 3 hours on the primary will stall replication for 3 hours on each replica, creating massive lag.

**Locking Issues**: Despite being "online", metadata locks can still block other DDL operations and some DML statements.

**Limited INSTANT Support**: Most complex operations like data type changes, index additions, and primary key modifications aren't supported by INSTANT.

## Best Practices

**Choose Wisely**: Use INSTANT when possible, INPLACE when necessary, and COPY as a last resort.

**Test First**: Always test schema changes in staging environments that mirror production.

**Monitor Performance**: Watch CPU, memory, and I/O usage during operations.

**Consider Third-Party Tools**: For complex migrations or strict availability requirements, tools like `gh-ost` or `pt-online-schema-change` offer better control and replication handling.

**Plan for Replication**: In replicated environments, consider running changes individually on each server or using external tools to avoid lag.

## Conclusion

MySQL Online DDL provides valuable capabilities for schema evolution, but understanding its limitations is crucial. INSTANT operations are ideal when supported, while INPLACE operations require careful resource planning. For mission-critical environments, combining native features with third-party tools often provides the best balance of safety and efficiency.


Introduction to MySQL Online DDL and its three algorithms: INSTANT, INPLACE, and COPY

MySQL Online DDL: A Practical Guide


Postgres is famously robust, but its connection model isn't. Each client connection maps to a dedicated backend process with non-trivial memory and CPU overhead. Spinning up and tearing down connections is expensive, and even "idle" sessions consume resources. Choosing a safe `max_connections` is more art than science: too low and your app queues; too high and the database wastes RAM and scheduler time.

A connection pooler sits between apps and Postgres to reuse a small number of server connections across many clients, smoothing spikes and protecting the database.

Below is a quick, practical tour of the leading **open-source** options.

---

## PgBouncer (the standard)

https://github.com/pgbouncer/pgbouncer

**What it is.** A lightweight, single-binary connection pooler that's been battle-tested for years. It supports **session**, **transaction**, and **statement** pooling modes, each with trade-offs. Recent releases added quality-of-life improvements like **per-user/per-database connection tracking/limits** and better TLS reload behavior; prepared-statement support is now on by default.

**Why it's popular:**

- Minimal footprint and easy to deploy nearly anywhere.
- Works with managed Postgres (RDS, AlloyDB, etc.) and on-prem clusters.
- Integrated by many operators (e.g., CloudNativePG offers a `Pooler` CRD).

**Good fit for:** most apps needing stable concurrency control and simple ops.

## Cloudflare's fork of PgBouncer (cf-pgbouncer)

https://github.com/cloudflare/cf-pgbouncer

**What it is.** Cloudflare open-sourced their internal fork to harden multi-tenant ops. It includes **auth bug fixes** and features to enforce **per-user and per-pool isolation**, addressing cases where upstream behavior with HBA auth limited those controls. The fork targets large-scale, multi-tenant environments where noisy-neighbor isolation matters.

**Good fit for:** high-scale providers and anyone needing stricter tenant isolation/concurrency enforcement at the pooler layer. For deeper context on CF's multi-tenant performance isolation goals, see their companion post.

## Supavisor (Supabase)

https://github.com/supabase/supavisor

**What it is.** A **cloud-native** pooler built by Supabase (written in Elixir) that emphasizes horizontal scalability and multi-project tenancy. Supabase demonstrated **handling ~1 million concurrent client connections** in testing, highlighting an architecture designed for massive fan-out with lightweight client slots mapped onto a smaller set of server connections. Supavisor now ships as their default pooler.

**Good fit for:** SaaS platforms or services expecting very high connection counts (IoT, event streams, lots of short-lived clients) and wanting modern autoscaling patterns.

## PgDog

https://github.com/pgdogdev/pgdog

**What it is.** A newer **Rust** project that combines **connection pooling**, **load balancing**, and **sharding** in one layer. It supports **session** and **transaction** pooling "like PgBouncer", and advertises the ability to fan out to **hundreds of thousands of clients** while also offering horizontal scale primitives. Docs include admin views (e.g., `SHOW POOLS`) and telemetry.

**Good fit for:** teams wanting a single, high-performance proxy that also tackles routing and (eventually) scale-out patterns beyond pooling.

---

## How to choose (and what to watch for)

**1) Pick the pooling mode wisely**

- **Session pooling:** 1 client ↔ 1 server connection for the session's life. Maximum compatibility (works with temp tables, GUCs, prepared statements) but fewer concurrency gains.
- **Transaction pooling:** server connections are returned to the pool after each transaction - **huge concurrency** boost, but features that rely on session state can break (many ORMs are fine if you avoid session-scoped behaviors).
- **Statement pooling:** most aggressive; rarely needed outside special cases.

**2) Enforce limits at the pooler**
Use per-user/per-database caps to protect Postgres from thundering herds and to keep tenants in bounds. Upstream PgBouncer and Cloudflare's fork provide controls here.

**3) Mind prepared statements & features**
Prepared statements historically clashed with transaction pooling; newer PgBouncer releases improved behavior and defaults, but test your ORM/framework.

**4) Deploy for HA like any stateless proxy**
Run multiple poolers behind a VIP/Load Balancer; configure health checks and drain/reload flows (especially with TLS). Cloud providers and operators (e.g., CloudNativePG) can simplify this.

**5) Tune the basics**
Right-size `server_pool_size`, `max_client_conn`, and timeouts based on your workload's **concurrency** and **transaction duration**. Heroku's PgBouncer guidance gives a clear mental model (pools are per user/db/host tuple).

## Quick comparison

| Project          | Headline strengths                           | Maturity         | Implementation language | Notes                                                      |
| ---------------- | -------------------------------------------- | ---------------- | -------------- | ---------------------------------------------------------- |
| **PgBouncer**    | Lightweight, ubiquitous, three pooling modes | Very mature      | **C**          | Great default choice; wide community/packager support.     |
| **cf-pgbouncer** | Multi-tenant isolation; auth fixes           | Mature fork      | **C**          | Useful when you need stricter per-user/pool controls.      |
| **Supavisor**    | Cloud-native scale; demonstrated ~1M clients | Young → maturing | **Elixir**     | Optimized for massive fan-out and provider use cases.      |
| **PgDog**        | Pooler + load balancer + sharder (Rust)      | Emerging         | **Rust**       | Ambitious all-in-one proxy with horizontal scale features. |

---

## Bottom line

- **Start with PgBouncer** for most apps; it's simple, well-documented, and production-proven.
- If you're a **multi-tenant platform** and need hard per-user/pool isolation, evaluate **Cloudflare's fork**.
- If your challenge is **sheer connection volume** (hundreds of thousands to millions), **Supavisor** is built for that world.
- If you also want **routing/sharding** alongside pooling, keep an eye on **PgDog**.

Whichever you choose, treat the pooler as part of your **capacity and SLO strategy**: cap concurrency, keep transactions short, and monitor pool saturation to keep Postgres fast and happy.

List of open source postgres connection poolers

Open Source Postgres Connection Pooler


**TL;DR:** Pick one style, document it, and stick to it. If you’re undecided, **plural tables** (`users`, `orders`) are a safe default: they read naturally in SQL and dodge some reserved words. Consistency matters more than the “right” choice.
## Why it matters

Names ripple through SQL, ORMs, migrations, BI, and docs. A consistent convention lowers cognitive load and onboarding time.

## The two camps

### Singular (`user`, `order_item`)

**Pros**

* **OOP alignment:** maps cleanly to classes/entities (a row is a `User`).
* **Language simplicity:** avoids irregular plurals for international teams.
* **Master–detail reads cleanly:** `order` ↔ `order_detail` can feel natural.

**Cons**

* **Reserved words:** `user`, `order`, `group`, `session` can clash.
* **Query feel:** `SELECT * FROM user` reads a bit stilted.

### Plural (`users`, `order_items`)

**Pros**

* **Natural language in SQL:** `SELECT * FROM users WHERE age > 21`.
* **Reserved-word avoidance:** `orders` > `order`.
* **Framework affinity:** popular in Rails and many modern stacks.

**Cons**

* **Semantic mismatch:** a row is a user but lives in `users`.
* **Irregulars:** `person/people`, `child/children` add edge cases.

## Industry split (rule of thumb)

| Approach | Common In                        | Key Advantage                   |
| -------- | -------------------------------- | ------------------------------- |
| Singular | Enterprise / traditional systems | OOP consistency; entity focus   |
| Plural   | Web frameworks / modern stacks   | Readability; framework defaults |

> There’s no universal standard; both conventions are widespread. What matters is a deliberate choice and consistency.

## Quick decision guide

* **Greenfield & undecided:** choose **plural tables**.
* **Existing codebase:** **match what’s there**—consistency beats preference.
* **Reserved-word risk:** prefer **plural** (`orders`, `groups`).
* **Strict DDD shops / heavy OOP mapping:** singular can fit better.

## Practical conventions (copy-paste)

1. **Tables:** plural, `snake_case` → `users`, `orders`, `order_items`, `audit_logs`.
2. **Join tables:** plural + plural, alphabetical → `orders_products`, `roles_users`.
3. **Columns:** singular, `snake_case` → `id`, `user_id`, `created_at`, `updated_at`.
4. **Views / MVs:** prefix + plural → `v_active_users`, `mv_daily_signups`.
5. **Reference tables:** plural → `countries`, `currencies`, `order_statuses`.
6. **Irregular nouns:** standardize once (e.g., always `people` *or* `persons`) and stick to it.

## Master–detail naming tips

* **Singular world:** `order` / `order_detail` is tidy and readable.
* **Plural world (most common):** `orders` / `order_items` keeps the collection metaphor; avoid `orders_details`.
  Use the object’s name (`order_items`) rather than “detail(s)”.

## Governance (make it stick)

* Write the rule in your engineering handbook.
* Add a schema linter/check in CI to block drift.
* Provide examples for tricky cases (irregular plurals, join tables, reserved words).

## Examples

**Plural (recommended default)**

```sql
CREATE TABLE users (
  id BIGSERIAL PRIMARY KEY,
  email TEXT UNIQUE NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE orders (
  id BIGSERIAL PRIMARY KEY,
  user_id BIGINT NOT NULL REFERENCES users(id),
  status TEXT NOT NULL,
  total_cents INTEGER NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE order_items (
  order_id BIGINT NOT NULL REFERENCES orders(id),
  product_id BIGINT NOT NULL REFERENCES products(id),
  quantity INTEGER NOT NULL DEFAULT 1,
  PRIMARY KEY (order_id, product_id)
);
```

**Singular (for strict entity alignment)**

```sql
CREATE TABLE user (
  id BIGSERIAL PRIMARY KEY,
  email TEXT UNIQUE NOT NULL
);

CREATE TABLE order (
  id BIGSERIAL PRIMARY KEY,
  user_id BIGINT NOT NULL REFERENCES user(id)
);
/* Beware: user/order can conflict with reserved words in some contexts */
```

## Final recommendation

Choose **one** convention, document it, enforce it. If you don’t have strong reasons, pick **plural tables + singular columns** with `snake_case`, and don’t revisit the debate in every PR. Consistency > perfection.

A comprehensive guide to understanding the dilemma of singular vs. plural table naming in SQL databases.

SQL Table Naming Dilemma: Singular vs. Plural


PostgreSQL 18 was officially released on [September 25, 2025](https://www.postgresql.org/about/news/postgresql-18-released-3142/). No doubt the most consequential change is the new Asynchronous I/O (AIO) subsystem. However, this post focuses on the features that will impact developers in their day-to-day work, starting with native UUID v7 support.

## Native UUID v7 Support

- Commit: [78c5e141e](https://postgr.es/c/78c5e141e)
- Docs: https://www.postgresql.org/docs/18/datatype-uuid.html

PostgreSQL 18 introduces the `uuidv7()` function for generating UUID version 7 values. This native support almost closes the long-standing debate about whether to use `SERIAL/IDENTITY` types or UUIDs as primary keys. UUIDv7 combines the best of both worlds: the global uniqueness of UUIDs with the sequential nature that makes serial keys perform well.

```sql
-- Generate a UUIDv7
SELECT uuidv7();

-- Create a table using UUIDv7 as primary key
CREATE TABLE orders (
    id UUID PRIMARY KEY DEFAULT uuidv7(),
    customer_id INT,
    total DECIMAL(10,2),
    created_at TIMESTAMP DEFAULT NOW()
);
```

Previously, while PostgreSQL extensions could generate UUIDv7, the most widely used `uuid-ossp` extension doesn't implement v7. Lesser-known extensions that do support v7 aren't typically packaged with PostgreSQL distributions, especially on cloud providers. This meant developers wanting UUIDv7 had to implement generation logic in their applications. Native support removes this friction entirely.

## VIRTUAL Generated Columns

- Commit: [83ea6c540](https://postgr.es/c/83ea6c540)
- Docs: https://www.postgresql.org/docs/18/sql-createtable.html#SQL-CREATETABLE-PARMS-GENERATED-STORED

Generated columns were first introduced in PostgreSQL 12, but only with the `STORED` option, which means the generated column is stored on write and occupies disk space. Adding a `STORED` generated column will rewrite the entire table.

PostgreSQL 18 introduces the `VIRTUAL` option and makes it the default behavior for generated columns. These columns compute their values when read rather than when written. This is a more common approach, and making it the default aligns PostgreSQL with other major databases.

```sql
-- Create table with virtual generated column
CREATE TABLE employees (
    id SERIAL PRIMARY KEY,
    first_name TEXT NOT NULL,
    last_name TEXT NOT NULL,
    salary DECIMAL(10,2),
    -- You can also skip VIRTUAL since it's the default option
    annual_salary DECIMAL(12,2) GENERATED ALWAYS AS (salary * 12) VIRTUAL
);

-- Insert data (generated columns are computed automatically)
INSERT INTO employees (first_name, last_name, salary) VALUES
    ('John', 'Doe', 5000.00),
    ('Jane', 'Smith', 6500.00);

SELECT first_name, last_name, salary, annual_salary
FROM employees;
```

Virtual column advantages:

- **Space Saving**: For derived data that doesn't need physical storage, virtual columns eliminate redundant disk usage.

- **Dynamic Computation**: Some computations benefit from being executed at query time, especially when they depend on current system state.

On the other hand, `STORED` generated columns remain valuable for heavy computations that rarely change, where the performance cost of repeated calculation outweighs storage concerns.

However, there are certain limitations as mentioned in the commit. The most noticeable one is that you cannot create indexes on `VIRTUAL` generated columns.

BTW, PostgreSQL 18 also adds support for generated columns in logical replication ([commit](https://postgr.es/c/745217a05), [docs](https://www.postgresql.org/docs/18/logical-replication-gencols.html)). This feature is only available to `STORED` generated columns as well.

## OLD and NEW Values in RETURNING

- Commit: [80feb727c](https://postgr.es/c/80feb727c)
- Docs: https://www.postgresql.org/docs/current/dml-returning.html

PostgreSQL 18 enhances the `RETURNING` clause to access both old and new row values in DML operations. This eliminates the need for separate queries to capture before/after values in audit logs.

**UPDATE Operations**

```sql
-- Update prices and see both old and new values
UPDATE products
SET price = price * 1.10
WHERE price <= 99.99
RETURNING
    name,
    old.price AS old_price,
    new.price AS new_price,
    new.price - old.price AS price_change;
```

**INSERT with ON CONFLICT**

```sql
-- Upsert with change tracking
INSERT INTO products (name, price) VALUES ('Widget', 25.00)
ON CONFLICT (name) DO UPDATE SET price = EXCLUDED.price
RETURNING
    name,
    old.price AS previous_price,
    new.price AS current_price,
    (old.price IS NULL) AS is_new_record;
```

**DELETE Operations**

```sql
-- Track what was deleted
DELETE FROM products
WHERE price < 10.00
RETURNING
    old.name AS deleted_product,
    old.price AS deleted_price;
```

## EXPLAIN ANALYZE with BUFFERS by Default

- Commit: [c2a4078eb](https://postgr.es/c/c2a4078eb)
- Docs: https://www.postgresql.org/docs/current/sql-explain.html

PostgreSQL 18 improves the `EXPLAIN` command to provide more detailed query execution information automatically.

`EXPLAIN ANALYZE` now includes buffer usage information by default, eliminating the need for the `BUFFERS` option. This default behavior helps developers spot I/O performance issues that would otherwise be overlooked:

```sql
postgres=# EXPLAIN ANALYZE SELECT * FROM users WHERE age > 25;

-- PostgreSQL 18: BUFFERS included automatically
 Seq Scan on users  (cost=0.00..18.75 rows=125 width=64) (actual time=0.029..0.087 rows=178 loops=1)
   Filter: (age > 25)
   Rows Removed by Filter: 89
   Buffers: shared hit=12
 Planning:
   Buffers: shared hit=156 read=3
   I/O Timings: shared read=0.024
 Planning Time: 0.512 ms
 Execution Time: 0.734 ms
```

## pg_get_acl() for Permission Inspection

- Commit: [4564f1ceb](https://postgr.es/c/4564f1ceb)
- Docs: https://www.postgresql.org/docs/18/functions-info.html#FUNCTIONS-INFO-OBJECT-TABLE

PostgreSQL 18 introduces the `pg_get_acl()` function for programmatically retrieving Access Control Lists (ACLs) for database objects. If you've ever spent time debugging the infamous `ERROR 42501: permission denied` messages, you'll appreciate having a unified way to inspect object privileges.

Previously, troubleshooting permissions required querying different system catalogs (`pg_class`, `pg_proc`, `pg_namespace`, `pg_attribute`) depending on the object type, each with their own ACL format.

The `pg_get_acl()` function provides a unified interface for retrieving ACLs from any database object, eliminating the need to remember which catalog to query for different object types.

```sql
postgres=# SELECT
    (pg_identify_object(s.classid,s.objid,s.objsubid)).*,
    pg_catalog.pg_get_acl(s.classid,s.objid,s.objsubid) AS acl
FROM pg_catalog.pg_shdepend AS s
JOIN pg_catalog.pg_database AS d
    ON d.datname = current_database() AND
       d.oid = s.dbid
JOIN pg_catalog.pg_authid AS a
    ON a.oid = s.refobjid AND
       s.refclassid = 'pg_authid'::regclass
WHERE s.deptype = 'a';
-[ RECORD 1 ]-----------------------------------------
type     | table
schema   | public
name     | testtab
identity | public.testtab
acl      | {postgres=arwdDxtm/postgres,foo=r/postgres}
```

While not as headline-grabbing as the new AIO, these quality-of-life improvements make day-to-day development easier—and clearer interfaces like `pg_get_acl()` benefit not only human developers, but AI agents as well.

## Further Readings

- [Postgres 18 full release notes](https://www.postgresql.org/docs/18/release-18.html)
- [DBA's perspective about Postgres 18](/blog/what-is-new-in-postgres-18)


Overview of PostgreSQL 18 features from a developer's perspective

What's New in PostgreSQL 18 - a Developer's Perspective


## Snowflake Dynamic Data Masking

Snowflake Dynamic Data Masking (DDM) is a security feature that allows you to mask sensitive data in real-time without creating separate masked copies of your data. It enables organizations to protect personally identifiable information (PII) and other sensitive data while maintaining data utility for authorized users.

Dynamic data masking works by applying masking policies at the column level, transforming data on-the-fly based on the user's role and permissions. This approach ensures that sensitive data remains protected while allowing different levels of data access for different human users and services.

Here's how to create and apply a simple masking policy in Snowflake:

```sql
-- Create a masking policy for email addresses
CREATE OR REPLACE MASKING POLICY email_mask AS (val STRING)
RETURNS STRING ->
  CASE
    WHEN CURRENT_ROLE() IN ('ADMIN', 'DATA_ANALYST') THEN val
    ELSE REGEXP_REPLACE(val, '.+@', '*****@')
  END;

-- Apply the masking policy to a column
ALTER TABLE customers
MODIFY COLUMN email SET MASKING POLICY email_mask;
```

## Limitations and Challenges

While Snowflake Dynamic Data Masking provides powerful security capabilities, there are several important considerations and limitations to be aware of:

### 1. Enterprise Edition Requirement and Cost Impact

Snowflake DDM, which is part of Column-level security, is only available in the **Enterprise edition and higher**, which significantly impacts pricing:

| Edition           | Price per Credit | Cost Increase | DDM Available |
| ----------------- | ---------------- | ------------- | ------------- |
| Standard          | $2               | -             | ❌            |
| Enterprise        | $3               | +50%          | ✅            |
| Business Critical | $4               | +100%         | ✅            |

For organizations currently on Standard edition, enabling DDM requires upgrading to Enterprise, resulting in a **50% increase in compute costs** across all workloads. This cost overhead applies to your entire Snowflake usage, not just masked data operations.

### 2. Policy Management Complexity

Managing Snowflake DDM at scale presents significant operational challenges due to policy proliferation across sensitive columns, lack of proper processes and audit trails to track policy changes, and the absence of Snowsight UI support - forcing teams to manage complex masking policies entirely through SQL commands, which becomes increasingly difficult to maintain and govern as the number of policies grows.

```sql
-- Example of policy complexity with multiple roles and conditions
CREATE OR REPLACE MASKING POLICY customer_pii_mask AS (val STRING)
RETURNS STRING ->
  CASE
    WHEN CURRENT_ROLE() = 'DATA_OWNER' THEN val
    WHEN CURRENT_ROLE() = 'ANALYST_SENIOR' AND
         CURRENT_WAREHOUSE() = 'ANALYTICS_WH' THEN val
    WHEN CURRENT_ROLE() = 'CUSTOMER_SERVICE' AND
         CURRENT_TIME() BETWEEN '09:00'::TIME AND '17:00'::TIME THEN
         CONCAT(LEFT(val, 3), '***', RIGHT(val, 2))
    WHEN CURRENT_ROLE() IN ('ANALYST_JUNIOR', 'INTERN') THEN '***MASKED***'
    ELSE NULL
  END;
```

## Alternatives to Snowflake Dynamic Data Masking

Given the limitations and costs associated with Snowflake DDM, organizations often explore alternative approaches to protect sensitive data:

### 1. Database Views

Database views provide a cost-effective way to implement data masking without requiring Enterprise edition. Views can incorporate role-based logic and masking functions to protect sensitive data at the query level.

```sql
-- Create a masked view for customer data
CREATE OR REPLACE VIEW customers_masked AS
SELECT
    customer_id,
    customer_name,
    CASE
        WHEN CURRENT_ROLE() IN ('ADMIN', 'DATA_ANALYST')
        THEN email
        ELSE REGEXP_REPLACE(email, '.+@', '*****@')
    END AS email,
    CASE
        WHEN CURRENT_ROLE() = 'FINANCE_ADMIN'
        THEN credit_card_number
        WHEN CURRENT_ROLE() = 'CUSTOMER_SERVICE'
        THEN CONCAT('****-****-****-', RIGHT(credit_card_number, 4))
        ELSE '****-****-****-****'
    END AS credit_card_number,
    registration_date
FROM customers;
```

- **Pros**: No additional license required as it works with all Snowflake editions.
- **Cons**: Even more complex to manage than masking policies due to view proliferation and lack of enforcement.

### 2. Bytebase

[Bytebase](https://docs.bytebase.com/security/data-masking/overview) is a database DevSecOps platform that provides dynamic data masking capabilities across multiple database systems, including Snowflake.

#### How Bytebase DDM Works

**Middleware Architecture**

Unlike Snowflake's native DDM, Bytebase doesn't rely on Snowflake's data masking features and operates as a middleware layer between users and Snowflake, intercepting all database queries and applying masking rules before returning results to the user.

![masking-overview](/content/blog/snowflake-dynamic-data-masking-and-alternatives/masking-overview.webp)

**Policy Configuration**

Users configure masking policies through Bytebase's web UI for intuitive management, or programmatically via Terraform Provider and REST API for automation and infrastructure-as-code workflows.

![masking-configuration](/content/blog/snowflake-dynamic-data-masking-and-alternatives/masking-configuration.webp)

**Query Execution**

When users query data through Bytebase's SQL Editor, the platform automatically applies the configured masking policies in real-time, ensuring sensitive data is protected without requiring changes to the underlying database structure.

![sql-editor](/content/blog/snowflake-dynamic-data-masking-and-alternatives/sql-editor-masking.webp)

**Pros:**

- Fractional cost at typically 10% of upgrading to Snowflake Enterprise edition.
- UI-based masking policy configuration and grant masking exemptions management, with support for Terraform Provider and API integration.
- Provides the same workflow to configure masking policies across different database systems beyond just Snowflake.

**Cons:**

- Masking is only enforced when users query data through Bytebase's SQL Editor.
- Only covers human-to-database path and does not enforce service-to-database connections.

## Comparison

| Solution       | Cost                            | Operational Complexity         | Human Access | Service Access |
| -------------- | ------------------------------- | ------------------------------ | ------------ | -------------- |
| Snowflake DDM  | High (+50% for Enterprise)      | High (SQL-only, no UI)         | Enforced     | Enforced       |
| Database Views | None                            | Very High (View proliferation) | Enforced     | Enforced       |
| Bytebase       | Low (10% of Enterprise edition) | Medium (UI + API/Terraform)    | Enforced     | Not enforced   |

While Snowflake's native DDM provides the most comprehensive coverage, organizations looking to balance cost, manageability, and protection for human access patterns may find Bytebase offers the best compromise for their data masking needs.


Explore Snowflake Dynamic Data Masking capabilities and compare with alternative data masking solutions

Snowflake Dynamic Data Masking (DDM) and Alternatives


**Postgres has become the go-to database for modern apps. But authentication (who a user is) and authorization (what they can do) are just as critical.**

In 2025, you have plenty of open-source options. To make it simple, here are the most relevant solutions, grouped by type, with clear "best for" recommendations.

## 1. Platforms (Postgres + Auth bundled)

These give you Postgres, authentication, and APIs in one package. Ideal if you want **RLS (Row Level Security) to work out of the box**.

### 🔹 [Supabase Auth](https://supabase.com/docs/guides/auth)

![Supabase Auth](/content/blog/top-open-source-postgres-auth-solution/supabase-auth.webp)

- **Features:** Email/password, magic links, OAuth, phone, Web3 logins.
- **Integration:** Deep RLS support (`auth.uid()`, `auth.jwt()` directly usable in policies).
- **Third-party support:** Can trust IdPs like Clerk, Firebase Auth, Cognito, WorkOS, etc. but only if they issue **asymmetric JWTs**. Key rotation can lag \~30 min. Supabase Auth itself can’t be disabled.
- **Best for:** Startups and teams that want the **fastest path** to secure Postgres apps.

### 🔹 [Nhost](https://nhost.io/product/auth) (Hasura-based)

![Nhost](/content/blog/top-open-source-postgres-auth-solution/nhost-auth.webp)

- **Features:** Postgres + Hasura GraphQL API + Auth.
- **Integration:** Auth ties into Hasura permissions, which map back to Postgres RLS.
- **Best for:** Teams building **GraphQL-first apps** who want a fully open-source stack.

## 2. Libraries (you own the server)

These plug into your backend, store users in Postgres, and issue JWTs. You stay in control.

### 🔹 [Auth.js](https://authjs.dev/)

![Auth.js](/content/blog/top-open-source-postgres-auth-solution/authjs.webp)

- **Features:** 50+ OAuth providers, session handling, JWT support.
- **Integration:** Postgres adapter for users and sessions.
- **Best for:** Apps with custom backends, especially **Next.js** or full-stack JS.

### 🔹 [Better Auth](https://better-auth.com/)

![better-auth](/content/blog/top-open-source-postgres-auth-solution/better-auth.webp)

- **Features:** TypeScript-first, supports multi-tenancy, 2FA, org management.
- **Integration:** Native Postgres support with Kysely/Drizzle + schema migration tooling.
- **Best for:** **TypeScript-heavy teams** who want modern DX and self-hosted control.

### 🔹 [Lucia](https://lucia-auth.com/) (maintenance mode)

![lucia](/content/blog/top-open-source-postgres-auth-solution/lucia-auth.webp)

- **Features:** Educational focus, lightweight packages.
- **Status:** v3 is deprecated, supported only until March 2025.
- **Best for:** Existing projects. Not recommended for new ones.

## 3. Identity Servers (standalone IdP)

These are separate services that act as the **source of truth for identity**. They issue JWTs for your apps.

### 🔹 [Ory Kratos](https://www.ory.sh/kratos/)

- **Features:** Registration, recovery, passwordless login, customizable flows.
- **Integration:** Uses Postgres as identity store; issues JWTs consumed by RLS.
- **Best for:** Centralized identity across multiple services.

### 🔹 [Keycloak](https://www.keycloak.org/)

- **Features:** Enterprise-grade IdP with OIDC, SAML, LDAP, and multi-realm/org support.
- **Integration:** Runs on Postgres and issues JWTs for your apps.
- **Best for:** Large orgs needing **enterprise SSO** and federation features.

### 🔹 [ZITADEL](https://zitadel.com/)

- **Features:** Modern IdP with org/project/role management.
- **Integration:** Postgres/Cockroach backend; OIDC → Postgres RLS.
- **Best for:** Cloud-native teams who want a modern, OSS alternative to commercial IdPs.

## 4. Other OSS Options

### 🔹 [SuperTokens](https://supertokens.com/)

- **Features:** Recipes for email/password, social login, passwordless, session management.
- **Integration:** Native Postgres support; cloud or self-host.
- **Best for:** Developers who want **prebuilt flows** but remain OSS-first.

## 5. [Postgres-Native Pattern](/blog/postgres-native-pattern/) (minimalist)

You can also skip heavy auth systems:

- Issue JWTs (from a small service or IdP).
- Validate them at the API edge (PostgREST, Supabase, or a proxy).
- Let **RLS** enforce access inside Postgres.

**Best for:** Small-to-mid apps where you want **maximum simplicity** and DB-driven auth.


## Comparison Table

| Solution                    | Type                 | Hosting           | Postgres Integration       | Learning Curve | Best for                      |
| --------------------------- | -------------------- | ----------------- | -------------------------- | -------------- | ----------------------------- |
| Supabase Auth               | Platform (BaaS)      | Cloud / Self-host | Native (RLS, JWT helpers)  | Low            | Startups, all-in-one apps     |
| Nhost                       | Platform (GraphQL)   | Cloud / Self-host | Hasura + RLS               | Medium         | GraphQL-first teams           |
| Auth.js                     | Library              | Self-host         | Postgres adapter           | Medium         | Flexible, multi-provider apps |
| Better Auth                 | Library              | Self-host         | Native schema + migrations | Medium         | TypeScript-first projects     |
| Lucia (v3)                  | Library (deprecated) | Self-host         | Postgres adapter           | High           | Legacy projects only          |
| Ory Kratos                  | Identity server      | Self-host         | Postgres identity store    | High           | Multi-app identity            |
| Keycloak                    | Identity server      | Self-host         | Native Postgres            | High           | Enterprise SSO                |
| ZITADEL                     | Identity server      | Self-host / Cloud | Postgres/Cockroach backend | Medium         | Cloud-native IdP              |
| SuperTokens                 | Library / Service    | Cloud / Self-host | Native Postgres            | Medium         | Prebuilt flows                |
| Postgres-native (JWT → RLS) | Pattern              | Self-host         | Direct via RLS claims      | Low            | Minimalist DB-driven          |

## Conclusion

- **Fastest startup path** → **Supabase Auth**
- **GraphQL-first stack** → **Nhost**
- **Custom backend** → **Auth.js** or **Better Auth**
- **Enterprise / multi-app identity** → **Keycloak, Ory Kratos, ZITADEL**
- **Minimalist & DB-driven** → Postgres-native JWT → RLS
- **Prebuilt recipes** → **SuperTokens**

No matter which tool you pick, the principle is the same: **JWT claims flow into Postgres RLS**, making the database itself the final gatekeeper.

Top open-source Postgres authentication solutions for 2025.

Postgres Case Sensitivity Explained


PostgreSQL 14 (released September 30, 2021) introduced a powerful predefined role: **`pg_database_owner`**. This role changes how database ownership and privileges are managed by providing a dynamic way to handle permissions for database owners.

## What is `pg_database_owner`?

`pg_database_owner` is a **built-in role** that automatically maps to the **owner of the current database**.

* You don’t grant it to users.
* Membership is **implicit and context-dependent** — whoever owns the database is the member.

This allows privileges granted to `pg_database_owner` in template databases to carry over automatically when new databases are created from them.

## Why it Exists

Two main problems it solves:

1. **Template Database Privileges**
   Before, ensuring consistent privileges across databases required manual grants. Now, you can grant to `pg_database_owner` in a template, and new DBs automatically apply them.

2. **Safer Defaults**
   Starting in PostgreSQL 15, the `public` schema is owned by `pg_database_owner`. This gives each DB owner direct control over schema access, fixing the long-standing risk of overly permissive defaults.

## Key Characteristics

| Property                           | Description                                    |
| ---------------------------------- | ---------------------------------------------- |
| **Implicit membership**            | Always maps to the database owner              |
| **Database scope**                 | Valid only in the current database             |
| **No login**                       | Cannot log in directly                         |
| **No explicit members**            | You cannot `GRANT ... TO pg_database_owner`    |
| **Template inheritance**           | Privileges granted in a template DB carry over |
| **Isolation**                      | Cannot be part of role hierarchies             |
| **Public schema ownership (v15+)** | Owns the `public` schema                       |

## Common Error

Trying to grant it explicitly produces an error:

```sql
GRANT pg_database_owner TO my_user;
ERROR: role "pg_database_owner" cannot have explicit members
```

✅ Correct usage:

* Grant privileges *to* `pg_database_owner`
* Or transfer DB ownership if you want a new effective owner

```sql
ALTER DATABASE mydb OWNER TO new_owner;
```

## Practical Examples

**Schema Privileges**

```sql
GRANT USAGE, CREATE ON SCHEMA app TO pg_database_owner;
ALTER DEFAULT PRIVILEGES IN SCHEMA app
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO pg_database_owner;
```

**Template Setup**
Grant monitoring or utility functions to `pg_database_owner` in a template DB so every new database owner gets access automatically.

## Best Practices

**Do:**

* Use it in template DBs for consistent privileges
* Grant privileges *to it*, not users
* Leverage it for schema or function access

**Don’t:**

* Try to grant it to users (will fail)
* Expect cross-database scope
* Assume it has privileges by default

## Conclusion

`pg_database_owner` is a small but important change: it gives PostgreSQL a clean, dynamic way to handle database ownership and privilege inheritance. Whether you’re building multi-tenant apps or just want safer defaults, it simplifies privilege management and improves security.

A comprehensive guide to understanding pg_database_owner in Postgres

What is pg_database_owner in Postgres


Postgres is a powerful and popular open-source object-relational database system. However, like any complex system, it has its intricacies. One of the most important concepts to understand for maintaining a healthy and performant Postgres database is the idea of a **table rewrite**.

When Postgres "rewrites" a table, it creates a brand-new copy of the relation (a new **relfilenode**) and swaps it in. This process often requires approximately 2× temporary disk space and usually holds strong locks that can block traffic. A handy mental model is: _if the on-disk row layout must change or the table has to be physically moved/reordered, expect a rewrite._

This article will explore which Postgres operations cause a table rewrite, which ones don't, and the locking implications of these operations.

## Which Operations Cause a Table Rewrite?

Here is a summary of common Postgres operations and whether they cause a table rewrite:

| Operation                                          | Rewrite?             | Lock Level             | Why?                                                                      |
| -------------------------------------------------- | -------------------- | ---------------------- | ------------------------------------------------------------------------- |
| `TRUNCATE`                                         | ✅ Yes               | ACCESS EXCLUSIVE       | Deletes table content by replacing its file.                              |
| `ALTER TABLE ... SET TABLESPACE`                   | ✅ Yes               | ACCESS EXCLUSIVE       | Moves data to new storage layout or tablespace.                           |
| `ALTER TABLE ... SET LOGGED/UNLOGGED`              | ✅ Yes               | ACCESS EXCLUSIVE       | Changing persistence forces a rewrite.                                    |
| `ALTER TABLE ... ALTER COLUMN TYPE` (incompatible) | ✅ Often             | ACCESS EXCLUSIVE       | Rewrite if type change isn't binary-compatible.                           |
| `ALTER TABLE ... ALTER COLUMN TYPE` (compatible)   | ❌ No                | ACCESS EXCLUSIVE       | Binary-compatible changes (e.g., varchar(50)→varchar(100)) avoid rewrite. |
| `ALTER TABLE ... ADD COLUMN DEFAULT` (constant)    | ❌ No (Postgres 11+) | ACCESS EXCLUSIVE       | Stored as metadata only. Old rows remain untouched.                       |
| `ALTER TABLE ... ADD COLUMN DEFAULT` (volatile)    | ✅ Yes               | ACCESS EXCLUSIVE       | Each existing row needs its own computed value.                           |
| `ALTER TABLE ... DROP COLUMN`                      | ❌ No (mostly)       | ACCESS EXCLUSIVE       | Column is marked dropped in metadata; no immediate rewrite.               |
| `ALTER TABLE ... ALTER COLUMN SET NOT NULL`        | ❌ No                | ACCESS EXCLUSIVE       | Metadata-only change (with validation scan).                              |
| `ALTER TABLE ... ALTER COLUMN SET STORAGE`         | ❌ No                | ACCESS EXCLUSIVE       | Only changes future TOAST strategy; existing rows unchanged.              |
| `CREATE INDEX`                                     | ❌ No                | SHARE                  | Creates a separate index file, blocks writes but not reads.               |
| `CREATE INDEX CONCURRENTLY`                        | ❌ No                | SHARE UPDATE EXCLUSIVE | Allows reads and writes; takes longer.                                    |
| `DROP INDEX`                                       | ❌ No                | ACCESS EXCLUSIVE       | Deletes index files, leaves table file alone.                             |
| `VACUUM FULL`                                      | ✅ Yes               | ACCESS EXCLUSIVE       | Compacts table into a new file, frees all space.                          |
| `CLUSTER`                                          | ✅ Yes               | ACCESS EXCLUSIVE       | Reorders the table based on an index into a new file.                     |
| `ANALYZE`                                          | ❌ No                | SHARE UPDATE EXCLUSIVE | Collects statistics, no file change.                                      |
| `VACUUM` (regular)                                 | ❌ No                | SHARE UPDATE EXCLUSIVE | Cleans dead tuples in-place, no rewrite.                                  |

The default lock level for most `ALTER TABLE` forms is **ACCESS EXCLUSIVE**:

- ACCESS EXCLUSIVE blocks everything
- SHARE blocks writes but not reads;
- SHARE UPDATE EXCLUSIVE allows normal reads & writes.

## How to Test for a Rewrite

You can verify whether an operation causes a table rewrite by checking the **relfilenode** before and after the operation:

```sql
-- 1) Check the current relfilenode
SELECT pg_relation_filenode('public.mytable');

-- 2) Run your DDL operation

-- 3) Check again
SELECT pg_relation_filenode('public.mytable');
```

If the value changed, a rewrite occurred. This is a reliable way to test risky DDL before running it in production.

## Playbook

To avoid production issues with re-write:

**Prefer non-rewriting forms:** Use binary-compatible type changes (e.g., `varchar(50)` to `varchar(100)`) when possible.

**Schedule rewrites:** Plan operations like `VACUUM FULL`, `CLUSTER`, and `SET TABLESPACE` during maintenance windows.

## Conclusion

Understanding which Postgres operations cause table rewrites and their associated locking behavior is crucial for maintaining a healthy and performant database. Modern Postgres databases know which Postgres ops rewrite tables and what they lock. Newer versions skip many rewrites (e.g., constant DEFAULT). Follow the playbook: prefer non-rewriting changes and schedule rewrites in off-peak hours.


Which operation causes a Postgres table rewrite

Which Postgres Operation causes a table rewrite


Postgres's Row-Level Security (RLS) is a powerful feature for implementing fine-grained access control, but it's riddled with subtle traps that can destroy performance or completely bypass security. This comprehensive guide covers all the major footguns with practical fixes and real-world examples.

---

## Performance Footguns

### 1. The LEAKPROOF Function Performance Killer

**The footgun:** Non-LEAKPROOF functions in RLS policies prevent index usage, causing catastrophic performance degradation.

**Why it happens:** Postgres must apply RLS filtering first, then non-LEAKPROOF functions, preventing the query planner from using indexes.

**Example problem:**

```sql
-- This will cause full table scans even with an index on title
CREATE POLICY user_documents ON documents
FOR SELECT
USING (owner_id = current_user_id() AND title ILIKE '%search%');
```

**The fix:**

```sql
-- Use LEAKPROOF functions or move complex logic out of policies
CREATE OR REPLACE FUNCTION safe_ilike(text, text)
RETURNS boolean
LANGUAGE sql
LEAKPROOF
AS $$ SELECT $1 ILIKE $2 $$;

CREATE POLICY user_documents ON documents
FOR SELECT
USING (owner_id = current_user_id() AND safe_ilike(title, '%search%'));
```

**Performance impact:** Queries can go from milliseconds to hours on large tables.

### 2. Complex Policy Performance Death

**The footgun:** Complex RLS policies with subqueries execute for every row, multiplying query cost exponentially.

**Bad example:**

```sql
CREATE POLICY complex_access ON orders
USING (
  EXISTS (
    SELECT 1 FROM user_permissions up
    JOIN departments d ON up.dept_id = d.id
    WHERE up.user_id = current_user_id()
    AND d.region = orders.region
  )
);
```

**Better approach:**

```sql
-- Move complexity to a LEAKPROOF function
CREATE OR REPLACE FUNCTION user_has_region_access(region_name text)
RETURNS boolean
LANGUAGE sql
LEAKPROOF
STABLE
AS $$
  SELECT EXISTS (
    SELECT 1 FROM user_permissions up
    JOIN departments d ON up.dept_id = d.id
    WHERE up.user_id = current_user_id()
    AND d.region = region_name
  );
$$;

CREATE POLICY simple_access ON orders
USING (user_has_region_access(region));
```

### 3. Missing Indexes on Policy Columns

**The footgun:** Forgetting to index columns used in RLS policies forces sequential scans.

**Essential indexes:**

```sql
-- Always index columns used in policies
CREATE INDEX ON orders(tenant_id);
CREATE INDEX ON orders(owner_id);
CREATE INDEX ON orders(tenant_id, owner_id); -- composite for AND conditions
```

---

## Security Footguns

### 4. The BYPASSRLS Superuser Trap

**The footgun:** Superusers and table owners bypass RLS by default, creating false security confidence during testing.

**Why it's dangerous:** Testing with superuser accounts makes RLS appear to work when it's actually being ignored.

**The fix:**

```sql
-- Force RLS even for table owners
ALTER TABLE sensitive_data ENABLE ROW LEVEL SECURITY;
ALTER TABLE sensitive_data FORCE ROW LEVEL SECURITY;

-- Never grant BYPASSRLS to application roles
-- Test with dedicated non-superuser accounts
```

**Testing pattern:**

```sql
-- Create proper test user
CREATE ROLE test_user;
GRANT app_user TO test_user;

-- Test as the actual application user
SET ROLE test_user;
SET app.tenant_id = 'test-tenant-id';
-- Run your tests here
RESET ROLE;
```

### 5. SECURITY DEFINER View Bypass

**The footgun:** Views are SECURITY DEFINER by default, running with creator's privileges and bypassing RLS.

**Dangerous example:**

```sql
-- Created by superuser - bypasses ALL RLS policies!
CREATE VIEW all_patient_data AS
SELECT * FROM patients;
```

**Secure approaches:**

```sql
-- Postgres 15+: Use SECURITY INVOKER
CREATE VIEW patient_data
WITH (security_invoker = true)
AS SELECT * FROM patients;

-- Older versions: Use SECURITY DEFINER with explicit RLS
CREATE OR REPLACE FUNCTION get_patient_data()
RETURNS TABLE(id uuid, name text, doctor_id uuid)
SECURITY DEFINER
SET search_path = pg_catalog, public
AS $$
BEGIN
  -- Explicitly check RLS in SECURITY DEFINER functions
  IF NOT row_security_active('patients') THEN
    RAISE EXCEPTION 'Row security must be active';
  END IF;

  RETURN QUERY SELECT p.id, p.name, p.doctor_id FROM patients p;
END;
$$ LANGUAGE plpgsql;
```

### 6. Timing Side-Channel Attacks

**The footgun:** Query execution time leaks information about restricted data, allowing sophisticated attacks.

**Attack scenario:** In a multi-tenant medical database, an attacker measures query times to determine if patients with specific conditions exist in other tenants' data.

**Technical details:**

- RLS policy enforcement creates measurable timing differences
- Attackers can infer secret cardinality information
- Works even across network latency in cloud environments

**Example vulnerable query:**

```sql
-- Timing reveals if forbidden patients exist
SELECT COUNT(*) FROM patients
WHERE condition = 'rare_disease'
AND tenant_id = current_setting('app.tenant_id')::uuid;
```

**Mitigation strategies:**

1. Use data-oblivious query patterns (performance cost)
1. Add artificial delays to normalize timing
1. Limit query complexity for untrusted users
1. Monitor for suspicious timing-based query patterns

**Research note:** This attack has been demonstrated in academic research and works in real-world cloud environments.

### 7. CVE-2019-10130: Statistics Leakage

**The footgun:** [CVE-2019-10130](https://www.postgresql.org/support/security/CVE-2019-10130/). Postgres's query planner statistics could leak sampled data from RLS-protected rows.

**Technical details:**

- Query planner collects statistics by sampling column data
- Users could craft operators to read statistics containing forbidden data
- Affected Postgres 9.5-11 before May 2019 patches

**Status:** Fixed in Postgres 9.5.17, 9.6.13, 10.8, 11.3

**Lesson:** Keep Postgres updated and remember that even internal mechanisms can leak data.

---

## Implementation Footguns

### 8. Missing FORCE ROW LEVEL SECURITY

**The footgun:** Enabling RLS without FORCE allows table owners to bypass policies.

**Problem:**

```sql
-- Table owner still sees everything!
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;
```

**Solution:**

```sql
-- Force RLS for everyone, including owners
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;
ALTER TABLE orders FORCE ROW LEVEL SECURITY;
```

### 9. USING vs WITH CHECK Confusion

**The footgun:** USING filters existing rows for SELECT/UPDATE/DELETE, but WITH CHECK validates new/modified rows for INSERT/UPDATE.

**Dangerous example:**

```sql
-- Users can INSERT data they can't see!
CREATE POLICY tenant_data ON orders
FOR ALL
USING (tenant_id = current_setting('app.tenant_id')::uuid);
```

**Correct approach:**

```sql
CREATE POLICY tenant_isolation ON orders
FOR ALL
USING (tenant_id = current_setting('app.tenant_id')::uuid)
WITH CHECK (tenant_id = current_setting('app.tenant_id')::uuid);
```

### 10. Connection Pooling Context Loss

**The footgun:** With connection pooling, `current_user` is a shared database role, useless for tenant isolation.

**Problem:**

```sql
-- Useless with PgBouncer - all connections share same user
CREATE POLICY user_data ON orders
USING (owner_id = current_user);
```

**Solution:**

```sql
-- Use application-controlled session variables
-- App sets per transaction:
SET app.user_id = 'user-uuid';
SET app.tenant_id = 'tenant-uuid';

-- Policy reads from session:
CREATE POLICY tenant_isolation ON orders
FOR ALL
USING (tenant_id = current_setting('app.tenant_id')::uuid)
WITH CHECK (tenant_id = current_setting('app.tenant_id')::uuid);
```

**Security hardening:**

```sql
-- Prevent clients from setting app.* directly
REVOKE ALL ON SCHEMA pg_catalog FROM app_user;
-- Only allow trusted server code to set these variables
```

### 11. Foreign Key Failures Under RLS

**The footgun:** INSERT into child tables fails FK checks because RLS blocks SELECT on parent rows.

**Example failure:**

```sql
-- This INSERT fails even if customer exists
INSERT INTO orders (customer_id, tenant_id)
VALUES ('existing-customer-id', 'my-tenant');
-- ERROR: insert or update on table "orders" violates foreign key constraint
```

**Solution:**

```sql
-- Parent table needs SELECT policy for FK checks
CREATE POLICY customer_fk_visibility ON customers
FOR SELECT
USING (tenant_id = current_setting('app.tenant_id')::uuid);
```

### 12. Unique Constraint Cross-Tenant Leakage

**The footgun:** Global unique constraints reveal data existence across tenants.

**Problem:**

```sql
-- This reveals that email exists in ANY tenant
CREATE UNIQUE INDEX users_email_unique ON users(email);
-- INSERT fails with "duplicate key" even for other tenants
```

**Solution:**

```sql
-- Scope uniqueness to tenant
CREATE UNIQUE INDEX users_email_per_tenant
ON users(tenant_id, lower(email));
```

### 13. Silent Failures

**The footgun:** RLS failures are silent - operations fail without errors or warnings.

**Example:** An UPDATE that should modify 100 rows silently affects 0 rows due to RLS policy.

**Debugging approach:**

```sql
-- Temporarily disable RLS to test
SET row_security = off;
-- Run your query to see if RLS is the issue
SET row_security = on;

-- Or check if policies are active
SELECT row_security_active('table_name');
```

### 14. Column-Level Security Gaps

**The footgun:** RLS filters rows, not columns. Sensitive columns remain visible in allowed rows.

**Problem:**

```sql
-- Users can see SSN in their own records
SELECT * FROM users WHERE tenant_id = current_setting('app.tenant_id')::uuid;
```

**Solutions:**

```sql
-- Option 1: Column privileges
REVOKE SELECT (ssn, salary) ON users FROM app_user;

-- Option 2: Secure views
CREATE VIEW users_safe AS
SELECT id, name, email,
       CASE WHEN has_role('hr_role')
            THEN ssn
            ELSE 'XXX-XX-' || right(ssn, 4)
       END as ssn_masked
FROM users;
```

### 15. Materialized Views and Background Jobs

**The footgun:** Data copied to materialized views or exported by jobs isn't automatically protected by source table policies.

**Problems:**

```sql
-- Materialized view bypasses RLS
CREATE MATERIALIZED VIEW order_summary AS
SELECT * FROM orders; -- Contains all tenants' data!

-- Background job exports everything
COPY (SELECT * FROM orders) TO '/tmp/backup.csv';
```

**Solutions:**

```sql
-- Apply filtering in materialized views
CREATE MATERIALIZED VIEW tenant_order_summary AS
SELECT tenant_id, COUNT(*), SUM(amount)
FROM orders
GROUP BY tenant_id;

-- Use RLS-aware exports
COPY (
  SELECT * FROM orders
  WHERE tenant_id = 'specific-tenant'
) TO '/tmp/tenant_backup.csv';
```

### 16. Multiple Policy Confusion

**The footgun:** Multiple permissive policies are OR-ed together; one broad policy can override stricter ones.

**Problem:**

```sql
-- These policies are OR-ed - users get access if EITHER is true
CREATE POLICY user_own_data ON orders
USING (owner_id = current_user_id());

CREATE POLICY admin_all_data ON orders
USING (has_role('admin')); -- Oops, too broad!
```

**Solutions:**

```sql
-- Option 1: Use restrictive policies (AND-ed)
CREATE POLICY tenant_restriction ON orders
AS RESTRICTIVE
USING (tenant_id = current_setting('app.tenant_id')::uuid);

-- Option 2: Combine logic in single policy
CREATE POLICY combined_access ON orders
USING (
  tenant_id = current_setting('app.tenant_id')::uuid
  AND (
    owner_id = current_user_id()
    OR has_role('tenant_admin')
  )
);
```

## Key Takeaways

1. **RLS is not a security silver bullet** - it can be bypassed through multiple vectors
1. **Performance impact is severe** - always index policy columns and keep policies simple
1. **Testing methodology is critical** - never test with superuser accounts
1. **Silent failures make debugging painful** - policies fail without warnings
1. **Context management is crucial** - use secure session variables, not `current_user`
1. **Defense in depth** - combine RLS with column privileges, secure views, and application-level controls

RLS is powerful when implemented correctly, but requires careful attention to these footguns to avoid catastrophic security and performance failures.


An engineering perspective to evaluate Postgres row-level-security footguns

Common Postgres Row-Level-Security footguns


Database integrity and recovery mechanisms are critical for any production system. Postgres provides multiple rollback strategies: built-in transaction rollback with SAVEPOINT, Point-in-Time Recovery (PITR), and modern cross-transaction DML rollback solutions with tools. Each serves different use cases with distinct limitations.

## Built-in Transaction Rollback and SAVEPOINT

Postgres transactions allow rolling back all changes within a transaction block. For granular control, `SAVEPOINT` create markers within transactions, enabling partial rollbacks without affecting earlier operations.

### Using SAVEPOINT

Create a savepoint:

```sql
SAVEPOINT my_savepoint;
```

Roll back to it:

```sql
ROLLBACK TO SAVEPOINT my_savepoint;
```

**Practical pattern for risky operations:**

```sql
BEGIN;

-- Step 1: safe operations
INSERT INTO employees (name, department) VALUES ('Alice', 'Engineering');

SAVEPOINT sp_batch;

-- Step 2: risky operations
INSERT INTO employees (name, department) VALUES ('Bob', 'Marketing');
-- Oops, Bob is actually in Sales

-- Roll back only the risky step
ROLLBACK TO SAVEPOINT sp_batch;

-- Step 3: continue with corrected operation
INSERT INTO employees (name, department) VALUES ('Bob', 'Sales');

COMMIT;
```

The savepoint remains usable after rollback, but any savepoints created after it are destroyed and invalidated by the rollback (not just released).

### Limitations

- Some DDL statements (`CREATE DATABASE`, `DROP DATABASE`, `CREATE TABLESPACE`, `DROP TABLESPACE`) cannot run inside transactions
- Only works for uncommitted transactions - once committed, `ROLLBACK` cannot undo changes

## Point-In-Time Recovery (PITR)

PITR restores databases to specific points in time using continuous WAL archiving. Postgres's Write-Ahead Log records every database change. PITR combines base backups with archived WAL files to replay changes up to any desired moment.

### Cloud Provider Support

Major cloud providers offer one-click PITR experiences:

- **AWS RDS for Postgres**: Restore to point in time via Console/CLI/API
- **Google Cloud SQL**: PITR from console interface
- **Azure Database for Postgres**: Portal "Restore" to latest or chosen restore point

### Named Restore Points

Create targeted recovery points for easier PITR:

```sql
-- Before risky migration
SELECT pg_create_restore_point('pre_migration_2025_09_04');
```

Later recover using `recovery_target_name = 'pre_migration_2025_09_04'` instead of guessing timestamps.

### Advantages

- Handles any rollback scenario regardless of transaction commit status
- Can recover from errors discovered hours or days later

### Limitations

- Operates at cluster level - rolls back entire database, not individual tables or rows
- Heavyweight operation unsuitable for small, isolated changes
- Rolling back one incorrect `UPDATE` also undoes all subsequent valid changes

## Cross-Transaction DML Rollback (Compensating Changes)

After a bad `UPDATE`/`DELETE`/`INSERT` is committed, you need **compensating DML** that restores previous values - think "git revert" for data.

### Manual Compensating DML Example

Accidentally ran:

```sql
UPDATE accounts SET status = 'inactive' WHERE org_id = 42;
```

Compensate using audit/history table:

```sql
-- Revert to last known status per row
UPDATE accounts a
SET status = h.old_status
FROM account_status_history h
WHERE a.id = h.account_id
  AND h.org_id = 42
  AND h.changed_at = (
       SELECT max(h2.changed_at)
       FROM account_status_history h2
       WHERE h2.account_id = a.id
         AND h2.changed_at < :mistake_time
     );
```

Real systems must handle sequences, cascades, triggers, and side-effects.

### Bytebase Solution

Bytebase provides [point-and-click rollback](https://docs.bytebase.com/change-database/rollback-data-changes) functionality:

1. **Prior Backup**: Automatically captures affected rows before DML execution and stores in the dedicated `bbdataarchive` schema
1. **1-Click Rollback**: Generates and executes rollback scripts automatically

### Workflow Benefits

- Eliminates manual rollback script creation
- Integrated review and approval process
- Multi-task rollback across databases
- Safe, controlled change management

## Choosing the Right Rollback Method

Now that you understand the three rollback approaches, here's how to choose the right one for your situation:

| Situation                             | Best Tool                                 | Why                                                  |
| ------------------------------------- | ----------------------------------------- | ---------------------------------------------------- |
| Still in session, haven't committed   | **Transaction rollback / SAVEPOINT**      | Instant, lossless; keep good work, discard bad chunk |
| Committed a small wrong UPDATE/DELETE | **Cross-transaction rollback (Bytebase)** | Surgical fix; no cluster restore                     |
| Dropped table / mass data corruption  | **PITR**                                  | Ubiquitous, reliable; recovers to clean time point   |

- **Use transactions + SAVEPOINT** to avoid mistakes in the first place
- **Use PITR** when blast radius is unclear or damage is large - it's ubiquitous and cloud-friendly
- **Use compensating DML (or Bytebase's rollback workflow)** for small, precise fixes after commit - without PITR's weight


An engineering perspective to evaluate Postgres rollback strategies

Postgres Rollback Explained


## Understanding Postgres Locks

Postgres uses a [sophisticated locking system](https://www.postgresql.org/docs/current/explicit-locking.html) with multiple lock modes. DDL operations often acquire locks that conflict with routine DML operations like `INSERT`, `UPDATE`, and `DELETE`. This means a seemingly harmless schema change can bring your entire application to a halt if not executed carefully.

Let's understand what locks DML operations acquire:

| DML                        | Lock Mode         | Description                                                            |
| -------------------------- | ----------------- | ---------------------------------------------------------------------- |
| `SELECT`                   | **ACCESS SHARE**  | Only conflicts with ACCESS EXCLUSIVE                                   |
| `INSERT`/`UPDATE`/`DELETE` | **ROW EXCLUSIVE** | Conflicts with SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, ACCESS EXCLUSIVE |

Now we can see how the two most common schema migration operations conflict with DML:

| DDL Operation              | Lock Mode            | SELECT        | INSERT/UPDATE/DELETE |
| -------------------------- | -------------------- | ------------- | -------------------- |
| `ALTER TABLE` (most cases) | **ACCESS EXCLUSIVE** | ❌ **BLOCKS** | ❌ **BLOCKS**        |
| `CREATE INDEX`             | **SHARE**            | ✅ Allows     | ❌ **BLOCKS**        |

Many `ALTER TABLE` forms acquire an **ACCESS EXCLUSIVE** lock, which is the most restrictive lock in Postgres. This completely blocks your application - no reads, no writes, nothing can happen on that table until the operation completes.

`CREATE INDEX` uses a **SHARE** lock, which allows `SELECT` statements but blocks all write operations (`INSERT`, `UPDATE`, `DELETE`). For a large table, index creation can take hours, effectively making your application read-only.

Both scenarios cause downtime. In the following sections, we'll explore alternatives that avoid these blocking behaviors.

## ALTER TABLE without Downtime

While most `ALTER TABLE` operations acquire `ACCESS EXCLUSIVE` locks, Postgres provides several techniques to avoid downtime for common schema changes.

### Adding Columns with DEFAULT Values

Since [Postgres 11](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=16828d5c0273b4fe5f10f42588005f16b415b2d8), adding a column with a `DEFAULT` value no longer requires a table rewrite in many cases.

```sql
-- ✅ Fast operation (Postgres 11+)
-- No table rewrite, minimal lock time
ALTER TABLE users ADD COLUMN status text DEFAULT 'active';

-- ❌ Slow operation (requires table rewrite)
-- Only use if you need a non-constant default
ALTER TABLE users ADD COLUMN created_at timestamp DEFAULT now();
```

**How it works**: Postgres stores the default value in the system catalog instead of rewriting every row. When you read a row that doesn't have the new column, Postgres automatically returns the default value.

### Adding Constraints in Two Phases

For constraints that need validation against existing data, use the `NOT VALID` approach:

#### Check Constraints

```sql
-- Step 1: Add constraint without validating existing data
-- Fast - only blocks briefly to update catalog
ALTER TABLE users ADD CONSTRAINT users_age_positive
  CHECK (age > 0) NOT VALID;

-- Step 2: Validate existing data (can take time but doesn't block writes)
-- Uses SHARE UPDATE EXCLUSIVE lock - allows reads/writes
ALTER TABLE users VALIDATE CONSTRAINT users_age_positive;
```

#### Foreign Key Constraints

```sql
-- Step 1: Add foreign key without validation
-- Fast - only updates catalog, doesn't validate existing data
ALTER TABLE orders ADD CONSTRAINT orders_user_id_fk
  FOREIGN KEY (user_id) REFERENCES users(id) NOT VALID;

-- Step 2: Validate existing relationships
-- Can take time but allows concurrent operations
ALTER TABLE orders VALIDATE CONSTRAINT orders_user_id_fk;
```

### Setting Columns to NOT NULL

The safest approach for `SET NOT NULL` is a four-step process:

```sql
-- Step 1: Add a check constraint (fast with NOT VALID)
ALTER TABLE users ADD CONSTRAINT users_email_not_null
  CHECK (email IS NOT NULL) NOT VALID;

-- Step 2: Validate the constraint (allows concurrent operations)
ALTER TABLE users VALIDATE CONSTRAINT users_email_not_null;

-- Step 3: Set NOT NULL (fast since constraint guarantees no nulls)
ALTER TABLE users ALTER COLUMN email SET NOT NULL;

-- Step 4: Drop the redundant check constraint
ALTER TABLE users DROP CONSTRAINT users_email_not_null;
```

**Why this works**: The final `SET NOT NULL` is fast because Postgres can see from the validated constraint that no null values exist.

**Postgres 18 Update**: The upcoming Postgres 18 introduces a [simplified form](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a379061a2) that allows `NOT NULL` constraints to use the `NOT VALID` attribute directly:

```sql
-- Postgres 18+ - Simplified approach
ALTER TABLE users ALTER COLUMN email SET NOT NULL NOT VALID;
ALTER TABLE users VALIDATE CONSTRAINT users_email_not_null;
```

## CREATE INDEX without Downtime

Instead of the blocking `CREATE INDEX`, use `CREATE INDEX CONCURRENTLY` to build indexes without interfering with application traffic:

```sql
-- ❌ Blocks all writes during index creation
CREATE INDEX idx_users_email ON users(email);

-- ✅ Allows writes to continue during index creation
CREATE INDEX CONCURRENTLY idx_users_email ON users(email);
```

`CREATE INDEX CONCURRENTLY` uses a **SHARE UPDATE EXCLUSIVE** lock instead of a **SHARE** lock, allowing `INSERT`, `UPDATE`, and `DELETE` operations to continue normally while the index is being built.

**Trade-offs**: While concurrent index creation avoids downtime, it takes longer to complete and has some limitations (cannot be used inside transactions, requires more careful error handling).

For a comprehensive guide on concurrent index creation, see our detailed article on [CREATE INDEX CONCURRENTLY](/blog/postgres-create-index-concurrently).

## Use lock_timeout

Regardless of the lock level, all DDL operations should use [lock_timeout](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCK-TIMEOUT) to prevent creating lock queues that can freeze your application.

### The Lock Queue Problem

When an `ACCESS EXCLUSIVE` operation waits for existing queries to complete, it blocks all subsequent operations from starting. Your application can appear completely frozen while waiting for one long query to finish.

```sql
-- Session 1: Long-running query (holds ACCESS SHARE)
SELECT COUNT(*) FROM large_table; -- Takes 5 minutes

-- Session 2: DDL waiting for ACCESS EXCLUSIVE
ALTER TABLE large_table ALTER COLUMN description TYPE text; -- Waits behind the SELECT

-- Session 3: New application query
SELECT * FROM large_table WHERE id = 123; -- BLOCKED by waiting DDL!
```

### Set lock_timeout

The `lock_timeout` parameter can be set at multiple levels:

```sql
-- Option 1: Session level (temporary)
SET lock_timeout = '5s';

-- Option 2: Role level (recommended - persistent across sessions)
CREATE ROLE ddl_user WITH LOGIN PASSWORD 'secure_password';
ALTER ROLE ddl_user SET lock_timeout = '5s';

-- Option 3: Database level (applies to all connections)
ALTER DATABASE mydb SET lock_timeout = '5s';
```

Setting `lock_timeout` at the Postgres role level is recommended because:

1. The setting applies to all sessions for that role
1. No need to remember to set it in each DDL session

```sql
-- Use the dedicated DDL user for schema migrations
-- Connect as ddl_user and run:
ALTER TABLE users ALTER COLUMN description TYPE text;
```

If the operation can't acquire the lock within 5 seconds, it fails with an error instead of blocking the application indefinitely.

### Retry Logic

Combine `lock_timeout` with retry logic to handle busy periods:

```sql
-- Example DDL script with retry logic
DO $$
DECLARE
    max_attempts INTEGER := 10;
    attempt INTEGER := 1;
    success BOOLEAN := FALSE;
BEGIN
    WHILE attempt <= max_attempts AND NOT success LOOP
        BEGIN
            SET lock_timeout = '2s';
            ALTER TABLE users ALTER COLUMN description TYPE text;
            success := TRUE;
            RAISE NOTICE 'DDL succeeded on attempt %', attempt;
        EXCEPTION
            WHEN lock_not_available THEN
                RAISE NOTICE 'Attempt % failed, retrying in 30 seconds...', attempt;
                PERFORM pg_sleep(30);
                attempt := attempt + 1;
        END;
    END LOOP;

    IF NOT success THEN
        RAISE EXCEPTION 'DDL failed after % attempts', max_attempts;
    END IF;
END $$;
```

<HintBlock type="info">

Bytebase automatically handles [retry logic](https://github.com/bytebase/bytebase/blob/c32bf12c40fec3adc16f56c274d490e407465b04/backend/plugin/db/pg/pg.go#L439-L460) during schema migrations.

</HintBlock>

## Conclusion

Achieving zero-downtime Postgres schema migrations requires understanding locks and applying the right techniques:

1. **Understand the impact**: Know which DDL operations block your application
1. **Use modern Postgres features**: Leverage `CONCURRENTLY`, `NOT VALID`, and other optimizations
1. **Always set lock_timeout**: Prevent lock queues that can freeze your application
1. **Plan for retries**: Handle busy periods gracefully with retry logic

By following these practices, you can deploy schema changes confidently without impacting your users.

## References

1. [Postgres locking mode](https://www.postgresql.org/docs/current/explicit-locking.html)
1. [Postgres 11: ADD COLUMN with DEFAULT optimization](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=16828d5c0273b4fe5f10f42588005f16b415b2d8)
1. [Postgres 18: NOT NULL constraints with NOT VALID](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a379061a2)
1. [How to Use Postgres CREATE INDEX CONCURRENTLY](/blog/postgres-create-index-concurrently)


Learn how Postgres locks affect schema migrations and which DDL operations conflict with DML to achieve zero-downtime deployments.

Postgres Schema Migration without Downtime Best Practice


[gh-ost](https://github.com/github/gh-ost) is a powerful tool for online schema changes in MySQL, but it's not a silver bullet. This article provides a concise, field-tested checklist of its limitations and potential gotchas, complete with real-world scenarios and examples to help you decide when to use it — and when to avoid.

---

## 1. Hard Blockers: If You Hit These, Stop!

Certain limitations in `gh-ost` are non-negotiable. If your environment falls into any of these categories, you'll need to find an alternative solution like [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html) or reconsider your migration strategy.

### Foreign Keys (FKs)

`gh-ost` does **not** support tables with foreign key constraints. If your table is referenced by other tables (i.e., it's a "parent" table), `gh-ost` will refuse to run. This is a critical safety feature to prevent data inconsistencies, but it's also a major blocker for many applications.

**Real-World Scenario:** Imagine you have a `customers` table with a primary key `id`, and an `orders` table with a foreign key `customer_id` that references `customers.id`. If you try to alter the `customers` table with `gh-ost`, the migration will fail because `customers` is a parent table. In this case, you would need to consider temporarily removing the foreign key constraints, using a different tool like [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html), or performing the migration during a maintenance window.

### Triggers

Tables that already have triggers are not supported by `gh-ost`. If your table has existing triggers for auditing, logging, or other purposes, `gh-ost` will not be able to perform the migration.

**Example:** If your `payments` table has triggers for auditing purposes, you'll need to either migrate the auditing logic to your application layer or use a different tool that can work with triggers.

### No Shared Unique Key

`gh-ost` requires that the source and ghost tables share an identical `PRIMARY KEY` or `UNIQUE KEY` to iterate over the rows. If your schema change involves altering or removing this key, `gh-ost` will not be able to proceed.

**Example:** If you need to change the layout of a multi-column primary key, you'll need to break the migration into multiple steps (e.g., add a new unique key, backfill the data, then switch to the new key) or use a different migration tool that can handle this type of change.

## 2. Infrastructure Requirements: Must-Haves for a Smooth Migration

Beyond the hard blockers, `gh-ost` has several infrastructure requirements that you must meet for a successful migration.

### Binary Log Format and Image

`gh-ost` requires **Row-Based Replication (RBR)** to be enabled. If your primary server uses Statement-Based Replication (SBR), you will need to configure a replica with RBR and have `gh-ost` connect to that replica. Additionally, many managed cloud database services require `binlog_row_image=FULL`.

**Example Command (for cloud environments):**
 ```bash
 gh-ost --assume-rbr --host=<replica_host> --cut-over=default ...
 ```

### Privileges

`gh-ost` requires specific replication-related privileges, and in some cases, `SUPER`-like capabilities. This can be challenging to obtain in managed database environments like [Amazon RDS](https://aws.amazon.com/rds/) or [Azure Database for MySQL](https://azure.microsoft.com/en-us/products/azure-database-for-mysql). Always check your platform's documentation for guidance on the required permissions for online schema migration tools.

## 3. The "Online" Lock: Cut-Over and Long-Running Transactions

Even though `gh-ost` is an "online" tool, the final cut-over phase still requires a **metadata lock (MDL)** to swap the old and new tables. This lock is typically brief, but any long-running queries on the table can block the swap, turning a minor pause into a major incident.

**Real-World Scenario:** A business intelligence (BI) team runs a 15-minute `SELECT` query on the table you are migrating, right as you are about to cut over. `gh-ost` will wait for the MDL, causing your deployment to hang and potentially blocking other application traffic. To avoid this, you need to coordinate with other teams, schedule cut-overs during low-traffic periods, and have a plan to kill any long-running queries that could interfere with the migration.

## 4. Performance and Capacity Pitfalls

`gh-ost` is not a magic bullet for performance. It has its own set of performance and capacity considerations that you need to be aware of.

### Headroom for a Full Copy

`gh-ost` creates a "ghost" table and copies all the data from the original table. This means you need to have enough free disk space for a full copy of the table, plus any additional growth in the binary logs during the migration.

### High Write Rates Can Outrun gh-ost

Under extreme write loads, `gh-ost`'s single-threaded binary log processing can become a bottleneck. The backlog of changes to apply can grow faster than `gh-ost` can process them, causing the migration to fall further and further behind. In some cases, [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html) may perform better under high write loads. It is always recommended to test both tools in your environment to see which one performs better for your specific workload.

### Dropping the Old Table

After the cut-over, `gh-ost` will drop the old table. On very large tables, this can cause a spike in I/O and replication lag. It is best to handle this during a low-traffic window or use a more careful housekeeping process to drop large tables.

## 5. Topology Compatibility and Native DDL

### Galera/PXC Not Supported

`gh-ost` is not compatible with [Galera](https://galeracluster.com/) or [Percona XtraDB Cluster (PXC)](https://www.percona.com/software/mysql-database/percona-xtradb-cluster) due to its locking and table swap strategy. For these environments, [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html) is generally the safer and recommended choice.

### Native MySQL May Be Faster

On modern versions of MySQL (8.0+), many DDL operations are now `INSTANT` or `INPLACE`, meaning they can be performed with near-zero impact without the need for an external tool like [gh-ost](https://github.com/github/gh-ost). Always check if a native DDL operation can meet your needs before reaching for a more complex tool.

**Example:** Adding a column with a default value may be an `INSTANT` operation in MySQL 8.0, depending on the data type and default value. In this case, a native `ALTER TABLE` is much simpler and faster than using `gh-ost`.

---

## Quick Decision Checklist

To help you decide if `gh-ost` is the right tool for your migration, here is a quick checklist:

- Table has **FKs or triggers**? → **Don't use gh-ost. Consider** [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html).
- Can you get **ROW binlogs** and a **shared UNIQUE/PRIMARY key**? → **OK.**
- Can you coordinate the **cut-over** (no long queries)? → **OK.**
- Using **Galera/PXC topology**? → **Prefer** [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html).
- On **MySQL 8.0+?** → **Check if native INSTANT/INPLACE DDL already solves it.**

If you can check all the boxes and have tested the migration on a replica first, `gh-ost` is an excellent choice for performing online schema changes on large, hot tables with minimal production impact.

A checklist of gh-ost limitations and potential gotchas

gh-ost Limitations You Need to Know Before Production


As an application developer, you can make your PostgreSQL applications more reliable and maintainable with just a few key practices. Here are 5 essential tips that focus on what's within your control as a developer—not database administration tasks—plus a bonus technique for managing index. Implementing these will make your DBA and infrastructure teams much happier.

## 1. Set application_name for Better Debugging

Setting `application_name` in your database connections helps with debugging. Without it, all connections look identical in monitoring tools, making troubleshooting difficult.

### How to Implement

**PostgreSQL connection string:**

```plain
postgresql://user:password@localhost:5432/mydb?application_name=user-service
```

**Node.js with pg:**

```javascript
const { Pool } = require('pg');

const pool = new Pool({
  host: 'localhost',
  database: 'mydb',
  user: 'myuser',
  password: 'mypassword',
  application_name: 'user-service',
});
```

**Python with psycopg2:**

```python
import psycopg2

conn = psycopg2.connect(
    host="localhost",
    database="mydb",
    user="myuser",
    password="mypassword",
    application_name="user-service"
)
```

**Java with JDBC:**

```java
String url = "jdbc:postgresql://localhost:5432/mydb?ApplicationName=user-service";
Connection conn = DriverManager.getConnection(url, "myuser", "mypassword");
```

**Go with lib/pq:**

```go
import (
    "database/sql"
    _ "github.com/lib/pq"
)

connStr := "host=localhost dbname=mydb user=myuser password=mypassword application_name=user-service"
db, err := sql.Open("postgres", connStr)
```

### What You Get

Once configured, you can easily identify your application's database activity:

```sql
-- See all active connections from your service
SELECT pid, usename, application_name, state, query
FROM pg_stat_activity
WHERE application_name = 'user-service';

-- Monitor long-running queries from specific services
SELECT application_name, query, now() - query_start as duration
FROM pg_stat_activity
WHERE state = 'active'
AND now() - query_start > interval '30 seconds';
```

## 2. Configure Sane Timeouts

Database timeouts prevent applications from hanging indefinitely. Without proper timeout configuration, your application can consume all connection pool slots and impact service availability.

### The Four Critical Timeouts

1. **`statement_timeout`** - Maximum time for any single SQL statement. Prevents runaway queries from consuming resources indefinitely.

1. **`lock_timeout`** - Maximum time to wait for a lock. Prevents deadlock scenarios from blocking your application forever.

1. **`idle_in_transaction_timeout`** - Maximum time a transaction can remain idle. Prevents abandoned transactions from holding locks and blocking other operations.

1. **`transaction_timeout`** - Maximum time for an entire transaction (PostgreSQL 17+). Prevents long-running transactions from holding locks too long.

### How to Configure

**In connection string:**

```plain
postgresql://user:pass@localhost/db?options=-c%20statement_timeout=30s%20-c%20lock_timeout=10s%20-c%20transaction_timeout=60s
```

**At transaction level:**

```sql
BEGIN;
SET LOCAL statement_timeout = '30s';
SET LOCAL lock_timeout = '10s';
SET LOCAL transaction_timeout = '60s'; -- PostgreSQL 17+
-- Your transaction operations here
COMMIT;
```

### Recommended Values by Use Case

| Use Case                | statement_timeout | lock_timeout | idle_in_transaction_timeout | transaction_timeout |
| ----------------------- | ----------------- | ------------ | --------------------------- | ------------------- |
| **Web Applications**    | 30s               | 10s          | 60s                         | 60s                 |
| **Background Jobs**     | 300s+             | 30s          | 300s                        | 600s                |
| **Reporting/Analytics** | 0 (disabled)      | 60s          | 600s                        | 0 (disabled)        |

**Reasoning:**

- **Web Applications**: User-facing requests should fail fast; clean up abandoned web requests
- **Background Jobs**: Data processing can take longer; jobs can wait a bit more for locks
- **Reporting/Analytics**: Reports can take hours; shouldn't block operations; longer analysis sessions

<HintBlock type="info">

To learn more, please refer [Postgres Timeout Explained](/blog/postgres-timeout).

</HintBlock>

## 3. Use Online Schema Migration Options

Schema migrations in production require careful planning. Standard migrations can lock your tables for extended periods, impacting application availability. Here's how to perform schema changes safely without downtime.

### The Problem with Standard Migrations

Traditional migrations acquire exclusive locks that block all reads and writes to your tables:

```sql
-- DON'T: This blocks ALL SELECT, INSERT, UPDATE, DELETE on users table
ALTER TABLE users ADD COLUMN email_verified boolean DEFAULT false;

-- DON'T: This blocks ALL writes (INSERT, UPDATE, DELETE) on orders table for hours
CREATE INDEX ON orders (user_id, created_at);
```

When these operations run on large tables, they can:

- **Block all application queries** for minutes or hours
- **Cause connection pool exhaustion** as queries queue up waiting for locks
- **Trigger cascading failures** across your entire application stack

### Online Migration Strategies

1. Use `CREATE INDEX CONCURRENTLY`

   ```sql
   -- Safe: Builds index without blocking reads/writes
   -- Note: Cannot be run inside a transaction
   CREATE INDEX CONCURRENTLY idx_orders_user_created
   ON orders (user_id, created_at);
   ```

   <HintBlock type="info">

   To learn more production tips, please refer [Postgres CREATE INDEX CONCURRENTLY](/blog/postgres-create-index-concurrently).

   </HintBlock>

1. Add columns and constraints safely

   ```sql
   -- Step 1: Add constraint without validation (fast)
   ALTER TABLE users ADD CONSTRAINT email_verified_not_null
   CHECK (email_verified IS NOT NULL) NOT VALID;

   -- Step 2: Validate constraint (can be done later, non-blocking)
   ALTER TABLE users VALIDATE CONSTRAINT email_verified_not_null;

   -- Step 3: Convert to NOT NULL constraint
   ALTER TABLE users ALTER COLUMN email_verified SET NOT NULL;
   ```

## 4. Split PostgreSQL Roles for Different Operations

Implementing separate database roles for different operations improves security. This approach prevents accidental damage and makes operations more manageable.

### The Problem with Single "Super User" Approach

Many applications use a single database user with broad permissions:

```sql
-- DON'T: Single user with excessive privileges
GRANT ALL PRIVILEGES ON DATABASE myapp TO app_user;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO app_user;
```

This creates several problems:

- **Security risk**: Application has unnecessary DDL permissions
- **Operational risk**: Code bugs can drop tables or modify schema
- **Audit trail**: Can't distinguish between different operation types
- **Performance**: Can't optimize connections for specific use cases

### The Three-Role Strategy

1. Read-Only Role - For analytics, reporting, read replicas
1. Application Role - For normal CRUD operations
1. Migration Role - For schema changes and DDL operations

### Implementation Example

**Create the roles:**

```sql
-- 1. Read-only role for analytics/reporting
CREATE ROLE app_reader;
GRANT CONNECT ON DATABASE myapp TO app_reader;
GRANT USAGE ON SCHEMA public TO app_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO app_reader;
-- Automatically grant SELECT on future tables
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT ON TABLES TO app_reader;

-- 2. Application role for normal operations
CREATE ROLE app_writer;
GRANT CONNECT ON DATABASE myapp TO app_writer;
GRANT USAGE ON SCHEMA public TO app_writer;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_writer;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO app_writer;
-- Auto-grant on future objects
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_writer;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT USAGE, SELECT ON SEQUENCES TO app_writer;

-- 3. Migration role for schema changes
CREATE ROLE app_migrator;
GRANT CONNECT ON DATABASE myapp TO app_migrator;
GRANT ALL PRIVILEGES ON SCHEMA public TO app_migrator;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO app_migrator;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO app_migrator;
```

**Create actual users:**

```sql
-- Create users and assign roles
CREATE USER analytics_user PASSWORD 'secure_password';
GRANT app_reader TO analytics_user;

CREATE USER application_user PASSWORD 'secure_password';
GRANT app_writer TO application_user;

CREATE USER migration_user PASSWORD 'secure_password';
GRANT app_migrator TO migration_user;
```

## 5. Use UPSERT for Robust Operations

In production environments, operations fail, networks are unreliable, and retries are inevitable. UPSERT (`INSERT ... ON CONFLICT`) helps you handle these scenarios:

- Avoiding duplicate records when retrying failed requests
- Handling race conditions between concurrent operations
- Building idempotent operations that can be safely repeated
- Simplifying application logic for "create or update" patterns

### Ignore Duplicates

```sql
-- Pattern 1: Ignore duplicates
INSERT INTO users (email, name, created_at)
VALUES ('john@example.com', 'John Doe', NOW())
ON CONFLICT (email) DO NOTHING;
```

### Update on Conflict

```sql
-- Pattern 2: Update on conflict
INSERT INTO users (email, name, updated_at)
VALUES ('john@example.com', 'John Smith', NOW())
ON CONFLICT (email)
DO UPDATE SET
  name = EXCLUDED.name,
  updated_at = EXCLUDED.updated_at;
```

### Conditional Update

```sql
-- Pattern 3: Conditional update
INSERT INTO user_stats (user_id, login_count, last_login)
VALUES ($1, 1, NOW())
ON CONFLICT (user_id)
DO UPDATE SET
  login_count = user_stats.login_count + 1,
  last_login = EXCLUDED.last_login
WHERE user_stats.last_login < EXCLUDED.last_login;
```

## Bonus: Implement "Invisible" Indexes

PostgreSQL doesn't have native invisible indexes like some other databases, but you can simulate this behavior by manipulating the `indisvalid` flag in the `pg_index` system catalog. This technique allows you to temporarily disable an index without dropping it, which is useful for A/B testing query performance.

### Making an Index "Invisible"

```sql
-- Disable an existing index (make it invisible to the query planner)
UPDATE pg_index
SET indisvalid = false
WHERE indexrelid = 'idx_orders_status'::regclass;
```

### Making the Index "Visible" Again

```sql
-- Re-enable the index (make it visible to the query planner)
UPDATE pg_index
SET indisvalid = true
WHERE indexrelid = 'idx_orders_status'::regclass;
```

### Checking Index Status

```sql
-- See which indexes are currently disabled
SELECT
  schemaname,
  tablename,
  indexname,
  indisvalid AS is_valid
FROM pg_indexes
JOIN pg_index ON pg_indexes.indexname = pg_class.relname
JOIN pg_class ON pg_index.indexrelid = pg_class.oid
WHERE NOT indisvalid;
```

**Note**: PostgreSQL itself uses the `indisvalid` flag internally. When you run `CREATE INDEX CONCURRENTLY`, PostgreSQL automatically sets `indisvalid = false` until the index is fully built.

## Conclusion

These 5 tips are quick wins that focus on what's within your control as a developer. Adopting them will make both your life and your DBA/infrastructure teams' lives much easier:

1. **Set `application_name`** - Get instant visibility into which service is doing what
1. **Configure timeouts** - Prevent your application from hanging indefinitely
1. **Use online migrations** - Deploy schema changes without downtime
1. **Split database roles** - Follow the principle of least privilege
1. **Use UPSERT** - Handle failures and retries effectively

Start with `application_name` and timeouts—they take minutes to implement but provide immediate benefits. These practices make your applications more reliable while making everyone's job easier.


5 quick wins for application developers to improve Postgres reliability and make both their own and DBA/infrastructure teams' lives easier.

5 Postgres Production Tips for Application Developers


## The Problem with Regular CREATE INDEX

When you run a standard `CREATE INDEX` command in PostgreSQL, it acquires a **SHARE (ShareLock)** on the table, which has severe implications for concurrent operations.

```sql
-- This will block all writes to the users table
CREATE INDEX idx_users_email ON users(email);
```

### Technical Lock Details

The `SHARE` lock acquired by `CREATE INDEX` conflicts with several other lock modes:

- **RowExclusiveLock** (used by INSERT, UPDATE, DELETE)
- **ShareUpdateExclusiveLock** (used by VACUUM, ANALYZE, REINDEX CONCURRENTLY)
- **ShareRowExclusiveLock** (used by CREATE TRIGGER and some ALTER TABLE variants)
- **ExclusiveLock** (used by REFRESH MATERIALIZED VIEW CONCURRENTLY)
- **AccessExclusiveLock** (used by DROP TABLE, TRUNCATE, REINDEX, CLUSTER)

This lock compatibility matrix explains why regular index creation is so disruptive. Per the [PostgreSQL locking documentation](https://www.postgresql.org/docs/current/explicit-locking.html), a SHARE lock allows concurrent SELECT operations but blocks all write operations.

## CREATE INDEX CONCURRENTLY: The Solution

PostgreSQL's `CREATE INDEX CONCURRENTLY` allows index building without blocking writes by using a **SHARE UPDATE EXCLUSIVE (ShareUpdateExclusiveLock)** instead of a SHARE lock:

```sql
-- This allows writes to continue during index creation
CREATE INDEX CONCURRENTLY idx_users_email ON users(email);
```

### Lock Mode Comparison

The key difference lies in the lock mode used:

| Operation                   | Lock Mode                    | ❌ Conflicts With                                      | ✅ Allows                      |
| --------------------------- | ---------------------------- | ------------------------------------------------------ | ------------------------------ |
| `CREATE INDEX`              | **ShareLock**                | INSERT, UPDATE, DELETE, VACUUM, other DDL              | SELECT only                    |
| `CREATE INDEX CONCURRENTLY` | **ShareUpdateExclusiveLock** | Other DDL operations, multiple concurrent index builds | SELECT, INSERT, UPDATE, DELETE |

![lock-mode](/content/blog/postgres-create-index-concurrently/lock-table.webp)

The `ShareUpdateExclusiveLock` is specifically designed to allow concurrent data modifications while preventing conflicting DDL operations.

### How It Works

`CREATE INDEX CONCURRENTLY` uses a multi-phase approach:

1. **Initial Catalog Entry**: Creates index metadata with `indisvalid = false`
2. **First Table Scan**: Builds initial index structure while allowing writes
3. **Second Table Scan**: Catches up with changes that occurred during first scan
4. **Validation**: Marks index as valid (`indisvalid = true`)

During this process:

- **Writes continue normally** - INSERT, UPDATE, DELETE work without interruption
- **Takes longer than regular indexing** - Typically 2-3x slower due to multiple scans
- **Uses more resources** - Higher CPU and I/O load from tracking concurrent changes

### Key Limitations

While powerful, `CREATE INDEX CONCURRENTLY` has important restrictions:

#### Cannot Run Inside Transactions

```sql
BEGIN;
CREATE INDEX CONCURRENTLY idx_users_email ON users(email); -- ERROR!
COMMIT;
-- ERROR: CREATE INDEX CONCURRENTLY cannot run inside a transaction block
```

This limitation exists because the operation needs to commit multiple internal transactions during its phases.

#### Other Limitations

- **Only one concurrent index per table** - Multiple concurrent index builds on the same table will serialize
- **Failure leaves invalid index** - Must be manually dropped if creation fails
- **Foreign keys reference check** - Creating unique index concurrently may fail if duplicate values are inserted during creation

## Tracking Index Creation Progress

### Using pg_stat_progress_create_index

We can use [pg_stat_progress_create_index](https://www.postgresql.org/docs/current/progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING) to track the index creation:

```sql
-- Monitor index creation progress
SELECT
    pid,
    datname,
    relid::regclass AS table_name,
    index_relid::regclass AS index_name,
    phase,
    lockers_total,
    lockers_done,
    current_locker_pid,
    blocks_total,
    blocks_done,
    tuples_total,
    tuples_done,
    partitions_total,
    partitions_done
FROM pg_stat_progress_create_index;
```

The `phase` column shows the current operation stage:

- `initializing`: Starting up
- `waiting for writers before build`: Waiting for concurrent writes to finish
- `building index`: Main index creation phase
- `waiting for writers before validation`: Preparing for validation
- `index validation: scanning index`: Validating index entries
- `index validation: scanning table`: Final validation
- `waiting for old snapshots`: Waiting for transactions to complete
- `waiting for readers before marking dead`: Final cleanup

### Quick Validity Check with pg_index

The fastest way to check if an index is ready for use:

```sql
-- Check if index is valid and ready for use
SELECT
    schemaname,
    tablename,
    indexname,
    indexdef
FROM pg_indexes i
JOIN pg_class c ON c.relname = i.indexname
JOIN pg_index idx ON idx.indexrelid = c.oid
WHERE NOT idx.indisvalid
  AND schemaname NOT IN ('pg_catalog', 'information_schema');
```

The `indisvalid` column:

- `true`: Index is complete and being used by the query planner
- `false`: Index is either being built or failed during concurrent creation

_Postgres doesn't have built-in INVISBLE INDEX clause. You can achieve the behavior by setting `indisvalid` to `false`_.

## Best Practices

### Automatic Review

To prevent developers from running `CREATE INDEX` and accidentally locking the database, implement automatic SQL linting during the review process.

<HintBlock type="info">

[Bytebase SQL Review](https://docs.bytebase.com/sql-review/review-rules#create-index-concurrently) provides automated enforcement of the `CREATE INDEX CONCURRENTLY` rule and can be integrated with [CI/CD pipelines](https://docs.bytebase.com/gitops/overview) to catch violations before deployment.

</HintBlock>

### Always Verify Index Validity After Creation

```sql
-- After CREATE INDEX CONCURRENTLY completes
SELECT indisvalid
FROM pg_index
WHERE indexrelid = 'idx_users_email'::regclass;

-- If false, the index creation failed and needs cleanup
DROP INDEX IF EXISTS idx_users_email;
```

### Clean Up Failed Indexes

```sql
-- Find and drop all invalid indexes
DO $$
DECLARE
    r RECORD;
BEGIN
    FOR r IN
        SELECT schemaname, indexname
        FROM pg_indexes i
        JOIN pg_class c ON c.relname = i.indexname
        JOIN pg_index idx ON idx.indexrelid = c.oid
        WHERE NOT idx.indisvalid
          AND schemaname NOT IN ('pg_catalog', 'information_schema')
    LOOP
        EXECUTE format('DROP INDEX %I.%I', r.schemaname, r.indexname);
        RAISE NOTICE 'Dropped invalid index %.%', r.schemaname, r.indexname;
    END LOOP;
END $$;
```

### Performance Consideration

| Aspect              | CREATE INDEX          | CREATE INDEX CONCURRENTLY              |
| ------------------- | --------------------- | -------------------------------------- |
| **Lock Level**      | SHARE (blocks writes) | SHARE UPDATE EXCLUSIVE (allows writes) |
| **Duration**        | Baseline (1x)         | 2-3x longer                            |
| **CPU Usage**       | High burst            | Sustained moderate                     |
| **I/O Impact**      | Single intensive scan | Multiple moderate scans                |
| **Memory Usage**    | maintenance_work_mem  | Similar, held longer                   |
| **Transaction Log** | Minimal               | Higher due to concurrent changes       |

Even though `CREATE INDEX CONCURRENTLY` doesn't block writes, it still impacts performance:

- Schedule during low-traffic periods when possible
- Monitor CPU and I/O metrics during creation
- Consider increasing `maintenance_work_mem` temporarily for faster indexing

## References

1. [Postgres locking mode](https://www.postgresql.org/docs/current/explicit-locking.html)
1. [Index creation progress table](https://www.postgresql.org/docs/current/progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING)
1. [Source code for CREATE INDEX CONCURRENTLY](https://github.com/postgres/postgres/blob/ef5b87b970dc28adeeb88191fbf66c9d6298b112/src/backend/commands/indexcmds.c#L542)


Learn how to use CREATE INDEX CONCURRENTLY in Postgres to build indexes without blocking writes. Pitfalls, safety checks, and performance tips.

How to Use Postgres CREATE INDEX CONCURRENTLY


The database landscape is shifting dramatically. [PostgreSQL's market share has surged to 16.85%](https://experience.percona.com/postgresql/postgresql-market-in-2025/the-growing-dominance-of-postgresql), while MongoDB faces criticism over licensing changes and vendor lock-in. Supabase, a PostgreSQL-powered Backend-as-a-Service, is challenging MongoDB with the power of SQL and the developer experience of NoSQL.

This represents two fundamentally different philosophies:

- **MongoDB** pioneered document-based development but evolved toward proprietary licensing (SSPL) and Atlas-centric strategy
- **Supabase** leverages PostgreSQL's versatility — JSON documents, vector embeddings, real-time features — while staying fully open-source

Companies like [Infisical](https://infisical.com/blog/postgresql-migration-technical) report 50% cost reductions migrating from MongoDB to PostgreSQL, while [Supabase's rapid growth](https://x.com/kiwicopple/status/1947572962972078357) suggests a fundamental shift in developer preferences.

## The Evolution Story: From Open Source Heroes to Different Destinies

### MongoDB's Transformation: The Proprietary Pivot

MongoDB evolved from open-source champion to proprietary powerhouse, focusing on monetizing through Atlas cloud services. The 2018 Server Side Public License (SSPL) effectively closed-sourced MongoDB for many commercial use cases, prompting major cloud providers to offer only older versions and creating deployment challenges.

### Supabase's Rise: PostgreSQL's Modern Avatar

Supabase emerged in 2020 by building a comprehensive developer platform around PostgreSQL's advanced capabilities — JSONB for documents, pgvector for AI, and ACID transactions — while delivering the developer experience that made Firebase popular. This timing proved perfect for the current era of AI-driven applications.

## Comprehensive Comparison: Multiple Dimensions Analysis

| Dimension | Supabase | MongoDB |
|-----------|----------|---------|
| **Database Foundation** | PostgreSQL (Relational + Document) | Document-oriented NoSQL |
| **Data Model** | Structured tables + JSONB support | Flexible document collections |
| **Schema Management** | Defined schema with migrations | Schema-less with optional validation |
| **Query Language** | SQL + PostgREST API | MongoDB Query Language (MQL) |
| **ACID Transactions** | Full ACID compliance | Limited (requires replica sets) |
| **Scalability Approach** | Vertical + Read replicas + Sharding | Horizontal sharding (native) |
| **Real-time Features** | Built-in via PostgreSQL replication | Change Streams (requires configuration) |
| **Authentication** | Integrated (JWT, OAuth, RLS) | Separate service required |
| **File Storage** | Integrated S3-compatible | Separate GridFS or external |
| **Edge Functions** | Built-in serverless functions | Requires separate Atlas Functions |
| **API Generation** | Auto-generated REST + GraphQL | Manual API development |
| **Pricing Model** | Transparent usage-based | Complex tiered with hidden costs |
| **Free Tier** | 500MB DB, 50K MAU, 1GB storage | 512MB storage, shared resources |
| **Vendor Lock-in Risk** | Low (standard PostgreSQL) | High (proprietary features) |
| **License** | Fully open source | SSPL (restrictive) |
| **Cloud Provider Support** | All major providers | Limited due to licensing |
| **Learning Curve** | SQL knowledge required | Easier for beginners |
| **Enterprise Features** | Row Level Security, Audit logs | Advanced security, compliance |
| **Ecosystem Maturity** | Growing rapidly | Mature but fragmenting |
| **Migration Complexity** | Standard SQL tools | Custom migration required |
| **Performance Profile** | Excellent for complex queries | Better for simple document ops |
| **AI/ML Support** | Native vector embeddings (pgvector) | Atlas Vector Search |
| **Backup & Recovery** | Point-in-time recovery | Continuous backup |
| **Monitoring** | Built-in dashboard + metrics | Comprehensive Atlas monitoring |
| **Multi-region** | Read replicas | Global clusters |
| **Compliance** | SOC 2, GDPR ready | SOC 2, HIPAA, PCI DSS |

## Architecture Deep Dive: Two Philosophies in Practice

### Supabase: The Integrated Ecosystem

Supabase's architecture represents a "batteries-included" approach to backend development. Built on PostgreSQL's foundation, it integrates multiple open-source tools into a cohesive platform:

**Core Components:**
- **PostgreSQL**: The database engine with extensions for JSON, vectors, and geospatial data
- **PostgREST**: Auto-generates RESTful APIs from database schema
- **Realtime**: Elixir-based WebSocket server for live updates
- **GoTrue**: JWT-based authentication with social providers
- **Storage**: S3-compatible object storage with CDN
- **Edge Functions**: Deno-based serverless runtime

This integrated approach provides immediate productivity gains, allowing developers to build full-featured applications without separate service configuration.

MongoDB's modular architecture offers flexibility but requires more integration work, with each service operating independently and creating complexity in managing authentication and real-time synchronization.

## Performance and Scalability: The Technical Reality

### Supabase Performance Characteristics

Supabase inherits PostgreSQL's performance profile, which excels in several key areas:

**Strengths:**
- **Complex Queries**: Superior performance for joins, aggregations, and analytical workloads
- **Consistency**: ACID transactions ensure data integrity without performance penalties
- **Indexing**: Advanced indexing strategies including partial, functional, and GIN indexes
- **Concurrency**: MVCC (Multi-Version Concurrency Control) handles high concurrent loads efficiently

Supabase scales primarily through vertical scaling and read replicas, with manual sharding available. PostgreSQL excels in complex queries and AI workloads via pgvector.

MongoDB offers native horizontal sharding and fast document operations but can struggle with complex queries.

## Pricing Analysis: The Total Cost of Ownership

> **Disclaimer:** Pricing information in this section is current as of the publication date (August 21, 2025) and may change over time. Please refer to the official [Supabase pricing](https://supabase.com/pricing) and [MongoDB Atlas pricing](https://www.mongodb.com/pricing) pages for the most up-to-date information.
[Supabase offers transparent, usage-based pricing](https://supabase.com/pricing) starting at $25/month for production workloads, while [MongoDB Atlas](https://www.mongodb.com/pricing) begins at $60/month with additional costs for authentication, storage, and enterprise features. MongoDB's complex pricing model often leads to unexpected costs as applications scale.

### Cost Comparison Analysis

For a typical web application with moderate traffic:

| Component | Supabase (Pro tier) | MongoDB Atlas |
|-----------|-------------------|---------------|
| **Core Database & Backend** | $25/month | $60/month (M10 cluster) |
| **Authentication** | Included | $20-50/month |
| **File Storage** | $5-15/month | $10-30/month |
| **Real-time Features** | Included | Additional development cost |
| **API Generation** | Included | Manual development required |
| **Total Monthly Cost** | **$30-40** | **$90-140** |

This analysis aligns with real-world reports of 50% cost reductions when migrating from MongoDB to PostgreSQL-based solutions.

## Conclusion

**Choose Supabase for:** Rapid development, relational data, cost predictability, and open-source values.

**Choose MongoDB for:** Massive scale, flexible schemas, and existing MongoDB expertise.

Supabase's PostgreSQL-powered platform offers integrated services with transparent pricing, while MongoDB's licensing changes have reduced its appeal. The shift toward Supabase reflects broader trends: the return to SQL and preference for developer-friendly, integrated platforms — marking the beginning of a PostgreSQL renaissance.

An extensive comparison between Supabase and MongoDB on performance, usability, operability, ecosystem and more.

Supabase vs. MongoDB: a Complete Comparison in 2025


When evaluating database management tools, DataGrip and Bytebase represent two different philosophies: individual developer productivity versus team collaboration and governance. Both tools have carved out distinct niches in the database ecosystem, each excelling in their respective domains.

DataGrip, developed by JetBrains, has earned widespread acclaim for its **polished user experience and comprehensive feature set**. As part of the renowned JetBrains suite, it brings the same attention to developer experience that has made IntelliJ IDEA and other JetBrains products industry standards.

![dg-head](/content/blog/bytebase-vs-datagrip/dg-head.webp)

Bytebase, on the other hand, emerged from the recognition that modern development teams need **collaborative database management tools that integrate with DevOps workflows**.

![bb-head](/content/blog/bytebase-vs-datagrip/bb-head.webp)

## Feature Comparison

| Feature | DataGrip | Bytebase |
|---------|----------|----------|
| **Product Position** | Powerful individual SQL IDE | Collaborative database development platform |
| **Interface** | Desktop application | Web-based platform |
| **Installation** | Desktop install required | Single binary deployment |
| **Database Support** | 35+ engines (complete + basic) | 22 enterprise-focused engines |
| **User Experience** | ⭐⭐⭐⭐⭐ Exceptional | ⭐⭐⭐ Good |
| **Query Development** | ⭐⭐⭐⭐⭐ Advanced | ⭐⭐⭐ Solid |
| **AI Assistance** | ⭐⭐⭐⭐ Comprehensive | ⭐⭐⭐ Basic |
| **Team Collaboration** | ⭐⭐ Limited | ⭐⭐⭐⭐⭐ Excellent |
| **Change Management** | ❌ Direct changes only | ✅ Review workflow |
| **Access Control** | ❌ Local credential storage | ✅ Centralized with RBAC |
| **Data Masking** | ❌ Not available | ✅ Advanced (Paid) |
| **Audit Log** | ❌ Limited local history | ✅ Comprehensive (Paid) |
| **Schema Compare** | ✅ 1:1 comparison | ✅ 1:N comparison |
| **API Integration** | ⭐⭐ Limited | ✅ REST/gRPC APIs |
| **GitOps Integration** | ❌ Manual processes | ✅ Native VCS integration |
| **Governance & Compliance** | ⭐ Limited | ⭐⭐⭐⭐⭐ Enterprise-grade |
| **Performance** | ⭐⭐⭐⭐⭐ Excellent | ⭐⭐⭐⭐ Good |
| **DevOps Integration** | ⭐⭐ Basic | ⭐⭐⭐⭐⭐ Native |
| **Learning Curve** | ⭐⭐⭐ Moderate | ⭐⭐⭐⭐ Easy |
| **Pricing Model** | Individual/team licenses | Free + paid tiers |

## Database Engine Support Comparison

### DataGrip: Extensive Multi-Database Coverage (35+ Engines)

DataGrip provides comprehensive database support with **complete support for 28 major database systems** and **basic support for 10+ additional engines**. This includes:

- **Core Systems**: MySQL, PostgreSQL, Oracle, SQL Server, MariaDB, SQLite
- **Cloud & Analytics**: BigQuery, Snowflake, Redshift, ClickHouse, Databricks
- **NoSQL**: MongoDB, Cassandra, Redis, DynamoDB
- **Specialized**: Apache Spark, Elasticsearch, Google Cloud Spanner

### Bytebase: Enterprise-Focused Support (22 Engines)

Bytebase supports **22 carefully selected database engines** focusing on enterprise environments:

- **Core Relational**: MySQL, PostgreSQL, Oracle, SQL Server, MariaDB, SQLite
- **Cloud Warehouses**: Snowflake, Redshift, BigQuery, Databricks
- **Modern Analytics**: ClickHouse, StarRocks, RisingWave
- **NoSQL & Emerging**: MongoDB, Redis, TiDB, OceanBase

**Key Difference**: DataGrip maximizes compatibility across the database ecosystem, while Bytebase focuses on deep integration with enterprise-critical systems.

## Target Roles and Responsibilities

### DataGrip: Individual Specialists and Power Users

**Best suited for:**
- **Database Developers & Data Analysts**: Complex SQL development, advanced query features, AI assistance
- **Data Scientists & Backend Developers**: Powerful query capabilities, desktop IDE integration
- **Database Consultants**: Comprehensive database support across client environments
- **BI Developers & Senior DBAs**: Advanced tools for optimization and complex development

### Bytebase: Team-Oriented Roles and Coordinators

**Best suited for:**
- **Team-Based Developers**: Collaborative environments, query sharing, web-based access
- **Database Administrators**: Production governance, team coordination, organizational policies
- **DevOps/Platform Engineers**: CI/CD integration, automated deployments, standardized operations
- **Engineering Managers & Security Engineers**: Change coordination, governance, access controls

### Key Difference

- **DataGrip**: Individual productivity, advanced features, comprehensive database support
- **Bytebase**: Team collaboration, governance, structured change management processes

## Goals and Objectives Each Tool Achieves

The choice between DataGrip and Bytebase often comes down to the specific goals your organization wants to achieve.

### DataGrip: Maximizing Individual Productivity

**Primary Goals:**
- **Accelerate Query Development**: Reduce time spent writing and debugging complex SQL through intelligent code completion and AI assistance
- **Improve Code Quality**: Leverage advanced refactoring tools and syntax checking to write more maintainable database code
- **Enhance Developer Experience**: Provide a polished, responsive interface that makes database work enjoyable and efficient
- **Integrate with Development Workflows**: Maintain consistency with other JetBrains tools in the development environment

**Secondary Goals:**
- **Reduce Learning Curve**: Familiar interface for teams already using JetBrains products
- **Optimize Performance**: Handle large datasets and complex queries efficiently
- **Enable Advanced Analysis**: Support sophisticated data exploration and reporting tasks

### Bytebase: Enabling Team Collaboration and Governance

**Primary Goals:**
- **Implement Database DevOps**: Integrate database changes into modern CI/CD pipelines with GitOps workflows
- **Ensure Change Control**: Prevent unauthorized or risky database modifications through review and approval processes
- **Maintain Compliance**: Provide comprehensive audit trails and access controls for regulatory requirements
- **Standardize Processes**: Establish consistent database management practices across teams and environments
- **Reduce Security Risks**: Centralize credential management and implement fine-grained access controls

**Secondary Goals:**
- **Improve Team Coordination**: Enable collaboration on database changes and knowledge sharing
- **Automate Governance**: Implement SQL review policies and automated quality checks
- **Scale Operations**: Support multi-environment deployments and tenant management

## Real-World Use Case Studies

### Use Case 1: E-commerce Data Analysis

**Scenario**: Data analyst analyzing customer behavior across PostgreSQL, MongoDB, and ClickHouse databases.

**DataGrip**: Direct multi-database connections, AI-assisted query development, local query organization, fast iteration.
*Result: High individual productivity and excellent user experience.*

**Bytebase**: Centralized access with data masking, shared query library, team collaboration, web-based setup.
*Result: Secure access with excellent team collaboration, though less sophisticated query development.*

### Use Case 2: Fintech Schema Migration

**Scenario**: Critical schema change deployment across environments with regulatory compliance requirements.

**DataGrip**: Local development and testing, but requires external coordination tools, manual consistency checks, separate audit systems.
*Result: Great development experience but challenging governance and coordination.*

**Bytebase**: Automated review workflows, approval processes, orchestrated deployments, GitOps integration, comprehensive audit trails.
*Result: Secure, auditable deployment with full governance, though less fluid individual development.*

### Use Case 3: Startup Development

**Scenario**: 5-developer startup needing rapid schema iteration with basic organization.

**DataGrip**: Excellent UX, rapid development, shared credentials, informal coordination suitable for small teams.
*Result: Maximum development velocity with manageable coordination overhead.*

**Bytebase**: Quick web-based setup, shared queries, centralized access, collaborative workflows that scale with team growth.
*Result: Good velocity with collaboration foundation that scales, plus easier onboarding.*

## What DataGrip and Bytebase Have in Common

Despite their different approaches, both platforms share several core capabilities:

- **Multi-database support**: Comprehensive support for major database systems, though with different breadth and depth
- **Advanced SQL editing**: Sophisticated editors with syntax highlighting, code completion, and error detection
- **Query execution and management**: Robust capabilities for running and organizing database queries
- **Data visualization**: Table views, result set management, and data browsing capabilities
- **Professional-grade reliability**: Both are built for serious database work with enterprise-level stability
- **Cross-platform availability**: Support for Windows, macOS, and Linux environments

## DataGrip's Strengths: Where It Truly Excels

- **Exceptional User Experience**: Polished, intuitive interface with JetBrains design consistency, customizable workspace, and responsive desktop performance
- **Advanced Query Development**: Sophisticated code completion, context-aware suggestions, automatic JOIN detection, AI Assistant with natural language to SQL conversion
- **JetBrains Ecosystem Integration**: Seamless workflow consistency with IntelliJ IDEA, PyCharm, shared configurations, and unified version control
- **Superior Performance**: Efficient handling of large datasets with lazy loading, intelligent caching, and responsive interface for millions of rows
- **Database-Specific Features**: Deep support for PostgreSQL data types, Oracle PL/SQL, and nuanced tooling for polyglot environments

## Bytebase's Strengths: Team Collaboration and Governance

- **Web-Based Collaboration**: Natural team collaboration, shared queries, collective knowledge management, single source of truth without desktop synchronization complexity
- **Database DevOps Integration**: GitOps workflows, schema changes through pull requests, CI/CD pipeline integration, automated testing and deployment coordination
- **Enterprise Governance**: Comprehensive audit logging, role-based access controls, data masking, approval workflows for compliance and regulatory requirements
- **Centralized Access Management**: Eliminates credential distribution risks, fine-grained permissions, instant access revocation, detailed activity logging

## Making the Right Choice

**Core Trade-off**: DataGrip prioritizes individual developer productivity with advanced query tools and desktop performance, while Bytebase focuses on team collaboration and governance with web-based accessibility.

### Choose DataGrip When:
- Individual productivity and advanced query development are priorities
- Working in small teams or solo with minimal coordination needs
- Already using JetBrains ecosystem (IntelliJ, PyCharm)
- Minimal governance/compliance requirements

### Choose Bytebase When:
- Team collaboration and centralized database management are essential
- Governance, compliance, and audit requirements are critical
- DevOps integration and automated workflows are priorities
- Web-based deployment and easy scaling are preferred

## Conclusion

DataGrip excels as a sophisticated database IDE with superior UX, powerful query capabilities, and excellent individual developer productivity. Bytebase focuses on team collaboration, database DevOps, and governance. The choice depends on whether you prioritize individual developer experience or team-based database management.


When evaluating database management tools, DataGrip and Bytebase represent two different philosophies: individual developer productivity versus team collaboration and governance.

Bytebase vs. DataGrip: a side-by-side comparison for database management


GitOps is a modern approach to manage infrastructure that emphasizes automation, collaboration, and continuous delivery. It’s based on the idea of using Git as the single source of truth for configuration and code. With GitOps, all changes to infrastructure are made through pull requests, which are reviewed and approved by other team members before being merged into the main branch.

In this post, we are reviewing several open-source GitOps tools:

- [Terraform](#terraform)
- [Pulumi](#pulumi)
- [ArgoCD](#argocd)
- [Bruno](#bruno)
- [Bytebase](#bytebase)
- [OPA](#opa)
- [mdx](#mdx)

[![star-history](/content/blog/top-open-source-gitops-tools/star.webp)](https://www.star-history.com/#hashicorp/terraform&pulumi/pulumi&argoproj/argo-cd&usebruno/bruno&open-policy-agent/opa&mdx-js/mdx&bytebase/bytebase&Date)

## Terraform

[Terraform](https://github.com/hashicorp/terraform) is an open-source infrastructure as code tool (IaC) that lets you build, change, and version infrastructure safely and efficiently. This includes low-level components like compute instances, storage, and networking; and high-level components like DNS entries and SaaS features.

Terraform uses HashiCorp Configuration Language (HCL) and also supports JSON for writing configurations.
![terraform](/content/blog/top-open-source-gitops-tools/terraform.webp)

To enable GitOps, create a Git repo with Terraform configs, define infrastructure in Terraform, and manage updates via a pipeline and pull requests.

![terraform-gitops](/content/blog/top-open-source-gitops-tools/terraform-gitops.webp)

## Pulumi

[Pulumi](https://github.com/pulumi/pulumi) is an open-source infrastructure as code tool that allows you to create, deploy, and manage cloud infrastructure using your favorite language. Unlike Terraform, Pulumi uses real programming languages like Python, JavaScript, and Go to define infrastructure.

![pulumi](/content/blog/top-open-source-gitops-tools/pulumi.webp)

Similar to Terraform, Pulumi can be used to enable GitOps by creating a Git repo with Pulumi configs, defining infrastructure in Pulumi, and managing updates via a pipeline and pull requests.

![pulumi-gitops](/content/blog/top-open-source-gitops-tools/pulumi-gitops.webp)

## ArgoCD

[ArgoCD](https://github.com/argoproj/argo-cd) is a declarative, GitOps continuous delivery tool for Kubernetes.

![argocd](/content/blog/top-open-source-gitops-tools/argocd.webp)

Argo CD follows the GitOps pattern of using Git repositories as the source of truth for defining the desired application state. Kubernetes manifests can be specified in several ways such as kustomize applications, helm charts and jsonnet files.

Argo CD automates the deployment of the desired application states in the specified target environments. Application deployments can track updates to branches, tags, or pinned to a specific version of manifests at a Git commit.

![argocd-diff](/content/blog/top-open-source-gitops-tools/argocd-diff.webp)

## Bruno

[Bruno](https://github.com/usebruno/bruno) is a Fast and Git-Friendly open-source API client, aimed at revolutionizing the status quo represented by Postman, Insomnia and similar tools out there.

![bruno](/content/blog/top-open-source-gitops-tools/bruno.webp)

Bruno stores your collections directly in a folder on your filesystem with a plain text markup language, Bru, to save information about API requests.

Regarding GitOps, you can use git or any version control of your choice to collaborate over your API collections.

![bruno-gitops](/content/blog/top-open-source-gitops-tools/bruno-gitops.webp)

## OPA

[OPA](https://github.com/open-policy-agent/opa) (Open Policy Agent) is an open-source policy engine designed to enforce policies across your stack. While OPA is not a GitOps tool itself, it can complement GitOps workflows by providing policy enforcement and compliance checks.

![opa](/content/blog/top-open-source-gitops-tools/opa.webp)

In the context of GitOps, OPA can be integrated into your CI/CD pipelines or Kubernetes admission controllers to validate changes to your infrastructure and applications. By enforcing policies defined in code, OPA ensures that all changes comply with your organization's standards before they are applied, enhancing the security and reliability of your GitOps processes.

## MDX

[MDX](https://github.com/mdx-js/mdx) is a tool that allows you to write JSX in Markdown documents, enabling the creation of interactive and dynamic content using React components within your Markdown files.

![mdx](/content/blog/top-open-source-gitops-tools/mdx.webp)

While MDX is not a traditional GitOps tool, it can complement GitOps workflows by enabling version-controlled documentation. This ensures that your documentation evolves alongside your code and infrastructure, maintaining alignment with the latest changes.

## Bytebase

[Bytebase](https://github.com/bytebase/bytebase) is an all-in-one database DevSecOps and CI/CD solution. It's like GitHub/GitLab that provides a GUI workspace for developers and DBAs to collaborate on database changes.

![bb](/content/blog/top-open-source-gitops-tools/bb.webp)

Besides its intuitive UI workflow, Bytebase also supports a GitOps workflow for managing database changes. This feature allows for initiating database issues through pull requests. Furthermore, Bytebase provides SQL review, custom approval workflows, and rollout policies, enabling database changes to be managed in an advanced GitOps manner.

![bb-rollout](/content/blog/top-open-source-gitops-tools/bb-rollout.webp)

## Summary

To summarize, these open-source GitOps tools offer a range of options for managing infrastructure and application deployments. They adhere to the GitOps principles, automating the deployment process to achieve the desired states in specified target environments.


In this post, we are reviewing several open-source GitOps tools.

Top Open Source GitOps Tools in 2025


![presentation](/content/blog/layerx-case-study/presentation.webp)

At CloudNative Days Summer 2025, Hokuto Hoshi—CISO and Head of SRE/Corporate Engineering at LayerX—shared how the company built a cloud-native governance framework that balances compliance, security, and developer agility. The presentation, titled [Realizing "Painless" Governance and Security in the Cloud](https://speakerdeck.com/kanny/effortless-governance-and-security-enabled-by-the-cloud), resonated with platform teams facing growing audit requirements while maintaining development velocity.

## About LayerX

![about-layerx](/content/blog/layerx-case-study/about-layerx.webp)

[LayerX](https://layerx.co.jp/) is a Tokyo-based technology company with a mission to enable digitalization across all economic activities. The company operates across several domains, including:

- [Bakuraku](https://bakuraku.jp/): A suite of AI-powered SaaS tools streamlining core enterprise workflows such as expense management, invoice processing, and approval flows.

- [Fintech](https://corp.mitsui-x.com/): Asset management and securities solutions delivered through a joint venture model.

- [AI/LLM](https://getaiworkforce.com/): An internal platform that transforms organizational knowledge into structured, retrievable data using large language models.

LayerX builds its products with a strong emphasis on automation, auditability, and developer experience—making it a prime example of a cloud-native enterprise balancing innovation with compliance.

## Governance Without the Pain

For many organizations, audit readiness still means manually collecting logs, managing permissions through spreadsheets, and retrofitting access controls into existing workflows. LayerX takes a different approach:

- Automate controls using infrastructure-as-code and cloud-native platforms
- Integrate security practices directly into development workflows
- Maintain transparency and explainability for auditors and engineers alike
- Treat compliance not as a checkbox but as part of system reliability

The implementation spans the entire stack—from user account management to infrastructure provisioning, from application deployment to database change management.

## From Identity to Infrastructure: A Systems Approach

### SSO as a Security Backbone

LayerX uses [Microsoft Entra ID](https://www.microsoft.com/en-us/security/business/identity-access/microsoft-entra-id) to unify identity across systems. Employee onboarding and offboarding are fully automated through HR data and Notion, with Slack-integrated workflows for approvals. Access is consistently enforced through SSO across all internal tools.

### Group-Based Permissioning

User groups are defined through [SmartHR](https://smarthr.jp/) and translated to [Terraform HCL](https://developer.hashicorp.com/terraform/language), then synced to Entra ID. This bridges HR systems with access control, enabling scalable role-based access patterns.

### Time-Bound Privilege Escalation

To handle temporary elevated permissions, LayerX leverages [Entra PIM](https://learn.microsoft.com/en-us/entra/id-governance/privileged-identity-management/pim-configure), allowing just-in-time access with automatic expiration—a modern answer to long-standing privilege management issues.

### Change Management in GitHub

Application and infrastructure changes are governed by pull request approvals using [GitHub's CODEOWNERS](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners). Even emergency changes are logged and traceable, and deployments are automated through tools like [ecspresso](https://github.com/kayac/ecspresso) and [Terraform](https://www.terraform.io/) within a monorepo setup.

### Unified Logging and Simplified Audits

Logs from [AWS CloudTrail](https://aws.amazon.com/cloudtrail/), Entra ID, [Datadog](https://www.datadoghq.com/), and [Amazon Athena](https://aws.amazon.com/athena/) are aggregated and searchable through APIs and CLI commands. LayerX stores logs in [Snowflake](https://www.snowflake.com/), making it easy to visualize and retrieve audit evidence. Log extraction is automated—eliminating ad hoc queries and manual exports.

## Database Governance with Bytebase

Traditionally, database operations are disconnected from CI/CD and governance workflows. LayerX addressed this gap by adopting [Bytebase](/) to introduce structured, auditable change management for their databases.

![bytebase](/content/blog/layerx-case-study/layerx-bytebase.webp)

With Bytebase, LayerX can:
- Review and approve schema changes through a web interface or API
- Maintain audit trails for all SQL operations
- Eliminate the need for bastion hosts by shifting to a secure, review-based model

This brings the same rigor to database operations that exists for code and infrastructure—an essential step for aligning security, reliability, and developer velocity.

## A Model for Modern Platform Teams

LayerX demonstrates that auditability and agility are not mutually exclusive. By embedding governance into the developer workflow and codifying it through infrastructure and policy engines, they've created a foundation that's secure, scalable, and operationally efficient.

For teams managing database changes, LayerX’s use of Bytebase highlights a growing trend: bringing CI/CD, approval workflows, and visibility to one of the last remaining blind spots in DevSecOps.

## Summary of Tools and Vendors Used

LayerX's accomplishments provide a pragmatic example of how modern engineering practices, when paired with the right tools, can turn governance from a bottleneck into a built-in strength.

| Tool / Platform                      | Vendor              | Purpose                                                  |
| ------------------------------------ | ------------------- | -------------------------------------------------------- |
| Microsoft Entra ID                   | Microsoft           | SSO, identity provider, group and access control         |
| SmartHR                              | SmartHR             | HR system for managing employee roles and attributes     |
| Terraform                            | HashiCorp           | Infrastructure-as-code, role syncing with IdP            |
| Entra PIM (Privileged Identity Mgmt) | Microsoft           | Time-limited elevated access with approvals              |
| GitHub + CODEOWNERS                  | Microsoft (GitHub)  | Pull request approval workflow for app and infra changes |
| ecspresso                            | KAYAC               | ECS deployment tool integrated with GitHub and Terraform |
| Bytebase                             | Bytebase            | CI/CD for database schema and data changes               |
| AWS CloudTrail                       | Amazon Web Services | Logging of AWS resource activities                       |
| Amazon Athena                        | Amazon Web Services | Serverless log querying and analysis                     |
| Snowflake                            | Snowflake           | Data warehouse for storing and visualizing audit logs    |
| Datadog                              | Datadog             | System monitoring and additional log aggregation         |
| Slack                                | Salesforce          | Notification and approval workflows                      |
| Notion                               | Notion              | Internal metadata and account provisioning source        |
| CLI / APIs / Scripts                 | Custom              | Automating log extraction and compliance tasks           |


How LayerX Achieves “Painless” Governance and Security in the Cloud


In today’s software-driven world, teams are moving faster than ever. **Infrastructure** is codified (Terraform, Pulumi, etc.), **Secrets** are rotated (AWS Secrets Manager, GCP Secret Manager, etc.), and **CI/CD** pipelines push changes every hour (GitHub, GitLab, etc.).

But the **database**? It's often the last part of the stack to catch up.

Most database changes are still manual. Access is rarely reviewed. And security? Often reactive — if addressed at all.

We believe it's time to bring **structure, automation, and security** to the database layer — not just for compliance, but for operational sanity.

Over the years, we’ve seen database security challenges evolve — and so has [Bytebase](https://bytebase.com). What has emerged is a unified model we now call **Database Security Lifecycle Management (DSLM)**: a structured approach to managing database **change**, **access**, and **security** from **Day 0 through Day 2**.


## Why DSLM?

Databases power your most critical data — customer records, financials, transactions — and yet:

- Schema changes happen without proper review
- Access is overly broad and rarely revoked
- Sensitive data is left unclassified or unprotected

[Bytebase](/) addresses these challenges head-on with a unified platform that integrates seamlessly into how teams work — combining features like **SQL review**, **batch schema changes**, **approval workflows**, **access control**, **data masking**, and **audit logging** to bring security, governance, and velocity to every stage of your database workflow.

## The Three Lifecycles of DSLM

### 🔁 Change Lifecycle — Plan, review, and deploy safely

Bytebase supports safe and structured schema changes through:

- **SQL review policies**: 100+ built-in rules to detect anti-patterns, enforce best practices, and catch security issues early
- **Approval flows**: define custom approval flows based on environment or project
- **Multi-environment deployment**: auto-promote changes across dev → staging → prod with rollback support
- **One-click rollback**: rollback to previous version with a single click
- **Change history & audit**: every change is logged — who, what, when, and why
- **GitOps workflows**: manage migrations directly from GitHub, GitLab, Azure DevOps or Bitbucket.

> 🔎 Bytebase supports PostgreSQL, MySQL, Oracle, SQL Server, MongoDB, TiDB, and more — so you can unify change workflows across your entire database estate.

### 🔐 Access Lifecycle — Just enough access, just in time

Database access in Bytebase is **request-based**, **temporary**, and **auditable**:

- **Role-based access control (RBAC)**: grant per-database, per-project, per-environment permissions
- **Access requests & approvals**: users request access; owners approve — all tracked
- **Auto-expiring access**: grant access for 1 hour, 1 day, or any custom duration
- **Query history & session audit**: see what was queried, by whom, and when
- **SSO integration**: connect to identity providers like Okta, Google, or LDAP for centralized user management

> 📌 This removes the risk of shared accounts and lingering admin access — a common source of data breaches.

### 🛡️ Security Lifecycle — Protect sensitive data and system integrity

Security in Bytebase goes beyond encryption — it's about proactive, policy-based defense:

- **Secrets management**: store DB credentials securely or integrate with AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault
- **Data masking**: apply column-level masking to protect sensitive data in query results
- **Sensitive data classification**: tag and monitor fields like PII or financial data
- **Audit logging**: every action — schema change, access grant, query execution — is logged for compliance

> 🛡️ Bytebase helps meet SOC 2, HIPAA, GDPR, and other regulatory requirements.

## DSLM from Day 0 to Day 2

| Day        | Focus    | 🔁 Change Lifecycle                         | 🔐 Access Lifecycle                   | 🛡️ Security Lifecycle                        |
| ---------- | -------- | ------------------------------------------- | ------------------------------------- | --------------------------------------------- |
| **Day 0**  | Setup    | Connect databases, configure SQL review     | Set up roles, SSO, approval workflows | Secure credentials, classify sensitive data   |
| **Day 1**  | Operate  | Submit changes via UI or Git, deploy safely | Approve requests, auto-expire access  | Enforce masking, monitor policy compliance    |
| **Day 2+** | Maintain | Rollback migrations, audit changes          | Audit usage, revoke idle permissions  | Rotate secrets, adapt to new compliance needs |

## Bytebase: Your DSLM Control Plane

Bytebase isn’t just a collection of features — it’s a **database DevSecOps platform** to manage your database lifecycle.

- 🔐 **Access management with audit trails**
- 🛡️ **Built-in security controls** for compliance
- 🎯 **GitOps-support**: Manage changes like code
- ⚙️ **API-first**: Integrate into your existing tools and workflows
- 🌐 **Self-hosted or cloud**: Open source, deploy the way your org needs

Everything is managed in one place, with a clean UI, robust automation, and role-aware permissioning for DBAs, developers, and security teams.

## Move Fast — Without Losing Control

Security and speed don’t have to be at odds.

With Bytebase and DSLM, you can:

- Make database changes with confidence
- Avoid over-privileged access and shared accounts
- Stay continuously audit-ready
- Align your database with modern DevSecOps practices

**Ready to secure your database lifecycle?**

Start with the [Community Plan](/pricing) or [get in touch](/contact) to see DSLM in action for your team.

Learn how Bytebase enables organizations to implement comprehensive Database Security Lifecycle Management, unifying security and operations from development to production.

Database Security Lifecycle Management


At Bytebase, we work extensively with both Postgres and MySQL. In our [previous post](/blog/features-i-wish-mysql-had-but-postgres-already-has), we highlighted features where Postgres shines compared to MySQL. In this post, we'll examine where MySQL excels and Postgres falls short.

## Performant Storage Engine for Write-Intensive Workloads

Postgres' heap-based storage with tuple versioning MVCC creates more overhead for write-intensive scenarios (write amplification). While Postgres has made improvements with features like HOT updates, it still generates more WAL (Write-Ahead Log) traffic and requires more frequent VACUUM operations under heavy write loads.

On the other hand, MySQL employs clustered index storage and undo logging, which is more performant especially for write heavy workload.

The comparison is detailed in [Uber's switch From Postgres to MySQL](https://www.uber.com/blog/postgres-to-mysql-migration/) and [The Part of Postgres We Hate the Most](https://www.cs.cmu.edu/~pavlo/blog/2023/04/the-part-of-Postgres-we-hate-the-most.html)

Postgres users need to deal with the infamous [transaction ID wraparound](https://blog.sentry.io/transaction-id-wraparound-in-postgres/), and the not-so-famous [MultiXact member exhaustion](https://metronome.com/blog/root-cause-analysis-Postgres-multixact-member-exhaustion-incidents-may-2025). [OrioleDB](https://github.com/orioledb/orioledb) is tackling these issues, but it's still years until it reaches the maturity of MySQL's InnoDB.

## Replication

MySQL's Group Replication provides both single-primary and multi-primary replication with automatic failover and conflict detection built-in. It handles network partitions gracefully with built-in quorum logic, automatically determining which partition remains writable.

Postgres relies on external tools like Patroni, pg_auto_failover for similar functionality. These solutions work but require non-trivial operational overhead and expertise to configure and maintain properly.

Postgres' logical replication, introduced in version 10, is also still catching up. Its most significant limitation is DDL operations are not replicated - they must be applied manually to subscribers.

## Invisible Indexes

MySQL's invisible indexes allow you to test index effectiveness without impacting existing queries or disable indexes temporarily - a crucial feature for production database optimization.

```sql
-- MySQL: Create invisible index for testing
CREATE INDEX idx_orders_status ON orders (status) INVISIBLE;

-- Test queries with the invisible index
SET SESSION optimizer_switch = 'use_invisible_indexes=on';
EXPLAIN SELECT * FROM orders WHERE status = 'pending';

-- Make index visible after confirming it helps
ALTER TABLE orders ALTER INDEX idx_orders_status VISIBLE;
```

They also help with index maintenance strategies. You can make an index invisible before dropping it, ensuring no queries break, then safely remove it later.

```sql
-- Make index invisible to see if it hurts
ALTER TABLE orders ALTER INDEX idx_orders_status INVISIBLE;

-- Then drop it if it doesn't break queries
DROP INDEX idx_orders_status ON orders;
```

Postgres doesn't provide built-in syntax to enable/disable an index. Though you can toggle `indisvalid` from the `pg_index` table as a workaround.

## Online Schema Migration Tooling

MySQL's ecosystem provides mature, production-ready tools for online schema changes that Postgres still lacks.

If your table doesn't have foreign keys, you can use GitHub's trigger-less [gh-ost](https://github.com/github/gh-ost/), otherwise, Percona's trigger-based [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html) is also a battle-tested solution.

The Postgres community has developed several solutions, but none has matched the adoption rate of MySQL's gh-ost or pt-online-schema-change.

## Connection Handling

While Postgres' process-per-connection provides better isolation, it consumes more memory per connection, incurs context switching overhead, and is more prune to hit connection limit.

MySQL's thread-per-connection model is subject to entire server crashes caused by a query of death, but is more scalable.

There is a comprehensive discussion in the Postgres community about making the switch [Let's make Postgres multi-threaded](https://www.Postgres.org/message-id/flat/31cc6df9-53fe-3cd9-af5b-ac0d801163f4%40iki.fi). But because the process model is too deeply coupled with the rest of Postgres, it will be quite a challenging task to tackle.

---

MySQL remains the most popular open source database today, serving as the de facto database of the internet era. Looking back, two pillars drove its popularity:

**Pragmatic core team.** The original MySQL team prioritized features based on real-world use cases.

- Replication, the most critical building block for scalability and fast failover on commodity hardware, was introduced in version 3.23 in 2001—nearly a decade before Postgres added replication in 2010 (version 9.0).

- When the core team lacked bandwidth to build a better storage engine, they designed a pluggable storage engine architecture that opened the door to InnoDB.

**Vibrant community.** When internet companies experienced hypergrowth and needed an open source database that was performant and scalable on commodity hardware, MySQL was the obvious choice. Google's initial ads system, Twitter, Facebook, Alibaba, and GitHub all chose MySQL. Though exceptions like Instagram chose Postgres, most successful internet-era companies selected MySQL and contributed enhancements like MHA, RocksDB, gh-ost, and replication improvements. This created a generation of MySQL experts advocating the technology and spawned dedicated consulting businesses like Percona, which provided educational resources and indispensable tools like the Percona toolkit.

Tides are changing, and Postgres has become the new hotness in the AI era. While MySQL retains several architectural advantages over Postgres, will it hold the fort or become obsolete within 10 years?


Areas where MySQL shines and Postgres falls short.

Features I Wish Postgres Had but MySQL Already Has


At Bytebase, we work extensively with PostgreSQL and MySQL. Both databases have pros and cons. In this post, we’ll highlight where PostgreSQL shines and MySQL falls short. We also write a [follow-up post on where MySQL excels](/blog/features-i-wish-postgres-had-but-mysql-already-has).

## Transactional DDL

One of PostgreSQL's most underrated features is its support for transactional DDL (Data Definition Language). In PostgreSQL, you can wrap schema changes in transactions and roll them back if something goes wrong. Even better, PostgreSQL supports savepoints within DDL transactions for fine-grained control.

```sql
-- PostgreSQL: This works!
BEGIN;
ALTER TABLE users ADD COLUMN email_verified BOOLEAN DEFAULT FALSE;
SAVEPOINT after_column_add;
CREATE INDEX idx_users_email_verified ON users(email_verified);
-- Oops, index creation failed, but we can keep the column
ROLLBACK TO SAVEPOINT after_column_add;
COMMIT;
```

In MySQL, DDL statements are auto-committed, meaning there's no going back once you execute them. This makes database migrations risky, especially in production environments.

## Custom Types and Domains

PostgreSQL allows you to create custom data types and domains, providing better data modeling capabilities and stronger type safety.

```sql
-- PostgreSQL: Create custom types
CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');
CREATE DOMAIN email AS TEXT CHECK (VALUE ~* '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$');

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    name TEXT,
    email email,  -- Custom domain with validation
    current_mood mood  -- Custom enum type
);
```

MySQL's ENUM support is limited and doesn't offer the same flexibility. Custom domains with built-in validation rules help enforce data integrity at the database level, reducing the need for application-level validation and ensuring consistency across all applications that access the database.

## Array Types

PostgreSQL's native array support is incredibly powerful for storing and querying lists of values without requiring separate tables.

```sql
-- PostgreSQL: Arrays are first-class citizens
CREATE TABLE articles (
    id SERIAL PRIMARY KEY,
    title TEXT,
    tags TEXT[],  -- Array of tags
    view_counts INTEGER[]  -- Array of daily view counts
);

-- Query with array operations
SELECT * FROM articles
WHERE 'postgresql' = ANY(tags)
AND array_length(view_counts, 1) > 7;
```

It's worth noting that PostgreSQL arrays don't support referential integrity constraints yet - you can't have foreign key relationships on array elements. Despite this limitation, native array support eliminates the need for many-to-many relationship tables in certain scenarios and can significantly simplify data models.

## Common Table Expressions (CTEs)

PostgreSQL has supported CTEs since version 8.4 (2009), while MySQL only added basic CTE support in version 8.0 (2018). PostgreSQL's implementation is more mature and feature-complete.

```sql
-- PostgreSQL: Recursive CTEs for hierarchical data
WITH RECURSIVE employee_hierarchy AS (
    -- Base case: top-level managers
    SELECT id, name, manager_id, 1 as level
    FROM employees
    WHERE manager_id IS NULL

    UNION ALL

    -- Recursive case: subordinates
    SELECT e.id, e.name, e.manager_id, eh.level + 1
    FROM employees e
    JOIN employee_hierarchy eh ON e.manager_id = eh.id
)
SELECT * FROM employee_hierarchy ORDER BY level, name;
```

PostgreSQL's CTE implementation supports more advanced features like `MATERIALIZED` and `NOT MATERIALIZED` hints, giving you control over query optimization strategies.

## Row Level Security (RLS)

PostgreSQL's Row Level Security allows you to implement fine-grained access control directly at the database level.

```sql
-- PostgreSQL: Enable RLS and create policies
ALTER TABLE documents ENABLE ROW LEVEL SECURITY;

-- Policy for user-specific data
CREATE POLICY user_documents ON documents
    FOR ALL TO app_user
    USING (owner_id = current_setting('app.current_user_id')::INTEGER);

-- Policy for multi-tenant applications
CREATE POLICY tenant_isolation ON documents
    FOR ALL TO app_user
    USING (tenant_id = current_setting('app.tenant_id')::INTEGER);

-- Users can only see their own documents automatically
SELECT * FROM documents;  -- Only returns user's documents in their tenant
```

While RLS is excellent for simple `user_id` or `tenant_id` filtering, it becomes complex for more sophisticated authorization rules involving roles, permissions, or dynamic relationships. However, it still serves as a valuable defense-in-depth mechanism in your database security chain, ensuring that even if application-level security fails, sensitive data remains protected.

MySQL lacks built-in RLS, forcing developers to implement access control in application code or through complex view structures.

## Partial Indexes

PostgreSQL allows you to create indexes with WHERE clauses, making them smaller and more efficient for specific query patterns.

```sql
-- PostgreSQL: Index only active users
CREATE INDEX idx_active_users ON users (last_login)
WHERE status = 'active';

-- Index only pending orders
CREATE INDEX idx_pending_orders ON orders (created_at)
WHERE status = 'pending';
```

MySQL requires you to index the entire column, leading to larger indexes and potentially slower performance when you only need to optimize queries for a subset of data.

## Spatial Support

While both databases support spatial data, PostgreSQL's PostGIS extension provides far more comprehensive geospatial capabilities.

```sql
-- PostgreSQL with PostGIS: Advanced spatial queries
SELECT name,
       ST_Distance(location, ST_MakePoint(-73.935242, 40.730610)) as distance_km
FROM restaurants
WHERE ST_DWithin(location, ST_MakePoint(-73.935242, 40.730610), 1000)  -- Within 1km
ORDER BY location <-> ST_MakePoint(-73.935242, 40.730610)  -- KNN search
LIMIT 10;
```

MySQL's spatial support, while improving, doesn't match the depth and performance of PostGIS. For applications requiring serious geospatial functionality, PostgreSQL with PostGIS is the clear winner.

## Vector Support

With the rise of AI, PostgreSQL has embraced vector storage and similarity search through pgvector.

```sql
-- PostgreSQL with pgvector: Store and search embeddings
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding VECTOR(1536)  -- OpenAI embedding dimension
);

-- Find similar documents using cosine similarity
SELECT content, 1 - (embedding <=> '[0.1, 0.2, ...]'::vector) AS similarity
FROM documents
ORDER BY embedding <=> '[0.1, 0.2, ...]'::vector
LIMIT 5;
```

MySQL recently added vector support in version 9.0 with basic VECTOR data types and limited distance functions, but PostgreSQL's pgvector extension remains more mature with broader operator support, better indexing options (IVFFlat, HNSW), and a richer ecosystem of vector operations.

## Parser

PostgreSQL adheres more closely to SQL standards and provides better error messages and more predictable behavior. MySQL's parser has historically been more lenient, which can lead to subtle bugs and makes it harder to write portable SQL.

For tooling developers, the [libpg_query](https://github.com/pganalyze/libpg_query) project extracts PostgreSQL's actual parser code, providing 100% compatibility with PostgreSQL's SQL parsing behavior. For MySQL, the most widely used parser is from [TiDB](https://github.com/pingcap/tidb/tree/master/pkg/parser), but it has compatibility differences since it was built specifically for TiDB's needs rather than pure MySQL compatibility.

## Openness

PostgreSQL pairs a permissive PostgreSQL License (akin to BSD) with an extensible architecture, giving developers full freedom to embed, modify, or commercialize the database without license friction. Its community operates in the open: design discussions happen on the public [pgsql-hackers mailing list](https://www.postgresql.org/list/pgsql-hackers/), and every commit carries an explanatory message—see, for instance, the recent PG 18 additions of [virtual generated columns](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=83ea6c54025bea67bcd4949a6d58d3fc11c3e21b) and [NOT VALID NOT NULL constraints](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a379061a22a8fdf421e1a457cc6af8503def6252).

By contrast, MySQL’s dual-licensing model (GPL plus commercial) restricts downstream choices, and its main public touchpoint is an [opaque bug tracker](https://bugs.mysql.com/)—some issues are hidden from both the community and the original reporter.

As MongoDB’s CEO put it on the FY26 Q1 earnings call:

![mongo-on-pg-openness](/content/blog/features-i-wish-mysql-had-but-postgres-already-has/mongo-on-pg-openness.webp)

_Open source, open standard, and most importantly, not owned by any one vendor._

PostgreSQL embodies that ideal in a way MySQL cannot.


Areas where PostgreSQL shines and MySQL falls short.

Features I Wish MySQL Had but Postgres Already Has


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database DevSecOps tool which is [SOC 2 compliant](https://www.bytebase.com/blog/soc2-type2). We update the post every year.

</HintBlock>

| Update History | Comment            |
| -------------- | ------------------ |
| 2025/06/03     | Initial version.   |

## What is HIPAA

**HIPAA** stands for the **Health Insurance Portability and Accountability Act**, a U.S. federal law enacted in 1996. Its primary goals are to:

- Protect patient health information (PHI)
- Ensure data privacy and security
- Streamline healthcare data flow and insurance processes

### Key Components of HIPAA

1. **Privacy Rule**
   - Defines standards for who can access and share PHI
   - Applies to healthcare providers, insurers, and their business associates
   - Requires patient consent for most disclosures of health data

2. **Security Rule**
   - Requires safeguards (administrative, physical, and technical) to protect electronic PHI (ePHI)
   - Examples include encryption, access controls, and audit logs

3. **Breach Notification Rule**
   - Mandates notification to individuals, HHS, and media (if applicable) in the event of a data breach

4. **Enforcement Rule**
   - Establishes penalties for non-compliance, with fines up to $1.5 million per violation type annually

### Glossary

| Term     | Meaning                                                                 |
|----------|-------------------------------------------------------------------------|
| **HIPAA** | Health Insurance Portability and Accountability Act                   |
| **PHI**   | Protected Health Information                                           |
| **ePHI**  | Electronic Protected Health Information                                |
| **MFA**   | Multi-Factor Authentication                                            |
| **RBAC**  | Role-Based Access Control                                              |
| **SIEM**  | Security Information and Event Management                              |
| **TDE**   | Transparent Data Encryption                                            |

## Who Must Comply

**Covered Entities:** Health plans, healthcare providers, and clearinghouses that transmit health information electronically.

**Business Associates:** Third-party vendors who handle ePHI on behalf of covered entities, including:

- Cloud database service providers
- Database management tool vendors
- Backup and recovery service providers
- Database consultants

**Electronic Protected Health Information (ePHI):** Any identifiable health data maintained or transmitted electronically, such as:

- Medical records
- Billing and insurance data
- Appointment schedules
- Metadata and logs containing patient identifiers

## Security Requirements at the Database Layer

### Access Controls

**Objective:** Ensure only authorized users and applications can access ePHI.

Access control is the cornerstone of HIPAA technical safeguards. It involves restricting data access to only authorized individuals and systems based on their roles and responsibilities.

| Specification           | Database Implementation                   |
| ----------------------- | ----------------------------------------- |
| **User Identification** | Unique logins linked to organizational ID |
| **Emergency Access**    | Special accounts with audit logging       |
| **Auto Logoff**         | Session timeouts by role                  |
| **Encryption**          | TDE and column-level encryption           |

#### Role-Based Access and Enforcement

Role-based access control (RBAC) assigns permissions based on a user's job function. This minimizes exposure of sensitive data.

- **Clinical roles:** Access to patient data for diagnosis and treatment
- **Administrative roles:** Access to billing and scheduling data
- **Technical roles:** Limited access to manage infrastructure
- **Audit roles:** Read-only access for compliance review

Additional safeguards include:

- View/row/column-level security
- Dynamic data masking for sensitive fields

**Change Management:**

- Standardized access request and approval process
- Non-prod testing
- Documented audit trail
- Quarterly review and cleanup

<HintBlock type="info">

Bytebase provides **RBAC, just-in-time access, change management, risk-based approval flows, data masking, and audit logging**.

</HintBlock>

### Audit Controls and Monitoring

**Objective:** Log and monitor all access and activity involving ePHI.

Audit controls are necessary to record access and changes to systems managing ePHI. These logs are crucial for detecting and responding to unauthorized actions and fulfilling regulatory investigations.

- Track user queries, data changes, login attempts, and privilege grants
- Monitor schema updates and configuration changes
- Detect anomalies and violations using alerting systems
- Integrate with SIEM tools for centralized logging and analysis

| Requirement     | Implementation               |
| --------------- | ---------------------------- |
| **Retention**   | 6+ years with auto-archival  |
| **Protection**  | Write-once, signed logs      |
| **Analysis**    | Use query/reporting tools    |
| **Performance** | Dedicated infra or streaming |

<HintBlock type="info">

Bytebase provides **audit logging and anomaly detection**.

</HintBlock>

### Data Integrity and Change Management

**Objective:** Prevent unauthorized data alteration or loss.

HIPAA requires organizations to protect data from improper alteration. In databases, this includes:

- Storage-level integrity using checksums and RAID
- Database-level constraints like referential integrity
- Application-level validation and business logic enforcement

Formal change management ensures that updates are reviewed, tested, and documented.

| Change Type | Approval               | Testing           | Timeline  |
| ----------- | ---------------------- | ----------------- | --------- |
| Emergency   | Security officer       | Immediate         | < 4 hrs   |
| Standard    | Technical & compliance | QA testing        | 5–10 days |
| Major       | Executive + full QA    | Regression + perf | 2–4 weeks |

- Risk assessment and rollback plans must be in place

<HintBlock type="info">

Bytebase provides **change management, risk-based approval flows, rollback and change history**.

</HintBlock>

### Encryption and Data Protection

**Objective:** Protect ePHI during storage and transmission.

Encryption is a required safeguard under the HIPAA Security Rule.

**At Rest:**

- Transparent Data Encryption (TDE) encrypts the full database without altering the application
- Column-level encryption is used for fields like SSNs or diagnoses
- Encryption keys should be stored in enterprise-grade key management systems with rotation policies

**In Transit:**

- TLS/SSL for all connections
- VPNs or internal certificate-based communication between services

**Change Management:**

- Key rotation schedules
- Algorithm updates and phased rollouts
- Secure handling of encryption configuration changes

<HintBlock type="info">

Bytebase provides **database connection encryption, external secret manager, secret variable and audit logging**.

</HintBlock>

### Authentication and Authorization

**Objective:** Confirm user identity and enforce privileges.

HIPAA requires strong user authentication to ensure that only approved individuals can access ePHI.

- **MFA:** Combine passwords with tokens or biometrics
- **SSO integration:** Streamlines access and centralizes identity control
- **LDAP/AD support:** Synchronize accounts across platforms
- **Privileged access:** Segregate and monitor admin accounts
- **Regular recertification:** Periodic access reviews

<HintBlock type="info">

Bytebase provides **MFA, SSO, LDAP support, and audit logging**.

</HintBlock>

## HIPAA Retention Requirements

HIPAA mandates that organizations retain compliance documentation, including audit logs and system configurations, for a minimum of 6 years. However, medical record retention varies by state law.

A data lifecycle management strategy helps fulfill these obligations.

**Implementation Tips:**

- Classify data by type, usage, and legal requirements
- Assign appropriate storage tiers
- Automate archival and deletion workflows
- Enable legal hold features for litigation and investigations

| Tier           | Use                  | Retention | Access                      |
| -------------- | -------------------- | --------- | --------------------------- |
| **Hot**        | Active clinical data | +1 year   | Real-time                   |
| **Warm**       | Recent history       | 2–7 years | Moderate speed              |
| **Cold**       | Archive              | 8+ years  | Occasional                  |
| **Compliance** | Legal mandates       | Per law   | Immutable + verified delete |

### Secure Disposal

When ePHI reaches the end of its lifecycle, it must be permanently and securely removed.

- Logical deletion: Remove data from active systems with verification
- Physical destruction: Shred drives or degauss tapes
- Cryptographic erasure: Destroy encryption keys, rendering data unreadable
- Maintain audit logs for all disposal activities

<HintBlock type="info">

Bytebase provides **data classification and audit logging**, all the meta data is stored in your own database.

</HintBlock>

## Conclusion

By integrating strong security controls, formal change management, and automation tools like Bytebase, healthcare organizations can safeguard sensitive data and operate efficiently while staying compliant.


This guide focuses on HIPAA compliance at the database layer, providing healthcare organizations and database teams with practical guidance for implementing HIPAA Security Rule requirements.

HIPAA Data Security and Retention Requirements


Online index creation is a critical capability for modern databases, allowing you to add indexes without blocking concurrent read/write operations. This comparison examines the two leading approaches: PostgreSQL 17's `CREATE INDEX CONCURRENTLY` and MySQL 8.0's Online DDL with `ALGORITHM=INPLACE`.

## Basic Syntax

### PostgreSQL 17

```sql
CREATE INDEX CONCURRENTLY idx_email ON users (email);
```

### MySQL 8.0

```sql
-- Basic online index (equivalent to PostgreSQL CONCURRENTLY)
CREATE INDEX idx_email ON users (email)
ALGORITHM=INPLACE, LOCK=NONE;

-- Different concurrency levels
CREATE INDEX idx_status ON users (status)
ALGORITHM=INPLACE, LOCK=SHARED;  -- Blocks writes, allows reads

CREATE INDEX idx_created ON users (created_at)
ALGORITHM=INPLACE, LOCK=EXCLUSIVE;  -- Fastest, blocks all access
```

## Internal Mechanisms: How They Actually Work

### PostgreSQL: CREATE INDEX CONCURRENTLY

**Traditional CREATE INDEX (Non-Online):**

```c
// Simplified internal flow
DefineIndex() {
    lock_table(EXCLUSIVE);        // Blocks ALL operations
    scan_table();                 // Single scan
    build_index_structure();
    write_to_disk();
    update_catalogs();
    unlock_table();
}
```

**CREATE INDEX CONCURRENTLY (Online):**

```sql
-- Phase 1: Create "invalid" index entry (brief exclusive lock)
INSERT INTO pg_class (relname, relkind) VALUES ('idx_name', 'i');
UPDATE pg_index SET indisvalid = false WHERE indexrelid = new_index_oid;

-- Phase 2: First table scan - build initial index
...

-- Phase 3: Second table scan - catch concurrent changes
...

-- Phase 4: Wait for conflicting transactions to complete
...

-- Phase 5: Mark index "valid" (brief exclusive lock)
UPDATE pg_index SET indisvalid = true WHERE indexrelid = index_oid;
```

### MySQL: ALGORITHM=INPLACE

**Traditional ALGORITHM=COPY:**

```c
mysql_alter_table() {
    create_temp_table_with_new_structure();
    lock_table_metadata(SHARED_UPGRADEABLE);
    copy_all_data_to_temp_table();      // 2x storage needed
    apply_concurrent_changes();
    lock_table_metadata(EXCLUSIVE);     // Brief exclusive lock
    swap_table_names();
    drop_old_table();
}
```

**ALGORITHM=INPLACE (Online):**

```c
// Phase 1: Preparation
innobase_inplace_alter_table_prepare() {
    dict_mem_index_create();              // Allocate index space
    row_log_allocate();                   // Set up change logging
    acquire_metadata_lock(SHARED_UPGRADEABLE);
}

// Phase 2: Building with change tracking
innobase_inplace_alter_table_build() {
    while (has_more_records()) {
        record = fetch_next_record();
        index_entry = build_index_entry(record);
        insert_into_index(index_entry);

        if (concurrent_change_detected()) {
            row_log_insert(change_info);   // Log concurrent changes
        }
    }
}

// Phase 3: Apply changes
innobase_inplace_alter_table_commit() {
    acquire_metadata_lock(EXCLUSIVE);     // Brief exclusive lock
    row_log_apply();                      // Apply logged changes
    dict_index_set_online_status(ONLINE);
    release_metadata_lock();
}
```

## Monitoring Progress

### PostgreSQL 17

```sql
-- Monitor concurrent index creation
SELECT
    pid,
    datname,
    phase,
    blocks_total,
    blocks_done,
    ROUND(100.0 * blocks_done / blocks_total, 2) as pct_complete
FROM pg_stat_progress_create_index;
```

### MySQL 8.0

```sql
-- Monitor online DDL progress
SELECT
    event_name,
    work_completed,
    work_estimated,
    ROUND(100 * work_completed / work_estimated, 2) as pct_complete
FROM performance_schema.events_stages_current
WHERE event_name LIKE '%alter%';
```

## Error Handling & Recovery

### PostgreSQL

```sql
-- Failed concurrent index leaves invalid index
SELECT schemaname, indexname
FROM pg_indexes
WHERE indexname LIKE '%_ccnew';

-- Manual cleanup required
DROP INDEX CONCURRENTLY idx_email_ccnew;

-- Retry
CREATE INDEX CONCURRENTLY idx_email ON users (email);
```

### MySQL

```sql
-- Automatic rollback on failure
-- No cleanup needed, just retry with different options

-- If LOCK=NONE fails, try LOCK=SHARED
CREATE INDEX idx_email ON users (email)
ALGORITHM=INPLACE, LOCK=SHARED;
```

## Postgres vs MySQL Comparison Series

- [Overall comparison](/blog/postgres-vs-mysql)
- [DDL Transaction Difference](/blog/postgres-vs-mysql-ddl-transaction)
- [Indexing Options](/blog/postgres-vs-mysql-indexing-options)
- [JSON Support](/blog/postgres-vs-mysql-json-support)

## References

1. **PostgreSQL 17 Documentation - Building Indexes Concurrently**  
   https://www.postgresql.org/docs/current/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY

1. **PostgreSQL 17 Documentation - Progress Reporting**  
   https://www.postgresql.org/docs/current/progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING

1. **MySQL 8.0 Reference Manual - Online DDL Operations**  
   https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html

1. **MySQL 8.0 Performance Schema - Events Stages**  
   https://dev.mysql.com/doc/refman/8.0/en/performance-schema-events-stages-current-table.html


Compare the online index creation difference between Postgres and MySQL

Postgres vs. MySQL: Online Index Creation


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database DevSecOps tool that is [SOC 2 Type II compliant](https://www.bytebase.com/blog/soc2-type2). Bytebase helps teams implement SOC 2 controls at the database layer, including access control, audit logging, change management, and data retention.

</HintBlock>

| Update History | Comment          |
|----------------|------------------|
| 2025/05/30    | Initial version. |

## What is SOC 2?

**SOC 2 (System and Organization Controls 2)** is a widely adopted security and compliance framework developed by the **American Institute of Certified Public Accountants (AICPA)**. It sets the standard for how service organizations should manage customer data to protect it from unauthorized access, security incidents, and operational failures — especially in cloud-based environments.

SOC 2 compliance is evaluated against five **Trust Services Criteria**:

- **Security** (mandatory) — Safeguarding systems and data from unauthorized access
- **Availability** — Ensuring systems operate reliably and as promised
- **Processing Integrity** — Verifying accurate and complete data processing
- **Confidentiality** — Protecting sensitive information from disclosure
- **Privacy** — Handling personal data appropriately

There are two report types:

- **Type I**: A snapshot of controls at a single point in time
- **Type II**: An audit of how well those controls operate over time (typically 3–12 months)

Type II reports are considered more rigorous and credible, especially for enterprise customers.

### Why SOC 2 Matters for Database Systems

SOC 2 is particularly relevant for:

- SaaS platforms and cloud service providers
- Organizations that manage large volumes of user data
- Teams responsible for storing, processing, or transmitting customer data in production environments

While SOC 2 encompasses multiple operational areas, **this article focuses specifically on the database-related aspects of SOC 2** — namely, **data security and retention requirements**.

These are the areas where tools like [Bytebase](/) can help engineering teams automate and enforce controls for compliance.

## Data Security Requirements

In SOC 2, **Security** is the only mandatory Trust Services Criteria — and it's especially relevant for **database systems**, which are often the **central store of sensitive customer data**. To meet SOC 2 expectations, organizations must establish strict controls to prevent unauthorized access, monitor database activities, and ensure secure operational practices.

### 1. Information Security

Organizations must implement robust technical controls to secure their database infrastructure:

- Enable **activity monitoring and alerting** for suspicious database queries
- Detect **database anomalies** and **slowdowns**
- Define **incident response plans** for database-specific security breaches

<HintBlock type="info">

**Bytebase** helps enforce these practices with built-in **audit log**, **anomaly detection**, and possible integration with incident workflows.

</HintBlock>

### 2. Access Controls

Access to production databases must be strictly limited:

- Use **multi-factor authentication (MFA)** for all database access
- Perform **quarterly permission reviews** for all database accounts
- Prevent data exfiltration through **download restrictions**
- Monitor production environments for **unauthorized access attempts**

<HintBlock type="info">

With **MFA**, **role-based access control (RBAC)**, and **just-in-time access**, Bytebase ensures only the right people in the right time can access your specific databases.

</HintBlock>

### 3. Change Management

Database changes are a major risk vector and must be carefully managed:

- Establish **formal change request processes**
- Enforce **approval workflows** before schema changes reach production
- Test all changes in **non-production environments**
- Maintain **audit logs** of every schema modification
- Implement **rollback procedures** for failed changes

<HintBlock type="info">

Bytebase offers **UI and GitOps-style workflows** for schema change management, **custom approval flows**, **change history**, and **1-click rollback**.

</HintBlock>

### Real-World Database Security Controls

Organizations may tailor their implementation to fit their architecture. Examples include:

- Using database management tool like Bytebase to manage database access approvals and schema deployments
- Restricting superuser roles to on-call DBAs with just-in-time access
- Setting up alerting for unusual query patterns in production databases
- Automating user access reviews and integrating audit logs into SIEM systems

The essential point is this: your controls must align with the **Security Trust Services Criteria** — regardless of whether they're manual, scripted, or automated through tools like **Bytebase**.

## Data Retention Requirements

Under SOC 2’s **Confidentiality** and **Privacy** criteria, organizations must manage data responsibly throughout its lifecycle. While SOC 2 doesn’t specify exact durations, it requires:

- Identifying and classifying sensitive data in databases  
- Defining retention periods by data type  
- Protecting data during retention  
- Ensuring secure and verifiable deletion afterward

<HintBlock type="info">

Bytebase users can align retention policies with **data classification, data masking, schema change management, and audit logs** to ensure enforcement across environments.

</HintBlock>

Here is the basic process to implement data retention policies:

### 1. Data Identification & Classification

- Audit what data is stored across all environments (e.g., production, staging)
- Classify by sensitivity: e.g., **Public**, **PII**, **PHI**, **Confidential**
- Map data types to retention and deletion policies

<HintBlock type="info">

Consider tagging schemas or tables in Bytebase by classification.

</HintBlock>

### 2. Legal and Compliance Requirements

- Identify applicable regulations: **GDPR**, **HIPAA**, **SOX**, **PCI DSS**, etc.
- Document how retention timelines map to legal obligations
- Include any **client contract clauses** affecting data storage

### 3. Retention Periods

- Define periods per data category or table
- Consider different rules for logs, transactions, and user profiles
- Document in a central retention policy referenced by DBAs and auditors

### 4. Secure Deletion Procedures

- Detect data that has reached its retention limit
- Apply deletion techniques appropriate to your database engine (e.g., row-level delete, partition drop, full table purge)
- Log and verify all deletions
- Automate purging when feasible

<HintBlock type="info">

Integrate deletion workflows with Bytebase’s change workflows and audit logs for traceability.

</HintBlock>

### Best Practices for Database Retention

- **Policy Review:** Update regularly to stay aligned with evolving laws and systems
- **Automation:** Use tooling to reduce manual effort and risk
- **Backups:** Include backup lifecycle management in your retention strategy
- **Auditing:** Log all deletion activities; regularly validate adherence
- **Legal Holds:** Implement workflows to override retention rules during investigations

## SOC 2 Implementation Strategies for Databases

| Focus Area              | Strategy |
|-------------------------|----------|
| **Access Control**      | RBAC, least privilege, just-in-time access, audit logs |
| **Encryption**          | At rest (AES-256), in transit (TLS), with secure key management |
| **Activity Monitoring** | Full query logging, alerting on suspicious behavior |
| **Change Management**   | Change history, approval flows, rollback plans |
| **Backup & Recovery**   | Secure, automated backups with tested restore plans |

<HintBlock type="info">

**Bytebase** helps implement and enforce these practices through DevSecOps automation and built-in auditability.

</HintBlock>

By formalizing your database data retention policy and combining it with secure, automated workflows, you strengthen both your **SOC 2 compliance** and your organization’s operational maturity. Tools like **Bytebase** not only enforce these controls, but also simplify evidence collection for audits.

SOC 2 Data Security and Retention Requirements for Database Systems and How Bytebase Helps

SOC 2 Data Security and Retention Requirements


_For the impatience, jump to the [comparison table](#comparison-table-postgresql-17-vs-mysql-8)_

PostgreSQL and MySQL stand as two of the most popular choices. Both systems have evolved significantly over the years, with PostgreSQL 17 and MySQL 8 representing their latest General Availability (GA) versions. One of the most critical aspects of database performance tuning is the effective use of indexes. This article compares the indexing capabilities of PostgreSQL 17 and MySQL 8, providing insights for experienced database administrators and developers.

## B-tree Indexes: The Foundation

B-tree indexes are the default and most commonly used index type in both PostgreSQL and MySQL. They are well-suited for equality and range queries on data that can be sorted.

### PostgreSQL B-tree Indexes

```sql
-- Basic B-tree index
CREATE INDEX idx_customer_name ON customers(name);

-- Multi-column B-tree index
CREATE INDEX idx_customer_location ON customers(city, state, country);

-- B-tree index with sort options
CREATE INDEX idx_product_price_desc ON products(price DESC NULLS LAST);
```

PostgreSQL B-trees can handle equality and range queries, as well as `BETWEEN`, `IN`, `IS NULL`, and `IS NOT NULL` conditions. They can also be used with pattern matching operators like `LIKE` and `~` when the pattern is anchored to the beginning of the string.

### MySQL B-tree Indexes

```sql
-- Basic B-tree index
CREATE INDEX idx_customer_name ON customers(name);

-- Multi-column B-tree index
CREATE INDEX idx_customer_location ON customers(city, state, country);

-- B-tree index with descending order (MySQL 8.0+)
CREATE INDEX idx_product_price_desc ON products(price DESC);
```

MySQL B-trees support equality and range queries, as well as `BETWEEN` and `IN` conditions. They can also be used with `LIKE` when the pattern is anchored to the beginning of the string.

### Key Differences

While both systems use B-tree as their default index type, PostgreSQL offers more control over NULL sorting with `NULLS FIRST`/`NULLS LAST` options and supports regex pattern matching with the `~` operator. MySQL added support for descending indexes in version 8.0, catching up with a feature PostgreSQL has had for many years.

## Hash Indexes: Equality-Only Optimization

Hash indexes are optimized for equality comparisons and generally provide faster lookups than B-trees for simple equality operations.

### PostgreSQL Hash Indexes

```sql
-- Hash index
CREATE INDEX idx_user_id_hash ON users USING HASH (user_id);
```

PostgreSQL's hash indexes are transaction-safe and crash-recoverable, making them suitable for production use.

### MySQL Hash Indexes

```sql
-- Hash index for MEMORY table
CREATE TABLE lookup (
    id INT NOT NULL,
    name VARCHAR(100),
    INDEX USING HASH (name)
) ENGINE = MEMORY;
```

In MySQL, explicit hash indexes are primarily available for MEMORY tables. InnoDB uses an adaptive hash index internally, but this is managed automatically by the storage engine.

### Key Differences

PostgreSQL's hash indexes are more versatile and can be used with any storage engine, while MySQL's explicit hash indexes are limited to MEMORY tables. PostgreSQL's implementation is also transaction-safe and crash-recoverable, making it more suitable for production environments.

## Specialized Index Types in PostgreSQL

PostgreSQL offers several specialized index types that have no direct equivalent in MySQL.

### GiST (Generalized Search Tree) Indexes

GiST indexes provide an infrastructure for various indexing strategies, particularly useful for complex data types:

```sql
-- GiST index for geometric data
CREATE INDEX idx_location ON cities USING GIST (location);

-- GiST index for text with trigram matching
CREATE EXTENSION pg_trgm;
CREATE INDEX idx_document_trigram ON documents USING GIST (content gin_trgm_ops);
```

### SP-GiST (Space-Partitioned GiST) Indexes

SP-GiST indexes support non-balanced disk-based data structures like quadtrees and k-d trees:

```sql
-- SP-GiST index for points
CREATE INDEX idx_points ON points USING SPGIST (point);

-- SP-GiST index for text pattern matching
CREATE INDEX idx_text_prefix ON documents USING SPGIST (content text_ops);
```

### GIN (Generalized Inverted Index) Indexes

GIN indexes are "inverted indexes" suitable for composite values like arrays and full-text search:

```sql
-- GIN index for array containment
CREATE INDEX idx_tags ON posts USING GIN (tags);

-- GIN index for full-text search
CREATE INDEX idx_document_search ON documents USING GIN (to_tsvector('english', content));
```

### BRIN (Block Range INdex) Indexes

BRIN indexes store summaries about values in consecutive physical block ranges, making them efficient for large tables with natural ordering:

```sql
-- BRIN index for timestamp data
CREATE INDEX idx_logs_timestamp ON logs USING BRIN (created_at);

-- BRIN index with larger block range
CREATE INDEX idx_temperature ON weather_data USING BRIN (recorded_at) WITH (pages_per_range = 32);
```

## Specialized Index Types in MySQL

MySQL also offers some specialized index types that have no direct equivalent in PostgreSQL.

### SPATIAL Indexes (R-tree)

MySQL uses R-tree indexes for spatial data:

```sql
-- Spatial index
CREATE TABLE locations (
    id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(100),
    location POINT NOT NULL SRID 4326,
    SPATIAL INDEX idx_location (location)
) ENGINE = InnoDB;

-- Query using spatial index
SELECT name, ST_AsText(location)
FROM locations
WHERE ST_Contains(
    ST_GeomFromText('POLYGON((-74 40.7, -74 40.9, -73.9 40.9, -73.9 40.7, -74 40.7))', 4326),
    location
);
```

### FULLTEXT Indexes

MySQL has built-in support for full-text search:

```sql
-- Full-text index
CREATE TABLE articles (
    id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    title VARCHAR(200),
    body TEXT,
    FULLTEXT INDEX idx_content (title, body)
) ENGINE = InnoDB;

-- Natural language mode search
SELECT id, title
FROM articles
WHERE MATCH(title, body) AGAINST('database optimization');

-- Boolean mode search
SELECT id, title
FROM articles
WHERE MATCH(title, body) AGAINST('+MySQL -PostgreSQL' IN BOOLEAN MODE);
```

### Multi-Valued Indexes

MySQL 8.0.17 introduced multi-valued indexes for JSON arrays:

```sql
-- Table with JSON column and multi-valued index
CREATE TABLE products (
    id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(100),
    tags JSON,
    INDEX idx_tags ((CAST(tags AS ARRAY)))
) ENGINE = InnoDB;

-- Query using multi-valued index
SELECT * FROM products
WHERE JSON_CONTAINS(tags, '"electronics"');
```

## Advanced Indexing Features

Both PostgreSQL and MySQL offer advanced indexing features that can significantly improve query performance in specific scenarios.

### Functional Indexes

Both systems support indexing expressions rather than just columns:

**PostgreSQL:**

```sql
-- Expression index
CREATE INDEX idx_email_domain ON users ((split_part(email, '@', 2)));
```

**MySQL:**

```sql
-- Functional index
CREATE INDEX idx_upper_name ON customers ((UPPER(name)));
```

### Covering Indexes

Both systems support covering indexes, which can satisfy a query without accessing the table:

**PostgreSQL:**

```sql
-- Covering index with included columns
CREATE INDEX idx_orders_customer ON orders (customer_id) INCLUDE (order_date, status);
```

**MySQL:**

```sql
-- Index that covers a query
CREATE INDEX idx_covering ON customers (id, name, email);
```

### Partial Indexes (PostgreSQL Only)

PostgreSQL supports partial indexes that only index a subset of rows:

```sql
-- Partial index for active users only
CREATE INDEX idx_active_users ON users (username) WHERE active = true;
```

### Invisible Indexes (MySQL Only)

MySQL 8.0 introduced invisible indexes for testing index removal without actually dropping the index:

```sql
-- Create invisible index
CREATE INDEX idx_test ON orders (status) INVISIBLE;

-- Make existing index invisible
ALTER TABLE orders ALTER INDEX idx_status INVISIBLE;

-- Temporarily use invisible indexes for a query
SET SESSION optimizer_switch = 'use_invisible_indexes=on';
```

## Index Maintenance

Proper index maintenance is crucial for maintaining optimal database performance.

### PostgreSQL

PostgreSQL provides several tools for index maintenance:

```sql
-- Rebuild an index
REINDEX INDEX idx_name;

-- Rebuild all indexes on a table
REINDEX TABLE table_name;

-- Update statistics
ANALYZE table_name;

-- Monitor index usage
SELECT * FROM pg_stat_user_indexes WHERE relname = 'table_name';
```

### MySQL

MySQL integrates index maintenance with table operations:

```sql
-- Update statistics
ANALYZE TABLE customers;

-- Rebuild table and indexes
OPTIMIZE TABLE customers;

-- View index information
SHOW INDEX FROM customers;

-- Query index metadata
SELECT * FROM information_schema.STATISTICS
WHERE table_schema = 'mydb' AND table_name = 'customers';
```

## Best Practices for Index Creation and Management

### PostgreSQL

1. **Use CONCURRENTLY for production environments**:

   ```sql
   CREATE INDEX CONCURRENTLY idx_new_index ON large_table(column);
   ```

   This avoids locking the table for writes during index creation.

1. **Monitor index usage**:

   ```sql
   SELECT indexrelname, idx_scan, idx_tup_read, idx_tup_fetch
   FROM pg_stat_user_indexes
   WHERE schemaname = 'public'
   ORDER BY idx_scan DESC;
   ```

   Regularly check which indexes are being used and which are not.

1. **Consider partial indexes for large tables with filtered queries**:

   ```sql
   CREATE INDEX idx_recent_orders ON orders(order_date)
   WHERE order_date > CURRENT_DATE - INTERVAL '3 months';
   ```

   This creates smaller, more efficient indexes.

1. **Use expression indexes for queries with functions**:
   ```sql
   CREATE INDEX idx_lower_email ON users(lower(email));
   ```
   This allows the optimizer to use the index for queries with `lower(email)`.

### MySQL

1. **Use ALGORITHM=INPLACE for online operations**:

   ```sql
   ALTER TABLE large_table ADD INDEX idx_new_index (column), ALGORITHM=INPLACE;
   ```

   This minimizes locking during index creation.

1. **Monitor index usage**:

   ```sql
   SELECT * FROM performance_schema.table_io_waits_summary_by_index_usage
   WHERE object_schema = 'your_database'
   ORDER BY count_star DESC;
   ```

   Check which indexes are being used and which are not.

1. **Use invisible indexes for testing**:

   ```sql
   CREATE INDEX idx_test ON orders(status) INVISIBLE;
   ```

   Test the impact of an index before making it visible to the optimizer.

1. **Consider prefix indexes for string columns**:
   ```sql
   CREATE INDEX idx_description ON products(description(50));
   ```
   This creates smaller indexes while maintaining good selectivity.

## Comparison Table: PostgreSQL 17 vs MySQL 8

| Feature                       | PostgreSQL 17             | MySQL 8                 |
| ----------------------------- | ------------------------- | ----------------------- |
| **B-tree indexes**            | Yes (default)             | Yes (default)           |
| **Hash indexes**              | Yes (all storage engines) | Limited (MEMORY tables) |
| **Spatial indexes**           | Yes (GiST)                | Yes (R-tree)            |
| **Full-text indexes**         | Via extensions            | Built-in                |
| **GiST indexes**              | Yes                       | No                      |
| **SP-GiST indexes**           | Yes                       | No                      |
| **GIN indexes**               | Yes                       | No                      |
| **BRIN indexes**              | Yes                       | No                      |
| **Multi-valued indexes**      | No                        | Yes (for JSON)          |
| **Functional indexes**        | Yes                       | Yes                     |
| **Partial indexes**           | Yes                       | No                      |
| **Covering indexes**          | Yes (explicit INCLUDE)    | Yes (implicit)          |
| **Invisible indexes**         | No                        | Yes                     |
| **Descending indexes**        | Yes                       | Yes                     |
| **Concurrent index creation** | Yes                       | No                      |
| **Index-only scans**          | Yes                       | Yes                     |
| **Parallel index scans**      | Yes                       | Limited                 |

## Conclusion and Recommendations

Both PostgreSQL 17 and MySQL 8 offer robust indexing capabilities that can significantly improve database performance when properly utilized. Choose PostgreSQL if you need specialized index types, complex data types, partial indexes, or concurrent index creation in production environments. Choose MySQL if you prefer built-in full-text search, invisible indexes for testing, multi-valued indexes for JSON data, or simpler configuration and management.

For most general-purpose applications, either system can provide excellent performance with proper index design. The key is understanding the specific query patterns of your application and selecting the appropriate index types and configurations to optimize those patterns.

Remember that indexes come with maintenance overhead and storage costs, so it's important to balance the benefits of faster queries against these costs. Regularly monitor index usage and performance to ensure your indexing strategy remains optimal as your application evolves.

## Postgres vs MySQL Comparison Series

- [Overall comparison](/blog/postgres-vs-mysql)
- [DDL Transaction Difference](/blog/postgres-vs-mysql-ddl-transaction)
- [JSON Support](/blog/postgres-vs-mysql-json-support)
- [Online Index Creation](/blog/postgres-vs-mysql-online-index-creation)

## References

**PostgreSQL Documentation**

1. [PostgreSQL 17 Index Types](https://www.postgresql.org/docs/current/indexes-types.html)
1. [PostgreSQL CREATE INDEX](https://www.postgresql.org/docs/current/sql-createindex.html)
1. [PostgreSQL Index-Only Scans](https://www.postgresql.org/docs/current/indexes-index-only-scans.html)
1. [PostgreSQL Partial Indexes](https://www.postgresql.org/docs/current/indexes-partial.html)
1. [PostgreSQL GiST and GIN Index Types](https://www.postgresql.org/docs/current/textsearch-indexes.html)
1. [PostgreSQL BRIN Indexes](https://www.postgresql.org/docs/current/brin-intro.html)

**MySQL Documentation**

1. [MySQL 8.0 CREATE INDEX Statement](https://dev.mysql.com/doc/refman/8.0/en/create-index.html)
1. [MySQL 8.0 How MySQL Uses Indexes](https://dev.mysql.com/doc/refman/8.0/en/mysql-indexes.html)
1. [MySQL 8.0 SPATIAL Index Optimization](https://dev.mysql.com/doc/refman/8.0/en/spatial-index-optimization.html)
1. [MySQL 8.0 Full-Text Search Functions](https://dev.mysql.com/doc/refman/8.0/en/fulltext-search.html)
1. [MySQL 8.0 Invisible Indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html)
1. [MySQL 8.0 Descending Indexes](https://dev.mysql.com/doc/refman/8.0/en/descending-indexes.html)
1. [MySQL 8.0 Multi-Valued Indexes](https://dev.mysql.com/doc/refman/8.0/en/create-index.html#create-index-multi-valued)


Compare the different indexing options between Postgres and MySQL

Postgres vs. MySQL: Indexing Options


JSON has become an essential format for storing and exchanging semi-structured data. Both PostgreSQL and MySQL have embraced JSON support. This article compares JSON support in their respective latest GA release PostgreSQL 17 and MySQL 8, examining their approaches to data types, functions, indexing, and performance. We'll provide code examples to illustrate key differences and help you decide which database might better suit your JSON-related needs.

## JSON Data Types and Storage

### PostgreSQL 17

PostgreSQL continues to offer two distinct JSON data types:

1. **`jsonb`** (Preferred): Stores data in a decomposed binary format. While slightly slower for input due to conversion overhead, it's significantly faster for processing since no reparsing is needed. It also supports indexing.

1. **`json`**: Stores an exact copy of the input text, preserving whitespace and the order of object keys. Each processing operation requires reparsing the text.

Additionally, PostgreSQL provides a `jsonpath` data type for efficient query mechanisms.

```sql
-- Creating JSON values in PostgreSQL 17
SELECT '{"name": "John", "age": 30}'::json;  -- json type
SELECT '{"name": "John", "age": 30}'::jsonb; -- jsonb type
```

### MySQL 8

MySQL implements a single `JSON` data type that's conceptually similar to PostgreSQL's `jsonb`. It stores JSON documents in an optimized binary format that enables quick access to elements without parsing the entire document.

```sql
-- Creating JSON values in MySQL 8
SELECT JSON_OBJECT('name', 'John', 'age', 30);
-- Or directly
SELECT CAST('{"name": "John", "age": 30}' AS JSON);
```

Both databases automatically validate JSON documents, ensuring only valid JSON can be stored in JSON columns.

## New Features and Enhancements

### PostgreSQL 17

`JSON_TABLE()` function is the most notable addition, which converts JSON data into a tabular format. This makes it easier to work with JSON data in SQL contexts.

```sql
-- PostgreSQL 17's new JSON_TABLE function
SELECT *
FROM JSON_TABLE(
    '{"users": [{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]}',
    '$.users[*]' COLUMNS(
        name TEXT PATH '$.name',
        age INT PATH '$.age'
    )
) AS jt;
```

### MySQL 8

MySQL 8 has had several years to refine its JSON support:

1. **JSON_TABLE Function**: MySQL 8 has offered the `JSON_TABLE()` function since its early releases, allowing for conversion between JSON and relational formats.

   ```sql
   -- MySQL 8's JSON_TABLE function
   SELECT *
   FROM JSON_TABLE(
       '{"users": [{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]}',
       '$.users[*]' COLUMNS(
           name VARCHAR(100) PATH '$.name',
           age INT PATH '$.age'
       )
   ) AS jt;
   ```

1. **JSON Schema Validation**: MySQL 8 provides built-in functions for validating JSON against JSON Schema.

   ```sql
   -- MySQL 8's JSON schema validation
   SELECT JSON_SCHEMA_VALID(
       '{"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}}',
       '{"name": "John", "age": 30}'
   );
   ```

1. **Multi-valued Indexing**: Since version 8.0.17, MySQL has supported multi-valued indexes on JSON arrays.

## Functions and Operators

### PostgreSQL 17

PostgreSQL provides a rich set of operators for JSON manipulation:

- `->`: Extract JSON object field or array element (returns JSON)
- `->>`: Extract JSON object field or array element as text
- `#>`: Extract JSON sub-object at specified path
- `#>>`: Extract JSON sub-object at specified path as text

For `jsonb` type, additional operators are available:

- `@>`: Contains (does the first JSON value contain the second?)
- `<@`: Contained by (is the first JSON value contained in the second?)
- `?`: Does the string exist as a top-level key?
- `?|`: Do any of these strings exist as top-level keys?
- `?&`: Do all of these strings exist as top-level keys?
- `||`: Concatenate JSON values

```sql
-- PostgreSQL 17 JSON operators
SELECT '{"user": {"name": "John", "email": "john@example.com"}}'::json -> 'user' ->> 'email';
-- Result: john@example.com

-- Check containment (jsonb only)
SELECT '{"name": "John", "tags": ["developer", "postgresql"]}'::jsonb @> '{"tags": ["postgresql"]}';
-- Result: true
```

### MySQL 8

MySQL uses functions for most JSON operations, though it also supports the `->` and `->>` operators:

- `JSON_EXTRACT()`: Extract data from JSON document
- `JSON_SET()`, `JSON_INSERT()`, `JSON_REPLACE()`: Modify JSON documents
- `JSON_CONTAINS()`, `JSON_CONTAINS_PATH()`: Check for values or paths
- `JSON_SEARCH()`: Find paths to specific values
- `JSON_MERGE_PATCH()`, `JSON_MERGE_PRESERVE()`: Combine JSON documents

```sql
-- MySQL 8 JSON functions
SELECT JSON_EXTRACT('{"user": {"name": "John", "email": "john@example.com"}}', '$.user.email');
-- Result: "john@example.com"

-- Using the -> operator (same as JSON_EXTRACT)
SELECT '{"user": {"name": "John", "email": "john@example.com"}}' -> '$.user.email';
-- Result: "john@example.com"

-- Using the ->> operator (same as JSON_UNQUOTE(JSON_EXTRACT(...)))
SELECT '{"user": {"name": "John", "email": "john@example.com"}}' ->> '$.user.email';
-- Result: john@example.com
```

## JSON Path Expressions

Both databases support JSON path expressions, but with different syntax and capabilities.

### PostgreSQL 17

PostgreSQL uses a path expression syntax with dots for object navigation and brackets for array access. In PostgreSQL 17, the SQL/JSON path language has been enhanced:

```sql
-- PostgreSQL 17 path syntax
SELECT '{"users": [{"name": "John"}, {"name": "Jane"}]}'::json #> '{users,1,name}';
-- Result: "Jane"

-- Using jsonpath for complex queries (enhanced in PostgreSQL 17)
SELECT jsonb_path_query(
    '{"users": [{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]}'::jsonb,
    '$.users[*] ? (@.age > 28).name'
);
-- Result: "John"
```

### MySQL 8

MySQL uses a dot notation similar to JavaScript for object properties and brackets for array indices:

```sql
-- MySQL 8 path syntax
SELECT JSON_EXTRACT('{"users": [{"name": "John"}, {"name": "Jane"}]}', '$.users[1].name');
-- Result: "Jane"

-- Using path expressions with conditions
SELECT JSON_SEARCH(
    '{"users": [{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]}',
    'one',
    'John'
);
-- Result: "$.users[0].name"
```

## Indexing JSON Data

### PostgreSQL 17

PostgreSQL's `jsonb` type supports several index types:

1. **GIN (Generalized Inverted Index)**: Ideal for checking containment and existence operations.

   ```sql
   -- Create a GIN index for all keys and values
   CREATE INDEX idx_data ON table_name USING GIN (data_column);

   -- Create a more compact GIN index for containment operations
   CREATE INDEX idx_data_path ON table_name USING GIN (data_column jsonb_path_ops);
   ```

1. **Functional indexes**: For specific JSON paths.

   ```sql
   -- Create an index on a specific JSON path
   CREATE INDEX idx_name ON table_name ((data_column->>'name'));
   ```

### MySQL 8

MySQL doesn't directly index JSON columns. Instead, it offers:

1. **Generated columns with indexes**: Extract values from JSON into generated columns that can be indexed.

   ```sql
   -- Create a virtual column and index it
   ALTER TABLE users ADD COLUMN name VARCHAR(100)
       GENERATED ALWAYS AS (data->>'$.name') STORED;

   CREATE INDEX idx_name ON users(name);
   ```

1. **Multi-valued indexes**: For JSON arrays (InnoDB, since MySQL 8.0.17).

   ```sql
   -- Create a multi-valued index on a JSON array
   CREATE INDEX idx_tags ON users((CAST(data->'$.tags' AS JSON) ARRAY));
   ```

## Performance Considerations

### PostgreSQL 17

- `jsonb` is optimized for processing but has conversion overhead during insertion
- GIN indexes provide efficient querying for complex JSON data
- Partial updates of JSON values are possible
- PostgreSQL 17 includes various query performance improvements, including for sequential reads

### MySQL 8

- Optimized binary storage format for quick element access
- Partial, in-place updates of JSON columns reduce overhead
- Compact binary log format for partial JSON updates
- Improved parsing and indexing performance in MySQL 8.0

## Code Example: Working with Nested JSON

Let's compare how both databases handle a common task: extracting and filtering nested JSON data.

### PostgreSQL 17

```sql
-- Create a table with JSON data
CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    data JSONB
);

-- Insert sample data
INSERT INTO products (data) VALUES
    ('{"name": "Laptop", "specs": {"cpu": "i7", "ram": 16, "storage": 512}}'),
    ('{"name": "Smartphone", "specs": {"cpu": "A15", "ram": 8, "storage": 256}}'),
    ('{"name": "Tablet", "specs": {"cpu": "M1", "ram": 8, "storage": 128}}');

-- Query for products with at least 8GB RAM and 256GB storage
SELECT data->>'name' AS product_name
FROM products
WHERE (data->'specs'->>'ram')::int >= 8
  AND (data->'specs'->>'storage')::int >= 256;

-- Result:
-- product_name
-- -----------
-- Laptop
-- Smartphone

-- Using PostgreSQL 17's JSON_TABLE function
SELECT jt.product_name, jt.ram, jt.storage
FROM products,
JSON_TABLE(
    data,
    '$' COLUMNS(
        product_name TEXT PATH '$.name',
        ram INT PATH '$.specs.ram',
        storage INT PATH '$.specs.storage'
    )
) AS jt
WHERE jt.ram >= 8 AND jt.storage >= 256;
```

### MySQL 8

```sql
-- Create a table with JSON data
CREATE TABLE products (
    id INT AUTO_INCREMENT PRIMARY KEY,
    data JSON
);

-- Insert sample data
INSERT INTO products (data) VALUES
    (JSON_OBJECT('name', 'Laptop', 'specs', JSON_OBJECT('cpu', 'i7', 'ram', 16, 'storage', 512))),
    (JSON_OBJECT('name', 'Smartphone', 'specs', JSON_OBJECT('cpu', 'A15', 'ram', 8, 'storage', 256))),
    (JSON_OBJECT('name', 'Tablet', 'specs', JSON_OBJECT('cpu', 'M1', 'ram', 8, 'storage', 128)));

-- Query for products with at least 8GB RAM and 256GB storage
SELECT data->>'$.name' AS product_name
FROM products
WHERE CAST(data->'$.specs.ram' AS UNSIGNED) >= 8
  AND CAST(data->'$.specs.storage' AS UNSIGNED) >= 256;

-- Result:
-- product_name
-- -----------
-- Laptop
-- Smartphone

-- Using MySQL 8's JSON_TABLE function
SELECT jt.product_name, jt.ram, jt.storage
FROM products,
JSON_TABLE(
    data,
    '$' COLUMNS(
        product_name VARCHAR(100) PATH '$.name',
        ram INT PATH '$.specs.ram',
        storage INT PATH '$.specs.storage'
    )
) AS jt
WHERE jt.ram >= 8 AND jt.storage >= 256;
```

## Conclusion

| Feature                      | PostgreSQL 17                    | MySQL 8                                 |
| ---------------------------- | -------------------------------- | --------------------------------------- |
| **JSON Data Types**          | `jsonb` and `json`               | Single `JSON` type                      |
| **Format Preservation**      | Yes (with `json` type)           | No                                      |
| **Binary Storage**           | Yes (with `jsonb` type)          | Yes                                     |
| **Automatic Validation**     | Yes                              | Yes                                     |
| **JSON_TABLE Function**      | Yes (new in PostgreSQL 17)       | Yes                                     |
| **Indexing Support**         | GIN indexes, functional indexes  | Generated columns, multi-valued indexes |
| **Containment Operators**    | Yes (`@>`, `<@`)                 | No (uses functions instead)             |
| **Path Expression**          | Yes (dot and bracket notation)   | Yes (dot and bracket notation)          |
| **Schema Validation**        | No built-in support              | Yes (`JSON_SCHEMA_VALID()`)             |
| **Partial Updates**          | Yes                              | Yes                                     |
| **Array Membership Testing** | Yes (`?`, `?\|`, `?&` operators) | Yes (`MEMBER OF()` operator)            |
| **Pretty Printing**          | No built-in function             | Yes (`JSON_PRETTY()`)                   |

PostgreSQL 17 offers broader JSON functionality. If your workloads rely heavily on JSON and demand advanced querying, its `jsonb` type with GIN indexes is often the better fit. MySQL, however, is steadily improving and already provides a growing set of built-in JSON functions.

## Postgres vs MySQL Comparison Series

- [Overall comparison](/blog/postgres-vs-mysql)
- [DDL Transaction Difference](/blog/postgres-vs-mysql-ddl-transaction)
- [Indexing Options](/blog/postgres-vs-mysql-indexing-options)
- [Online Index Creation](/blog/postgres-vs-mysql-online-index-creation)

## References

1. PostgreSQL Documentation. (2024). "JSON Types." Retrieved from https://www.postgresql.org/docs/current/datatype-json.html

1. PostgreSQL Documentation. (2024). "JSON Functions and Operators." Retrieved from https://www.postgresql.org/docs/current/functions-json.html

1. MySQL Documentation. (2024). "The JSON Data Type." Retrieved from https://dev.mysql.com/doc/refman/8.0/en/json.html

1. MySQL Documentation. (2024). "JSON Function Reference." Retrieved from https://dev.mysql.com/doc/refman/8.0/en/json-function-reference.html

1. PostgreSQL Documentation. (2024). "PostgreSQL 17.0 Release Notes." Retrieved from https://www.postgresql.org/docs/release/17.0/

1. Stokes, D. (2023). "JSON Data Improvements in MySQL 8.0." Oracle MySQL Summit 2023. Retrieved from https://downloads.mysql.com/events/mysql-summit-2023/Oracle_MySQL_Summit_2023_JSON.pdf


Compare the JSON support difference between Postgres and MySQL

Postgres vs. MySQL: JSON Support


PostgreSQL's [built-in Row-Level Security (RLS)](https://www.postgresql.org/docs/current/ddl-rowsecurity.html) allows database administrators to define policies that control which rows can be returned by queries or modified by data manipulation commands on a per-user basis. When enabled, all access to a table must be allowed by a security policy, with table owners typically bypassing these restrictions unless explicitly configured otherwise.

Key characteristics include:

- Enabled at the table level with `ALTER TABLE ... ENABLE ROW LEVEL SECURITY`
- Policies can be specific to commands (SELECT, INSERT, UPDATE, DELETE) or roles
- Uses Boolean expressions to determine which rows are visible/modifiable
- Multiple policies can be combined using OR (permissive) or AND (restrictive)
- Superusers and roles with `BYPASSRLS` attribute bypass row security

While powerful, RLS has performance impact , limited flexibility, and operational overhead that hinder broader adoption. Let's review them in details and discuss the alternatives.

## Performance Impact

Let me break down the detailed performance differences between PostgreSQL RLS and views, examining the internal mechanisms that cause these performance disparities.

RLS policies are evaluated for _every single row_ during query execution. When you query a table with RLS enabled, PostgreSQL:

1. Scans the table (or uses an index)
1. For each row retrieved, evaluates ALL applicable RLS policies
1. Only returns rows that pass the policy checks
1. This happens even if the query has its own WHERE clauses

```sql
-- RLS Policy
CREATE POLICY tenant_isolation ON orders
FOR ALL TO app_user
USING (tenant_id = current_setting('app.tenant_id')::int);

-- This query will:
-- 1. Scan orders table
-- 2. For EACH row, check if tenant_id matches current setting
-- 3. Apply the user's WHERE clause
-- 4. Return matching rows
SELECT * FROM orders WHERE status = 'pending';
```

**Policy Composition Complexity**
Multiple RLS policies on the same table are combined with AND logic, requiring evaluation of each policy for every row:

```sql
-- Multiple policies compound the evaluation cost
CREATE POLICY tenant_policy ON orders
USING (tenant_id = current_setting('app.tenant_id')::int);

CREATE POLICY department_policy ON orders
USING (department_id IN (
  SELECT dept_id FROM user_departments
  WHERE user_id = current_setting('app.user_id')::int
));

-- Now EVERY row must pass BOTH policy checks
```

### Alternatives: Views

**Pre-Filtered Result Sets**
Views with security predicates are resolved during query planning, allowing the optimizer to push predicates down and use indexes effectively:

```sql
-- Security view
CREATE VIEW tenant_orders AS
SELECT * FROM orders
WHERE tenant_id = current_setting('app.tenant_id')::int;

-- Query against view
SELECT * FROM tenant_orders WHERE status = 'pending';

-- PostgreSQL can optimize this as:
-- SELECT * FROM orders
-- WHERE tenant_id = current_setting('app.tenant_id')::int
--   AND status = 'pending'
-- And use compound indexes on (tenant_id, status)
```

**Index Utilization**
The query planner can effectively use indexes when security predicates are part of the base query rather than row-by-row filters:

```sql
-- With RLS: Index on tenant_id might not be used optimally
-- because the security check happens after row retrieval

-- With views: Can use compound index (tenant_id, status) effectively
CREATE INDEX idx_orders_tenant_status ON orders(tenant_id, status);
```

The fundamental issue is that RLS operates as a post-processing security filter rather than an integral part of query optimization, while views allow the security constraints to be optimized as part of the core query execution plan.

The RLS version must:

1. Perform the full join
1. Evaluate RLS policies on each joined row
1. Filter results

The View version can:

1. Use indexes to pre-filter both tables
1. Join only relevant rows
1. Apply final predicates efficiently

Of course, View has its own limitations:

- **Proliferation of Views**: May need many views for different access patterns
- **Maintenance Overhead**: Changes to base tables may require view updates
- **Limited Dynamism**: Less flexible for highly dynamic permission models
- **Write Operations**: More complex to handle INSERT/UPDATE operations

## Flexibility Limitation

**SQL-Only Policy Expressions**

RLS policies are constrained to SQL expressions, which limits what you can express:

```sql
-- RLS can only do this:
CREATE POLICY simple_tenant_policy ON documents
USING (tenant_id = current_setting('app.tenant_id')::int);

-- But cannot do complex business logic like:
-- "Users can access documents if they're in the same department
--  AND it's during business hours
--  AND they've completed required training
--  AND the document hasn't been flagged by content moderation
--  AND external compliance API approves access"
```

**Static Context Limitations**

RLS policies are evaluated with limited context - primarily database session variables and current user information:

```sql
-- Limited to session context
CREATE POLICY time_based_policy ON sensitive_data
USING (
  EXTRACT(hour FROM NOW()) BETWEEN 9 AND 17 -- Crude business hours
  AND EXTRACT(dow FROM NOW()) BETWEEN 1 AND 5 -- Weekdays only
);

-- Cannot consider:
-- - User's actual timezone and location
-- - Company-specific holiday calendar
-- - User's current device/IP reputation
-- - Recent security events or risk scores
-- - Dynamic compliance requirements
```

**No External System Integration**

RLS cannot make external API calls or integrate with other systems:

```sql
-- This is impossible in RLS:
-- CREATE POLICY compliance_policy ON financial_records
-- USING (external_compliance_service_check(user_id, document_id, current_timestamp));

-- You're limited to data that exists in your PostgreSQL instance
```

### Alternatives: Application-Level Logic

While RLS forces you to work within the constraints of SQL and database session context, modern authorization systems can incorporate any data source, apply any business logic, and make decisions based on the full context of each request.

Below shows an ABAC (Attribute-Based Access Control) system that incorporates different attribute types into authorization decisions:

```javascript
// Using Casbin or similar ABAC engine
const authorizationEngine = {
  async checkAccess(subject, resource, action, context) {
    const policy = await this.evaluatePolicy({
      // User attributes
      user: {
        id: subject.id,
        roles: subject.roles,
        department: subject.department,
        clearanceLevel: subject.clearanceLevel,
        lastTrainingDate: subject.lastTrainingDate,
        currentLocation: await geoService.getLocation(context.ip),
        riskScore: await riskEngine.getUserRiskScore(subject.id),
      },

      // Resource attributes
      resource: {
        id: resource.id,
        classification: resource.dataClassification,
        owner: resource.owner,
        createdDate: resource.createdDate,
        lastModified: resource.lastModified,
        complianceFlags: await complianceService.getFlags(resource.id),
      },

      // Environmental attributes
      environment: {
        currentTime: new Date(),
        userAgent: context.userAgent,
        networkLocation: context.networkLocation,
        isBusinessHours: await businessCalendar.isBusinessHours(),
        threatLevel: await securityService.getCurrentThreatLevel(),
      },

      action: action,
    });

    return policy.decision;
  },
};

// Usage
const canAccess = await authorizationEngine.checkAccess(
  currentUser,
  requestedDocument,
  'read',
  requestContext,
);
```

## Operational Overhead

**Management Overhead**

DBAs become organizational bottlenecks when managing RLS policies due to the manual intervention required at every stage of the policy lifecycle. Policy creation demands deep PostgreSQL expertise to craft secure SQL logic while analyzing performance implications and cross-policy dependencies. Managing existing policies becomes increasingly complex as organizations accumulate hundreds of interconnected policies without proper versioning, dependency tracking, or impact analysis capabilities. Decommissioning policies requires manual archaeological work to identify relationships and ensure safe removal without breaking dependent systems.

**Developer Self-Service Usability Limitations**

Developers face friction when working with RLS as there are no self-service capabilities for access management or testing. RLS provides no mechanism for just-in-time access requests, emergency debugging access, or temporary permissions, forcing developers to rely on manual DBA intervention even during critical incidents. The development experience is impacted by inability to create local test environments with proper authorization logic, poor debugging capabilities for authorization failures, and no way to unit test against different access scenarios.

### Alternatives: Middleware + Tooling

Instead of using database-level RLS, you can implement an authorization middleware layer that intercepts queries and enforces security policies at the application tier.

**[Open Policy Agent (OPA)](https://www.openpolicyagent.org/)**: Policies are defined in the Rego language and managed in version control systems like Git, enabling best practices such as code reviews, automated testing, and CI/CD integration.

**Bytebase**: Use Bytebase as a database access management platform to provide developers with self-service access portals. It automates approval workflows and policy generation, removing the need for DBAs to manually create and revoke temporary permissions. Bytebase also offers audit trails, break-glass access procedures, and integrates with existing identity systems.

## Summary

RLS works well for straightforward data filtering—such as `tenant_id` isolation—where database-level enforcement offers fast and reliable security baselines. It excels at applying simple predicates that map directly to database queries, without requiring external integrations or complex business logic.

However, for a production-grade multi-tenant service, a layered approach is often more scalable:

1. **RLS for Basic Tenant Isolation**
   Use RLS to enforce simple, row-level access control based on fields like `tenant_id`. This provides efficient, in-database filtering with minimal operational overhead.

2. **Application Logic + OPA for Complex Policies**
   For advanced authorization needs, integrate policy engines like Open Policy Agent (OPA) at the application layer. OPA handles dynamic logic that SQL cannot express—such as time-based access, workflow states, external API checks, or user roles—while supporting Git-based version control, automated testing, and policy simulation.

3. **Bytebase for Just-in-Time Human Access**
   For operational scenarios like production troubleshooting or emergency access, Bytebase offers a self-service portal with automated approval workflows, audit trails, and break-glass procedures. This eliminates manual DBA intervention while preserving governance and traceability.


While PostgreSQL RLS is a powerful feature, it has inherent design limitations that hinder broader adoption. In this post, we’ll examine these limitations and explore alternatives.

PostgreSQL Row-level Security (RLS) Limitations and Alternatives


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database DevSecOps tool. We update the post every year.

</HintBlock>

| Update History | Comment            |
| -------------- | ------------------ |
| 2025/05/27     | Initial version.   |

Previously, understanding database structures required parsing lengthy SQL scripts or complex command-line interfaces, which was unintuitive and error-prone. To solve this, developers created graphical tools for visualizing database schemas, helping teams design and document databases more easily. Here, we explore four top open-source options for schema visualization in 2025.

![star-history](/content/blog/top-database-schema-diagram-tools/star-history.webp)

## DrawDB

[DrawDB](https://www.drawdb.app/)  is a free, browser-based database diagram editor with a simple drag-and-drop interface. It lets you design ER diagrams visually and export SQL scripts for MySQL, PostgreSQL, SQLite, MariaDB, and SQL Server — no account required.

**Key Features:**

- Intuitive UI with real-time collaboration
- Export diagrams as SQL or images
- Works offline (PWA-supported)

**GitHub Stats:** [30.3k stars](https://github.com/drawdb-io/drawdb), 2.1k forks

**Best For:** Quick schema design & sharing

![drawdb](/content/blog/top-database-schema-diagram-tools/drawdb.webp)
![drawdb-ui](/content/blog/top-database-schema-diagram-tools/drawdb-ui.webp)

## ChartDB

[ChartDB](https://chartdb.io/) is a smart, query-driven visualization tool that generates diagrams instantly from your database schema. It supports AI-powered DDL exports, making migrations between databases seamless.

**Key Features:**

- "Smart Query" for instant JSON schema parsing
- Supports PostgreSQL, MySQL, SQL Server, and NoSQL options
- No database credentials needed (security-focused)

**GitHub Stats:** [16.6k stars](https://github.com/chartdb/chartdb)

**Best For:** Teams managing multiple database systems

![chartdb](/content/blog/top-database-schema-diagram-tools/chartdb.webp)
![chartdb-ui](/content/blog/top-database-schema-diagram-tools/chartdb-ui.webp)

## Azimutt

[Azimutt](https://azimutt.app/) specializes in exploring large, complex schemas with advanced filtering and relationship tracking. Unlike traditional ERD tools, it handles messy real-world databases effortlessly.

**Key Features:**

- Search, filter, and trace table relationships
- Supports SQL (PostgreSQL, MySQL, SQL Server, Oracle ...) and NoSQL (MongoDB, Counchbase, SQLite ...)

**GitHub Stats:** [1.9k stars](https://github.com/azimuttapp/azimutt)

**Best For:** Enterprise-scale database documentation

![azimutt](/content/blog/top-database-schema-diagram-tools/azimutt.webp)
![azimutt-ui](/content/blog/top-database-schema-diagram-tools/azimutt-ui.webp)

## Liam ERD

[Liam ERD](https://liambx.com/) auto-generates interactive ER diagrams from existing schemas (Rails, Prisma, PostgreSQL) with zero setup. It’s ideal for visualizing large projects (100+ tables).

**Key Features:**

- Instant diagrams from schema files
- Zoom, pan, and filter for clarity
- Open-source (Apache 2.0 license)

**GitHub Stats:** [4k stars](https://github.com/liam-hq/liam)

**Best For:** Developers needing fast, no-fuss visualization

![liamerd](/content/blog/top-database-schema-diagram-tools/liamerd.webp)
![liamerd-ui](/content/blog/top-database-schema-diagram-tools/liamerd-ui.webp)

## Key Options Compared

| Tool       | Best For                  | Standout Features                     | Database Support               | GitHub Stars |
|------------|---------------------------|---------------------------------------|---------------------------------|-------------|
| **DrawDB** | Simple visualization      | Clean UI, no account needed           | MySQL, PostgreSQL, SQL Server   | 30.3k ⭐     |
| **ChartDB**| Multi-database teams      | AI-powered SQL exports                | 7+ SQL/NoSQL options            | 16.6k ⭐     |
| **Azimutt**| Complex schemas           | Advanced search & relationship maps    | SQL + MongoDB/Couchbase         | 1.9k ⭐      |
| **Liam ERD**| Auto-generated diagrams  | Zero-config setup                     | Rails, Prisma, PostgreSQL       | 4k ⭐        |

## Quick Recommendations

- **Startups/Simple Projects**: DrawDB (easiest to use)
- **Enterprise Teams**: Azimutt (handles complexity best)
- **Multi-DB Environments**: ChartDB (best for migrations)
- **Rails/Prisma Users**: Liam ERD (most automated)

## Emerging Trends

1. **AI Integration**: More tools adding smart suggestions (like ChartDB's exports)
2. **Real-Time Collaboration**: Team editing features becoming standard
3. **Broader NoSQL Support**: Tools expanding beyond relational DBs

All tools are actively maintained and free to use. For most developers, trying 2-3 options will help identify the best fit for their workflow.

Evaluate several database schema diagram tools, and help you to choose the right one for 2025

Top 4 Free, Open Source Database Schema Diagram Tools to Visualize Database Easier in 2025


Database schema changes are critical operations that require careful planning and execution. The ability to perform these changes safely and reliably is a key consideration when choosing a database management system. In this post, we'll compare how PostgreSQL 17 and MySQL 8 handle Data Definition Language (DDL) transactions, with a focus on atomicity and rollback capabilities.

## Understanding DDL Transactions

Before diving into the comparison, let's clarify what we mean by DDL transactions. DDL statements can be grouped together and either committed as a unit or rolled back entirely if something goes wrong.

There are two important concepts to distinguish:

1. **Transactional DDL**: The ability to include DDL statements within a multi-statement transaction block, with the option to commit or roll back all statements together.

2. **Atomic DDL**: The guarantee that individual DDL statements are atomic (all-or-nothing), but not necessarily supporting their inclusion in multi-statement transactions.

## PostgreSQL 17: True Transactional DDL

In PostgreSQL 17, DDL operations are fully transactional, meaning:

- DDL statements can be included in transaction blocks alongside DML statements
- Multiple DDL operations can be committed or rolled back as a single unit
- Savepoints can be used within transactions that include DDL statements
- If a transaction fails, all DDL changes are rolled back, leaving the database in its original state

There are only a few exceptions to this rule: operations on databases and tablespaces themselves (such as `CREATE DATABASE` or `DROP TABLESPACE`) cannot be rolled back. However, all other catalog operations are reversible.

## MySQL 8: Atomic DDL

Prior to MySQL 8, DDL operations in MySQL were not atomic at all. If a DDL statement failed partway through execution—such as an `ALTER TABLE` operation that was adding multiple columns or indexes—the database could be left in an inconsistent, partially-modified state.

MySQL 8 introduced a feature called **Atomic DDL** which represents a significant improvement over previous versions but differs fundamentally from PostgreSQL's approach.

In MySQL 8, DDL statements are atomic at the statement level, meaning:

- Individual DDL statements are either fully completed or fully rolled back
- DDL statements **implicitly commit** any active transaction before execution
- DDL statements cannot be part of a multi-statement transaction that can be rolled back
- Crash recovery ensures statement-level atomicity

MySQL's atomic DDL is implemented through a special internal `DDL_LOG` table in InnoDB that tracks the creation of files and structures during DDL execution. This log is used at commit/rollback time to clean up properly, ensuring that no orphaned files or index trees remain after a crash.

It's important to note that atomic DDL is only supported in MySQL 8 InnoDB storage engine. For tables using other storage engines, partial updates may still occur.

## Examples

To better understand these differences, let's run some DDLs that demonstrate how DDL transactions behave in both systems.

### PostgreSQL 17

Our 1st example makes all DDL operations wrap in a transaction. When we execute the `ROLLBACK` command, all tables and indexes are removed as if they were never created.

```sql
-- Start a transaction block
BEGIN;

-- Create a simple table
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    username VARCHAR(50) NOT NULL
);

-- Add an index
CREATE INDEX idx_username ON users(username);

-- Oops! We made a mistake and want to roll back all changes
ROLLBACK;

-- Verify that the table was not created
SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'users';
-- Should return no rows, as the transaction was rolled back
```

The 2nd example demonstrates how PostgreSQL allows partial rollbacks using `SAVEPOINT`, giving developers fine-grained control over schema changes.

```sql
BEGIN;

-- Create a table
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    username VARCHAR(50) NOT NULL
);

-- Create a savepoint
SAVEPOINT after_users_table;

-- Alter the table to add a column
ALTER TABLE users ADD COLUMN email VARCHAR(100);

-- Oops! We only want to roll back the column addition
ROLLBACK TO after_users_table;

-- Add a different column instead
ALTER TABLE users ADD COLUMN active BOOLEAN DEFAULT TRUE;

-- Commit the transaction
COMMIT;
```

### MySQL 8

In this MySQL example, despite wrapping the DDL statements in a transaction block, the `ROLLBACK` command does not undo the table creation. This is because each DDL statement implicitly commits the transaction before executing, making it impossible to roll back multiple DDL statements as a unit.

```sql
-- Try to use a transaction block (note: this won't work as expected for DDL)
START TRANSACTION;

-- Create a simple table
CREATE TABLE users (
    id INT AUTO_INCREMENT PRIMARY KEY,
    username VARCHAR(50) NOT NULL
);

-- Add an index
CREATE INDEX idx_username ON users(username);

-- Try to roll back all changes (won't work for DDL statements)
ROLLBACK;

-- Verify that the table was created despite the ROLLBACK
SHOW TABLES;
-- Will show 'users' table
```

However, MySQL 8 does provide statement-level atomicity:

```sql
-- This will either create all users or none
CREATE USER 'user1'@'localhost' IDENTIFIED BY 'password1',
           'user2'@'localhost' IDENTIFIED BY 'password2';

-- This will either drop all tables or none
DROP TABLE IF EXISTS table1, table2, table3;
```

## Conclusion

Let's examine the key differences between PostgreSQL 17 and MySQL 8 in how they handle DDL transactions:

| Feature                               | PostgreSQL 17                      | MySQL 8                                                 |
| ------------------------------------- | ---------------------------------- | ------------------------------------------------------- |
| **DDL Transaction Support**           | Full transactional DDL             | Atomic DDL (statement-level only)                       |
| **Multi-statement DDL Transactions**  | Yes                                | No                                                      |
| **DDL Rollback in Transaction**       | Yes                                | No (implicit commit)                                    |
| **Atomicity of Single DDL Statement** | Yes                                | Yes (InnoDB only)                                       |
| **Crash Recovery**                    | Full transaction recovery          | Statement-level recovery                                |
| **Savepoints with DDL**               | Supported                          | Not supported                                           |
| **Exceptions**                        | Database and tablespace operations | Database directory removal not part of atomic operation |
| **Implementation Mechanism**          | Write-Ahead Log (WAL)              | DDL_LOG in InnoDB                                       |

PostgreSQL's transactional DDL provides a higher level of safety for complex schema migrations. By wrapping multiple related changes in a transaction, administrators can ensure that the database remains in a consistent state, even if something goes wrong during the migration process. In contrast, MySQL 8 introduces atomic DDL at the statement level, meaning each individual DDL statement is executed as an all-or-nothing operation. While this is a notable improvement over earlier versions of MySQL, it still does not support multi-statement DDL transactions.

## Postgres vs MySQL Comparison Series

- [Overall comparison](/blog/postgres-vs-mysql)
- [Indexing Options](/blog/postgres-vs-mysql-indexing-options)
- [JSON Support](/blog/postgres-vs-mysql-json-support)
- [Online Index Creation](/blog/postgres-vs-mysql-online-index-creation)

## References

1. PostgreSQL Wiki: [Transactional DDL in PostgreSQL: A Competitive Analysis](https://wiki.postgresql.org/wiki/Transactional_DDL_in_PostgreSQL:_A_Competitive_Analysis)

1. MySQL Documentation: [Atomic Data Definition Statement Support](https://dev.mysql.com/doc/refman/8.4/en/atomic-ddl.html)

1. MySQL Blog: [Atomic DDL in MySQL 8.0](https://dev.mysql.com/blog-archive/atomic-ddl-in-mysql-8-0/)

1. MySQL Documentation: [Statements That Cannot Be Rolled Back](https://dev.mysql.com/doc/refman/8.2/en/cannot-roll-back.html)


Compare the DDL transaction difference between Postgres and MySQL

Postgres vs. MySQL: DDL Transaction Difference


<HintBlock type="info">

This post is maintained by Bytebase, an open-source database DevSecOps tool. We update the post every year.

</HintBlock>

| Update History | Comment          |
| -------------- | ---------------- |
| 2025/05/26     | Initial version. |

## Introduction

Object-Relational Mapping (ORM) libraries have become essential tools in modern TypeScript development, **bridging the gap between application code and database systems**. A good ORM not only simplifies database interactions but also enhances type safety, improves developer productivity, and helps maintain clean, maintainable code.

In 2025, TypeScript developers have several mature ORM options to choose from, this article provides a comprehensive comparison of five leading TypeScript ORMs: **Drizzle ORM**, **Prisma ORM**, **TypeORM**, **Sequelize**, and **MikroORM**.

## Drizzle ORM

[Drizzle](https://orm.drizzle.team/) is a lightweight, TypeScript-first ORM with a SQL-centric approach and zero dependencies — ideal for serverless environments. Described as **"a headless TypeScript ORM with a head,"** it emphasizes familiarity: **"If you know SQL, you know Drizzle."**

![drizzle](/content/blog/top-typescript-orm/drizzle.webp)

**Highlights:**

- Dual query APIs (SQL-like and relational)
- Zero dependencies (\~7.4kb min+gzip)
- Fully built in TypeScript
- Schema defined in TypeScript, closely mirroring SQL
- Dialect-optimized and supports automatic migrations via Drizzle Kit

**Code Example:**

```typescript
import { pgTable, serial, varchar } from 'drizzle-orm/pg-core';

export const users = pgTable('users', {
  id: serial('id').primaryKey(),
  name: varchar('name', { length: 50 }),
  email: varchar('email', { length: 100 }),
});
```

## Prisma ORM

[Prisma](https://www.prisma.io/) is a modern ORM for **Node.js and TypeScript** that prioritizes developer experience and type safety through a declarative, schema-first approach and powerful tooling.

![prisma](/content/blog/top-typescript-orm/prisma.webp)

**Highlights:**

- Define models in **Prisma Schema Language (PSL)**
- Auto-generated, type-safe client with strong IDE support
- Declarative migrations via Prisma Migrate
- Visual database management with Prisma Studio
- Introspect existing databases into schema
- Raw SQL support when needed
- Scaling tools like Prisma Accelerate and Pulse

**Code Example:**

```prisma
// schema.prisma
model User {
  id    Int    @id @default(autoincrement())
  name  String
  email String @unique
}

```

```typescript
// Usage
const user = await prisma.user.create({
  data: { name: 'Alice', email: 'alice@example.com' },
});
```

## TypeORM

[TypeORM](https://typeorm.io/) is a flexible and mature ORM for TypeScript and JavaScript that supports both **Active Record** and **Data Mapper** patterns, making it adaptable to different project architectures.

![typeorm](/content/blog/top-typescript-orm/typeorm.webp)

**Highlights:**

- Define entities using **TypeScript classes and decorators**
- Supports complex relationships and lazy/eager loading
- Powerful SQL-like query builder
- Migration tools for schema evolution
- Works with multiple databases and across platforms
- Dual architectural pattern support (Active Record & Data Mapper)
- Repository abstraction for structured data access

**Code Example:**

```typescript
import { Entity, PrimaryGeneratedColumn, Column } from 'typeorm';

@Entity()
export class User {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  name: string;

  @Column({ unique: true })
  email: string;
}
```

## Sequelize ORM

[Sequelize](https://sequelize.org/) is a mature, feature-rich ORM for Node.js with solid support for TypeScript. It balances abstraction and control with a comprehensive, Promise-based API.

![sequelize](/content/blog/top-typescript-orm/sequelize.webp)

**Highlights:**

- Async/await-friendly with Promise-based API
- Define models with validations, associations, and data types
- Supports PostgreSQL, MySQL, MariaDB, SQLite, and SQL Server
- Robust transaction and connection pooling support
- Comprehensive relationship handling (1:1, 1\:N, N\:M)
- Lifecycle hooks and raw SQL support

**Code Example:**

```typescript
import { Sequelize, DataTypes } from 'sequelize';

const sequelize = new Sequelize('sqlite::memory:');

const User = sequelize.define('User', {
  id: { type: DataTypes.INTEGER, autoIncrement: true, primaryKey: true },
  name: DataTypes.STRING,
  email: { type: DataTypes.STRING, unique: true },
});
```

## MikroORM

[MikroORM](https://mikro-orm.io/) is a TypeScript-first ORM for Node.js, built on the **Data Mapper, Unit of Work, and Identity Map** patterns. It emphasizes type safety, clean architecture, and domain-driven design.

![mikroorm](/content/blog/top-typescript-orm/mikroorm.webp)

**Highlights:**

- Type-safe, decorator-based entity definitions
- Data Mapper for clean domain/data separation
- Unit of Work for change tracking and batching
- Identity Map to avoid duplicate instances
- Powerful query builder and migration system
- Supports PostgreSQL, MySQL, MariaDB, SQLite, MongoDB

**Code Example:**

```typescript
import { Entity, PrimaryKey, Property } from '@mikro-orm/core';

@Entity()
export class User {
  @PrimaryKey()
  id!: number;

  @Property()
  name!: string;

  @Property({ unique: true })
  email!: string;
}
```

## Comprehensive Comparison

| Feature               | Drizzle                         | Prisma                             | TypeORM                            | Sequelize                        | MikroORM                           |
| --------------------- | ------------------------------- | ---------------------------------- | ---------------------------------- | -------------------------------- | ---------------------------------- |
| **Philosophy**        | SQL-first, lightweight          | Schema-first, declarative          | Flexible, SQL-friendly             | Traditional ORM                  | Data Mapper, Unit of Work          |
| **Type Safety**       | Strong via TypeScript inference | Excellent via generated client     | Strong via decorators              | Moderate, added later            | Excellent TypeScript integration   |
| **Schema Definition** | TypeScript code, SQL-like       | Prisma Schema Language (.prisma)   | TypeScript classes with decorators | Model objects or classes         | Decorators or EntitySchema         |
| **Migrations**        | Drizzle Kit CLI                 | Prisma Migrate                     | Automatic or manual                | Migration system                 | Schema diffing and migration       |
| **Query Building**    | SQL-like and relational APIs    | Abstracted, object-oriented        | SQL-like query builder             | Comprehensive query API          | Type-safe query builder            |
| **Bundle Size**       | Extremely lightweight (~7.4kb)  | Larger with Rust engine            | Moderate                           | Larger                           | Larger, especially with ts-morph   |
| **Serverless**        | Optimized for serverless        | Requires Prisma Accelerate         | Requires configuration             | Not optimized                    | Not specifically optimized         |
| **Tooling**           | Drizzle Kit, Studio             | Client, Studio, Accelerate, Pulse  | CLI and utilities                  | CLI and plugins                  | CLI, schema generator, REPL        |
| **Learning Curve**    | Low for SQL users               | Moderate                           | Steeper                            | Steeper                          | Steeper due to patterns            |
| **Database Support**  | PostgreSQL, MySQL, SQLite       | PostgreSQL, MySQL, SQLite, MongoDB | Extensive SQL & MongoDB            | PostgreSQL, MySQL, SQLite, MSSQL | PostgreSQL, MySQL, SQLite, MongoDB |
| **Code Generation**   | Minimal                         | Heavy client generation            | Minimal                            | Minimal                          | Varies by metadata provider        |
| **Community Size**    | Growing rapidly                 | Large and active                   | Large, established                 | Very mature, widespread          | Growing                            |
| **Maturity**          | Newer but stable                | Mature but evolving                | Very mature                        | Most mature                      | Mature and stable                  |
| **Open Source**       | Fully open-source               | Most components (Studio is not)    | Fully open-source                  | Fully open-source                | Fully open-source                  |

### 💡 Philosophy

- **Drizzle**: Minimal, SQL-first. Great for those who know SQL.

- **Prisma**: Developer-friendly with type-safe auto-generated client.

- **TypeORM**: Traditional feel, supports both Active Record and Data Mapper.

- **Sequelize**: Full-featured ORM with wide adoption, less modern TS support.

- **MikroORM**: Clean separation of concerns, ideal for domain-driven design.

### ✅ Type Safety & Schema

- **Prisma** and **MikroORM** lead in type safety.

- **Drizzle** is excellent if you're defining schema directly in TS.

- **TypeORM** is solid, but complex queries may lose some safety.

- **Sequelize** has improved TS support, but still trails behind.

### ⚙️ Migrations & Tooling

- **Prisma** and **MikroORM** offer rich migration and tooling systems.

- **Drizzle** provides a clean CLI with SQL migration generation.

- **TypeORM** is flexible, but complex migrations need manual handling.

- **Sequelize** is powerful but can be verbose and harder to manage at scale.

### ⚡ Performance & Serverless

- **Drizzle** is the most lightweight and serverless-friendly.

- **Prisma** performs well but needs Accelerate in serverless.

- **Sequelize** and **MikroORM** have more overhead in serverless environments.

### 🌐 Ecosystem & Community

- **Sequelize** and **TypeORM** have the largest legacy user base.

- **Prisma** has a thriving modern community and frequent updates.

- **Drizzle** and **MikroORM** are rising fast with strong TypeScript focus.

## Use Case Comparison

| Use Case / Strength              | **Drizzle**         | **Prisma**          | **TypeORM**        | **Sequelize**       | **MikroORM**           |
| -------------------------------- | ------------------- | ------------------- | ------------------ | ------------------- | ---------------------- |
| **Serverless / Lightweight**     | ✅ Ideal            | ⚠️ Needs Accelerate | ⚠️ Needs tuning    | ❌ Not optimized    | ⚠️ Larger bundle       |
| **Type Safety (TypeScript)**     | ✅ Strong           | ✅ Excellent        | ✅ Good            | ⚠️ Moderate         | ✅ Excellent           |
| **Developer Experience**         | ✅ SQL-first        | ✅ Very high        | ⚠️ Traditional     | ⚠️ Verbose          | ✅ High (DDD-friendly) |
| **Schema-first Design**          | ❌ TS-as-schema     | ✅ PSL schema       | ⚠️ Decorator-based | ⚠️ Object/Decorator | ✅ Flexible            |
| **Complex SQL / Raw Queries**    | ✅ Native SQL style | ⚠️ Abstracted       | ✅ Supported       | ✅ Supported        | ✅ Supported           |
| **Visual Tools**                 | ⚠️ Basic Studio     | ✅ Prisma Studio    | ⚠️ CLI only        | ⚠️ Plugins          | ⚠️ REPL/CLI            |
| **Flexible Query Building**      | ✅ Dual API         | ✅ Client API       | ✅ Query builder   | ✅ Fluent API       | ✅ Query builder       |
| **DDD / Clean Architecture**     | ❌ Minimalist       | ⚠️ Not focused      | ⚠️ Optional        | ❌ Not suitable     | ✅ Designed for DDD    |
| **Validation / Lifecycle Hooks** | ❌ Not built-in     | ⚠️ Minimal          | ✅ Supported       | ✅ Strong           | ✅ Supported           |
| **Microservices Architecture**   | ✅ Very suitable    | ⚠️ Heavier client   | ✅ Possible        | ⚠️ Moderate size    | ⚠️ Bigger footprint    |

## Conclusion

In 2025, the TypeScript ORM ecosystem offers strong choices for every need:

- **Drizzle** is lightweight and serverless-friendly with a SQL-first design.
- **Prisma** shines in type safety and developer experience.
- **TypeORM** brings flexibility and mature features.
- **Sequelize** is stable and well-suited for legacy projects.
- **MikroORM** is ideal for TypeScript-heavy, DDD-style architectures.


Evaluate several TypeScript ORMs, and help you to choose the right one for 2025

The Knowledge Base

Explore the standard for database development

Solutions & Platform

Comparisons

Resources

Company

Legal

Solutions & Platform

Resources

Comparisons

Company

Legal